Scheduled or Delayed Message Execution in Jasper

This is definitely not a replacement for something like Hangfire, but it’s very handy for what it does. 

Here’s a couple somewhat common scenarios in an event driven or messaging system:

• Handling a message failed, but with some kind of problem that might be resolved if the message is retried later in a few seconds or maybe even minutes
• A message is received that starts a long running process of some kind, and you may want to schedule a “timeout” process later that would send an email to users or do something to escalate the process if it has not finished by the scheduled time

For these kinds of use cases, Jasper supports the idea of scheduled execution, where messages can be sent with a logical “execute this message at this later time.”

Retry Later in Error Handling

The first usage of scheduled execution is in message handling error policies. Take this example below where I tell Jasper to retry handling an incoming message that fails with a TimeoutException again after a 5 second delay:

public class GlobalRetryApp : JasperRegistry
{
    public GlobalRetryApp()
    {
        Handlers
            .OnException<TimeoutException>()
            .RetryLater(5.Seconds());
    }
}

Behind the scenes, the usage of the RetryLater()method causes Jasper to schedule the incoming message for later execution if that error policy kicks in during processing.

Scheduling Messages Locally

To schedule a message to be processed by the current system at a later time, just use the IMessageContext.Schedule() method as shown below:

public async Task schedule_locally(IMessageContext context, Guid issueId)
{
    var timeout = new WarnIfIssueIsStale
    {
        IssueId = issueId
    };

    // Process the issue timeout logic 3 days from now
    // in *this* system
    await context.Schedule(timeout, 3.Days());
}

This method allows you to either express an exact time or to use a TimeSpan argument for delayed scheduling. Either way, Jasper ultimately stores the scheduled message against a UTC time. IMessageContext is the main service in Jasper for sending and executing messages or commands. It will be registered in your IoC container for any Jasper application.

Sending Scheduled Messages to Other Systems

To send a message to another system that should wait to execute the message on its end, use this syntax:

public async Task send_at_5_tomorrow_afternoon_yourself(IMessageContext context, Guid issueId)
{
    var timeout = new WarnIfIssueIsStale
    {
        IssueId = issueId
    };

    var time = DateTime.Today.AddDays(1).AddHours(17);


    // Process the issue timeout at 5PM tomorrow
    // Do note that Jasper quietly converts this
    // to universal time in storage
    await context.Send(timeout, e => e.ExecutionTime = time);
}

Do note that Jasper immediately sends the message. For right now, the thinking is that the receiving application will be responsible for handling the execution scheduling. We may choose later to make this be the responsibility of the sending application instead to be more usable when sending messages to other systems that aren’t using Jasper.

Persistent Job Scheduling

The default, in memory message execution scheduling is probably good enough merely for delayed message processing retries as shown above, However, if you’re using the scheduled execution as part of your business workflow, you probably want to be using the durable message persistence. Today your two options are to use either Postgresql with Marten, or the new Sql Server backed message persistence.

With durable messaging, the scheduled messages are persisted to a backing database so they aren’t lost if any particular node is shut down or the whole system somehow crashes. Behind the scenes, Jasper just uses polling to check the database for any scheduled messages that are ready to execute, and pulls these expired messages into the local worker queues of a running node for normal execution.

The scheduled messages can be processed by any of the running nodes within your system, but we take some steps to guarantee that only one node will execute specific scheduled messages. Rather than using any kind of leader election, Jasper just takes advantage of advisory locks in Postgresql or application locks in Sql Server as a lightweight, global lock between the running nodes within your application. It’s a much simpler (read, less effort on my time) mechanism than trying to do some kind of leader election between running nodes. It also allows Jasper to better spread the work across all of the nodes.