How it works?

The Task Scheduler relies on two technologies internally:

Quartz.NET for internal triggering of scheduled actions. For its operation, it uses tables created in the Task Scheduler database. These tables are internal and should never be manipulated directly.
Neos background server methods for running scheduled jobs. This means that the Task Scheduler never runs scheduled jobs. It communicates with the Task Runner service of the concerned cluster to make it run the job.

What happens when a scheduled task is created or activated?

A Quartz.NET trigger is created for the job with the key StartBackgroundServerMethod. The trigger key takes the ID of the scheduled task to make it easier to track.

What happens when a scheduled task is triggered?

Triggering a scheduled task corresponds to Quartz.NET calling the StartBackgroundServerMethod job.

An execution ID is first created.

If a StartupTimeout is specified, the processing starts by creating a Quartz.NET trigger for the job with the key HandleBackgroundServerMethodStartupTimeout. The trigger key takes the execution ID with the suffix /StartupTimeout. This trigger is set to execute when the StartupTimeout expires.

If the StartupTimeout is not specified, the processing starts by creating a Quartz.NET trigger for the job with the key HandleBackgroundServerMethodGlobalTimeout. The trigger key takes the execution ID with the suffix /GlobalTimeout. This trigger is set to execute when the Timeout expires.

A record is then created in the executions table with execution ID as the key. If an execution is still running on the same tenant, the execution is put into Skipped state and processing stops.

A message is finally sent via pub/sub to request the execution of the background server method. The TimeToLive of this message takes the StartupTimeout or, failing that, the Timeout. The Task Scheduler cluster then updates the execution state with the messages sent by the Task Runner when executing the background server methods.

What happens when the job with the key HandleBackgroundServerMethodStartupTimeout is triggered?

Case	Action
When the treatment is finished	Nothing happens in this case.
When processing is in progress	A `Quartz.NET` trigger for the job with the key `HandleBackgroundServerMethodGlobalTimeout` is created to put the execution into the `TimeoutExpired` state if it does not complete within the timeout period.
When treatment has not started	The execution is put in the `TimeoutExpired` state and a log is added. Due to the asynchronous and distributed operation of scheduled tasks, the execution may be in progress or start later. In this case, the execution will automatically exit the `TimeoutExpired` state.

What happens when the job with the key HandleBackgroundServerMethodGlobalTimeout is triggered?

Case	Action
When the treatment is finished	Nothing happens in this case.
When processing is not completed or not started	The execution is put in the `TimeoutExpired` state and a log is added. Due to the asynchronous and distributed operation of scheduled tasks, the execution may be in progress or start later. In this case, the execution will automatically exit the `TimeoutExpired` state.

What happens when events notifying the progress of background server methods are received?

The state of the execution is updated and a log line is created.

When the server method is completed (successfully or in error), the following tasks are additionally executed:

Existing Quartz.NET triggers for the execution of type HandleBackgroundServerMethodStartupTimeout or HandleBackgroundServerMethodGlobalTimeout are removed as they are no longer needed.
A Quartz.NET trigger is created for the job with the key PurgeBackgroundServerMethod. The trigger key has the format {TargetClusterName}/{TenantIdentifier}/{ExecutionIdentifier}. This trigger requests purging of traces of background server method execution.

When an event is received while the execution was in the TimeoutExpired state, it means that despite the timeout expiration, the server method is still running. The execution exits the TimeoutExpired state and a Quartz.NET trigger for the job with the key HandleBackgroundServerMethodGlobalTimeout is recreated to put the execution back to the TimeoutExpired state if it is not finished when the Timeout expires again.

Examples

Successful execution without StartupTimeout

sequenceDiagram
  autonumber

  participant TaskScheduler as TaskScheduler backend
  participant QuartzNET as QuartzNET
  participant TaskRunner as BusinessCluster task runner

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job StartBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Creates a Guid for execution: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodGlobalTimeout
  TaskScheduler->>TaskScheduler: Saves the execution with id AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>TaskScheduler: Publishes event StartBackgroundServerMethod
  Note right of TaskScheduler: Identifier: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Created

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Started

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Succeeded

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: BusinessCluster/T1/AD3830F8-7F4D-42A7-81D8-35F93C45012D, job: PurgeBackgroundServerMethod
  TaskScheduler->>QuartzNET: Unschedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodGlobalTimeout
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job PurgeBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Calls IBackgroundServerMethodManager.RequestPurgeAsync
  end

Successful execution with StartupTimeout when processing completes before the timeout expires

sequenceDiagram
  autonumber

  participant TaskScheduler as TaskScheduler backend
  participant QuartzNET as QuartzNET
  participant TaskRunner as BusinessCluster task runner

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job StartBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Creates a Guid for execution: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>TaskScheduler: Saves the execution with id AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>TaskScheduler: Publishes event StartBackgroundServerMethod
  Note right of TaskScheduler: Identifier: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Created

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Started

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Succeeded

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: BusinessCluster/T1/AD3830F8-7F4D-42A7-81D8-35F93C45012D, job: PurgeBackgroundServerMethod
  TaskScheduler->>QuartzNET: Unschedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodStartupTimeout
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job PurgeBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Calls IBackgroundServerMethodManager.RequestPurgeAsync
  end

Successful execution with StartupTimeout when processing starts before the startup timeout expires

sequenceDiagram
  autonumber

  participant TaskScheduler as TaskScheduler backend
  participant QuartzNET as QuartzNET
  participant TaskRunner as BusinessCluster task runner

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job StartBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Creates a Guid for execution: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>TaskScheduler: Saves the execution with id AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>TaskScheduler: Publishes event StartBackgroundServerMethod
  Note right of TaskScheduler: Identifier: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Created

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Started

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodGlobalTimeout
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Succeeded

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: BusinessCluster/T1/AD3830F8-7F4D-42A7-81D8-35F93C45012D, job: PurgeBackgroundServerMethod
  TaskScheduler->>QuartzNET: Unschedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodGlobalTimeout
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job PurgeBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Calls IBackgroundServerMethodManager.RequestPurgeAsync
  end

Execution not started before StartupTimeout expires

sequenceDiagram
  autonumber

  participant TaskScheduler as TaskScheduler backend
  participant QuartzNET as QuartzNET
  participant TaskRunner as BusinessCluster task runner

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job StartBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Creates a Guid for execution: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>TaskScheduler: Saves the execution with id AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>TaskScheduler: Publishes event StartBackgroundServerMethod
  Note right of TaskScheduler: Identifier: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>TaskScheduler: Sets execution state to TimeoutExpired and adds a log
  end

Execution not completed before GlobalTimeout expires

sequenceDiagram
  autonumber

  participant TaskScheduler as TaskScheduler backend
  participant QuartzNET as QuartzNET
  participant TaskRunner as BusinessCluster task runner

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job StartBackgroundServerMethod
  TaskScheduler->>TaskScheduler: Creates a Guid for execution: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>TaskScheduler: Saves the execution with id AD3830F8-7F4D-42A7-81D8-35F93C45012D
  TaskScheduler->>TaskScheduler: Publishes event StartBackgroundServerMethod
  Note right of TaskScheduler: Identifier: AD3830F8-7F4D-42A7-81D8-35F93C45012D
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Created

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  TaskRunner->>TaskRunner: Publishes event BackgroundServerMethodProgress with state Started

  rect rgb(240, 240, 240)
  TaskScheduler->>TaskScheduler: Event BackgroundServerMethodProgress received
  TaskScheduler->>TaskScheduler: Updates the execution state and adds a log
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job HandleBackgroundServerMethodStartupTimeout
  TaskScheduler->>QuartzNET: Schedules job
  Note right of TaskScheduler: Key: AD3830F8-7F4D-42A7-81D8-35F93C45012D/GlobalTimeout, job: HandleBackgroundServerMethodGlobalTimeout
  end

  rect rgb(240, 240, 240)
  QuartzNET->>TaskScheduler: Triggers job HandleBackgroundServerMethodGlobalTimeout
  TaskScheduler->>TaskScheduler: Sets execution state to TimeoutExpired and adds a log
  end

Table of Contents