BackgroundService execution mode

The BackgroundTaskExecutionManager is a hosted background service responsible for executing background tasks for server methods. When a background task is triggered, it is persisted in the state store and enqueued for execution. The manager runs a continuous polling loop to claim and dispatch tasks with a configurable degree of concurrency.

When the state store becomes unavailable, the manager automatically switches to degraded mode: tasks are executed in-memory on the current replica without persistence. A circuit breaker monitors state store health and coordinates the transition between normal and degraded modes.

Warning

This execution mode does not support retries and error handling features. If a background server method fails, it will not be retried and no error handling method will be called even if it is defined.

Warning

Tasks executed in degraded mode are not persisted. They are lost if the replica restarts before completion.

Architecture

Normal Mode — Task Lifecycle

In normal mode, a task goes through the following stages:

sequenceDiagram
    participant Backend as Backend API
    participant Manager as BackgroundTaskExecutionManager
    participant Store as State Store
    participant Queue as State Store Queue

    Backend->>Manager: CreateAsync(args)
    Manager->>Store: Check existing task (GetStateAsync)
    Store-->>Manager: Not found
    Manager->>Queue: EnqueueAsync(taskId)
    Manager->>Store: TrySaveStateAsync(Created)
    Store-->>Manager: OK

    Note over Manager: Fast-path: try to claim immediately
    Manager->>Store: TryClaimSpecificTaskAsync(taskId)
    Store-->>Manager: Claimed (or declined if another replica wins)

    alt Fast-path succeeded
        Manager->>Manager: ExecuteJobAsync (inline dispatch)
    else Fast-path declined
        loop Polling loop (every QueuePollingIntervalMs)
            Manager->>Store: TryClaimNextCreatedTaskAsync
            Store-->>Manager: Next claimable task
            Manager->>Manager: ExecuteJobAsync
        end
    end

    Manager->>Store: TrySaveStateAsync(Started)
    Manager->>Manager: ProcessBackgroundTaskAsync
    Manager->>Store: TrySaveStateAsync(Succeeded / Failed)
    Manager->>Queue: TryRemoveFromQueueIfTerminalAsync

Task State Machine

stateDiagram-v2
    [*] --> Created : CreateAsync
    Created --> Started : Claimed and execution begins
    Started --> Succeeded : Execution completed successfully
    Started --> Failed : Unhandled exception
    Started --> StartupTimeoutExpired : StartupTimeout exceeded before execution
    Started --> Created : Lease expired and ResumeStartedTasksAfterLeaseExpiration=true
    Succeeded --> [*]
    Failed --> [*]
    StartupTimeoutExpired --> [*]

Fast-path vs. Polling-path

When a task is created, the manager immediately attempts a fast-path claim on the same replica (non-blocking, zero-wait). This minimises latency for lightly loaded systems. If the fast-path fails (e.g., concurrency slots are full or another replica claimed first), the task is picked up by the background polling loop that continuously monitors the queue.

flowchart TD
    A[CreateAsync] --> B{Circuit breaker open?}
    B -- Yes --> C[Degraded mode: in-memory channel]
    B -- No --> D[Persist task in state store + enqueue]
    D --> E{Concurrency slot available?}
    E -- No --> F[Polling loop will claim later]
    E -- Yes --> G[Fast-path: TryClaimSpecificTask]
    G --> H{Claim succeeded?}
    H -- No --> F
    H -- Yes --> I[ExecuteJobAsync]
    F --> J[Polling loop: TryClaimNextCreatedTask]
    J --> I
    C --> K[Degraded channel worker: ExecuteJobAsync]

Lease Heartbeat

For each claimed persistent task, a background heartbeat renews the lease at ClaimLeaseRenewIntervalMs intervals. If renewal fails (state store error or lease stolen by another replica), the execution is cancelled.

sequenceDiagram
    participant Executor as ExecuteJobAsync
    participant Heartbeat as RunClaimLeaseHeartbeatAsync
    participant Store as State Store

    Executor->>Heartbeat: Start heartbeat task
    loop Every ClaimLeaseRenewIntervalMs
        Heartbeat->>Store: TryRenewClaimLeaseAsync
        alt Renewal succeeded
            Store-->>Heartbeat: OK
        else Renewal failed
            Store-->>Heartbeat: Error / lease stolen
            Heartbeat->>Executor: CancelAsync (executionCts)
        end
    end
    Executor->>Heartbeat: CancelAsync (on completion)

Degraded Mode and Circuit Breaker

Overview

The StateStoreCircuitBreaker monitors the health of every state store operation performed by the manager. When too many consecutive failures occur, it opens the circuit and the manager enters degraded mode: tasks are executed locally in-memory without any state store interaction.

Circuit Breaker State Machine

stateDiagram-v2
    [*] --> Closed

    Closed --> Open : ConsecutiveFailures ≥ CircuitBreakerFailureThreshold
    Open --> HalfOpen : CircuitBreakerOpenDurationMs elapsed
    HalfOpen --> Closed : Next state store operation succeeds
    HalfOpen --> Open : Next state store operation fails

State	Behaviour
Closed	Normal operation. All state store requests are allowed. Consecutive failures are counted.
Open	State store is considered unavailable. All state store requests are blocked. Incoming tasks are routed directly to the in-memory degraded channel. After `CircuitBreakerOpenDurationMs`, one probe is allowed (→ HalfOpen).
HalfOpen	A single probe request is sent to the state store. On success the circuit closes; on failure it re-opens immediately.

Degraded Mode Behaviour

When the circuit is Open:

CreateAsync skips the state store and writes directly to an in-memory Channel<BackgroundTask>.
A dedicated degraded worker (RunDegradedWorkerAsync) reads from this channel and executes tasks concurrently, respecting the same MaxConcurrentTasks limit.
Task state is tracked in a ConcurrentDictionary (_degradedTasksInFlight) so that GetBackgroundTaskAsync still returns a result for in-progress tasks.
Progress messages (SignalR) are still published so the frontend receives status updates.
No recovery is possible after a replica restart: degraded tasks are lost.

Note

The metrics neos_background_tasks_degraded_created_total, neos_background_tasks_degraded_queue_size, and neos_background_tasks_circuit_breaker_state allow you to detect and monitor degraded mode usage.

Configuration Settings

All settings are configured through the application's IConfiguration object and follow the prefix BackgroundTaskExecution:.

1. ClaimLeaseDurationMs

Configuration Key: BackgroundTaskExecution:ClaimLeaseDurationMs

Default Value: 30000 (30 seconds)

Type: int

Minimum Value: 1000 (1 second)

Description: Defines the duration (in milliseconds) for which a background task executor can claim ownership of a task. After this lease period expires, the task may be claimed by another executor instance in the cluster.

Use Case:

Adjust this value based on your expected task execution time and cluster failover requirements
Longer leases reduce the likelihood of duplicate task execution but increase failover time if an executor crashes
Shorter leases enable faster failover but may cause duplicate execution attempts

Example Configuration:

{
  "BackgroundTaskExecution:ClaimLeaseDurationMs": 60000
}

2. ClaimLeaseRenewIntervalMs

Configuration Key: BackgroundTaskExecution:ClaimLeaseRenewIntervalMs

Default Value: 5000 (5 seconds)

Type: int

Minimum Value: 250 (250 milliseconds)

Description: Specifies the interval (in milliseconds) at which the background task executor renews its lease ownership of a currently executing task. This is essential for distributed scenarios where multiple executor instances may be running.

Use Case:

Controls how frequently the heartbeat is sent to renew the task lease
Should be significantly shorter than ClaimLeaseDurationMs to ensure reliable renewal
Recommended ratio: ClaimLeaseRenewIntervalMs ≤ ClaimLeaseDurationMs / 6

Example Configuration:

{
  "BackgroundTaskExecution:ClaimLeaseRenewIntervalMs": 10000
}

Note: If lease renewal fails, the executor logs a warning and cancels the in-flight execution.

3. ResumeStartedTasksAfterLeaseExpiration

Configuration Key: BackgroundTaskExecution:ResumeStartedTasksAfterLeaseExpiration

Default Value: false

Type: bool

Description: Determines whether background tasks that are in the "Started" state should be resumed by another executor instance if their lease expires without being completed.

Use Case:

Set to true when you want automatic recovery of tasks that were interrupted due to executor failures
Set to false when you prefer tasks to be abandoned if their lease expires (more conservative approach)
Enable this for critical long-running tasks that must eventually complete
Disable this if duplicate execution is a concern

Example Configuration:

{
  "BackgroundTaskExecution:ResumeStartedTasksAfterLeaseExpiration": true
}

Important: When enabled, this setting increases the risk of duplicate task execution if the original executor recovers after the lease expires.

4. MaxConcurrentTasks

Configuration Key: BackgroundTaskExecution:MaxConcurrentTasks

Default Value: 10

Type: int

Minimum Value: 1

Description: Controls the maximum number of background tasks that can be executed concurrently on a single executor instance. This limit applies to both normal and degraded mode.

Use Case:

Adjust based on available system resources (CPU, memory, database connections)
Increase on high-performance servers with sufficient resources
Decrease to prevent resource exhaustion on shared infrastructure
Set to 1 for sequential task execution

Example Configuration:

{
  "BackgroundTaskExecution:MaxConcurrentTasks": 20
}

Behavior:

The executor uses a SemaphoreSlim to enforce the concurrency limit across both the polling loop and the degraded channel worker
If the limit is reached, incoming tasks wait until a running task finishes

5. QueuePollingIntervalMs

Configuration Key: BackgroundTaskExecution:QueuePollingIntervalMs

Default Value: 5000 (5 seconds)

Type: int

Minimum Value: 10 (10 milliseconds)

Description: Specifies the interval (in milliseconds) at which the executor polls the task queue for new tasks when there are no running tasks or when running tasks haven't reached the concurrency limit.

Use Case:

Shorter intervals improve task latency but increase CPU usage from polling
Longer intervals reduce CPU overhead but increase task start latency
Balance based on your task volume and latency requirements
Typical range: 500-10000 ms

Example Configuration:

{
  "BackgroundTaskExecution:QueuePollingIntervalMs": 2000
}

Note: The fast-path mechanism bypasses this interval when a task is created on a replica with an available concurrency slot, ensuring minimal latency for the common case.

6. CircuitBreakerFailureThreshold

Configuration Key: BackgroundTaskExecution:CircuitBreakerFailureThreshold

Default Value: 5

Type: int

Minimum Value: 1

Description: The number of consecutive state store failures required to open the circuit breaker and switch to degraded mode. A failure is any exception thrown by a state store operation (excluding OperationCanceledException).

Use Case:

Lower values make the system switch to degraded mode more aggressively (fewer errors needed)
Higher values tolerate more transient errors before switching to degraded mode
Tune based on the reliability of your state store and acceptable error tolerance

Example Configuration:

{
  "BackgroundTaskExecution:CircuitBreakerFailureThreshold": 3
}

7. CircuitBreakerOpenDurationMs

Configuration Key: BackgroundTaskExecution:CircuitBreakerOpenDurationMs

Default Value: 30000 (30 seconds)

Type: int

Minimum Value: 1 (1 millisecond)

Description: The duration (in milliseconds) the circuit breaker remains open before allowing a single probe request to test state store availability (Half-Open state). If the probe succeeds, the circuit closes and normal mode resumes.

Use Case:

Shorter durations allow faster recovery but may cause repeated open/close cycles during prolonged outages
Longer durations reduce probe frequency during extended state store outages
Should be aligned with expected state store recovery times

Example Configuration:

{
  "BackgroundTaskExecution:CircuitBreakerOpenDurationMs": 60000
}

Configuration Example

appsettings.json

{
  "BackgroundTaskExecution": {
    "ClaimLeaseDurationMs": 30000,
    "ClaimLeaseRenewIntervalMs": 5000,
    "ResumeStartedTasksAfterLeaseExpiration": false,
    "MaxConcurrentTasks": 10,
    "QueuePollingIntervalMs": 500,
    "CircuitBreakerFailureThreshold": 5,
    "CircuitBreakerOpenDurationMs": 30000
  }
}

appsettings.Production.json (Recommended Production Settings)

{
  "BackgroundTaskExecution": {
    "ClaimLeaseDurationMs": 60000,
    "ClaimLeaseRenewIntervalMs": 10000,
    "ResumeStartedTasksAfterLeaseExpiration": true,
    "MaxConcurrentTasks": 20,
    "QueuePollingIntervalMs": 1000,
    "CircuitBreakerFailureThreshold": 5,
    "CircuitBreakerOpenDurationMs": 60000
  }
}

Kubernetes Secret

In Kubernetes, .NET configuration keys are mapped to environment variables using __ as the hierarchy separator (e.g., BackgroundTaskExecution:ClaimLeaseDurationMs → BackgroundTaskExecution__ClaimLeaseDurationMs).

Note

stringData values are stored as plain text in the manifest and converted to base64 by the Kubernetes API server. Use data with pre-encoded values in production pipelines.

Default settings secret:

apiVersion: v1
kind: Secret
metadata:
  name: background-task-execution-config
type: Opaque
stringData:
  BackgroundTaskExecution__ClaimLeaseDurationMs: "30000"
  BackgroundTaskExecution__ClaimLeaseRenewIntervalMs: "5000"
  BackgroundTaskExecution__ResumeStartedTasksAfterLeaseExpiration: "false"
  BackgroundTaskExecution__MaxConcurrentTasks: "10"
  BackgroundTaskExecution__QueuePollingIntervalMs: "500"
  BackgroundTaskExecution__CircuitBreakerFailureThreshold: "5"
  BackgroundTaskExecution__CircuitBreakerOpenDurationMs: "30000"

Recommended production settings secret:

apiVersion: v1
kind: Secret
metadata:
  name: background-task-execution-config
type: Opaque
stringData:
  BackgroundTaskExecution__ClaimLeaseDurationMs: "60000"
  BackgroundTaskExecution__ClaimLeaseRenewIntervalMs: "10000"
  BackgroundTaskExecution__ResumeStartedTasksAfterLeaseExpiration: "true"
  BackgroundTaskExecution__MaxConcurrentTasks: "20"
  BackgroundTaskExecution__QueuePollingIntervalMs: "1000"
  BackgroundTaskExecution__CircuitBreakerFailureThreshold: "5"
  BackgroundTaskExecution__CircuitBreakerOpenDurationMs: "60000"

Reference the secret in your Deployment by injecting each key as an environment variable:

containers:
  - name: neos-api
    envFrom:
      - secretRef:
          name: background-task-execution-config

Or selectively for individual keys:

containers:
  - name: neos-api
    env:
      - name: BackgroundTaskExecution__MaxConcurrentTasks
        valueFrom:
          secretKeyRef:
            name: background-task-execution-config
            key: BackgroundTaskExecution__MaxConcurrentTasks
      - name: BackgroundTaskExecution__ResumeStartedTasksAfterLeaseExpiration
        valueFrom:
          secretKeyRef:
            name: background-task-execution-config
            key: BackgroundTaskExecution__ResumeStartedTasksAfterLeaseExpiration

Configuration Best Practices

Lease Ratio: Ensure ClaimLeaseRenewIntervalMs is at least 5-6 times smaller than ClaimLeaseDurationMs
- Example: If ClaimLeaseDurationMs = 30000, set ClaimLeaseRenewIntervalMs ≤ 5000
Concurrent Tasks: Monitor resource usage when adjusting MaxConcurrentTasks
- Start conservative and increase gradually while monitoring performance
- Consider database connection pool size, memory, and CPU availability
Polling Interval: Lower QueuePollingIntervalMs for high-throughput scenarios
- Trade-off: lower latency vs. higher CPU usage
- The fast-path mechanism already minimises latency when a slot is available
Failover Strategy: For high-availability setups:
- Enable ResumeStartedTasksAfterLeaseExpiration
- Use shorter lease durations (20000-30000 ms) for faster failover
- Monitor for duplicate executions and implement idempotency if needed
Circuit Breaker: Tune CircuitBreakerFailureThreshold and CircuitBreakerOpenDurationMs together
- A lower threshold with a longer open duration gives a stable degraded mode during prolonged outages
- Monitor neos_background_tasks_circuit_breaker_state to detect unexpected degraded mode activation
Development vs. Production:
- Use default values for development
- Tune production values based on load testing and monitoring

Prometheus Metrics for Background Tasks

Implementation of Prometheus metrics to monitor background task execution.

📊 Exposed Metrics

Counters

Metric	Labels	Description
`neos_background_tasks_created_total`	`server_method_name`, `neos_tenant`	Total number of tasks created (normal + degraded)
`neos_background_tasks_succeeded_total`	`server_method_name`, `neos_tenant`	Total number of successfully completed tasks
`neos_background_tasks_failed_total`	`server_method_name`, `neos_tenant`	Total number of failed tasks
`neos_background_tasks_claimed_total`	`claimed_by`, `neos_tenant`	Total number of tasks claimed (acquired)
`neos_background_tasks_lease_renewed_total`	`claimed_by`	Total number of successful lease renewals
`neos_background_tasks_lease_renewal_failed_total`	`claimed_by`	Total number of failed lease renewals
`neos_background_tasks_startup_timeout_expired_total`	`server_method_name`, `neos_tenant`	Number of tasks that exceeded startup timeout
`neos_background_tasks_degraded_created_total`	`server_method_name`, `neos_tenant`	Total number of tasks created in degraded (in-memory) mode
`neos_background_tasks_state_store_errors_total`	(none)	Total number of state store errors that contributed to circuit breaker trips

Gauges

Metric	Description
`neos_background_tasks_queue_size`	Number of tasks waiting in the persistent state store queue
`neos_background_tasks_running`	Number of tasks currently executing (both normal and degraded)
`neos_background_tasks_by_state`	Distribution of tasks by state (Created, Started, Succeeded, Failed)
`neos_background_tasks_concurrent_capacity`	Configured maximum concurrent tasks capacity
`neos_background_tasks_degraded_queue_size`	Number of tasks currently in the in-memory degraded queue
`neos_background_tasks_circuit_breaker_state`	Current circuit breaker state: `0`=Closed, `1`=HalfOpen, `2`=Open

Histograms

Metric	Labels	Buckets	Description
`neos_background_tasks_execution_duration_seconds`	`server_method_name`, `neos_tenant`, `outcome`	0.1s - 51s	Task execution duration
`neos_background_tasks_wait_time_seconds`	`server_method_name`, `neos_tenant`	0.01s - 10s	Time spent waiting in queue
`neos_background_tasks_lease_validity_seconds`	`claimed_by`	1s - 5min	Lease validity duration

🎯 Alert Use Cases

Case	Query	Threshold	Action
Queue overload	`neos_background_tasks_queue_size`	> 1000	Scale horizontally
High error rate	`rate(neos_background_tasks_failed_total[5m])`	> 10%	Investigate logs
Unstable lease	`rate(neos_background_tasks_lease_renewal_failed_total[5m])`	> 0	Check cluster stability
Slow execution	`histogram_quantile(0.95, ...)`	> 30s	Profile server methods
Capacity reached	`neos_background_tasks_running / neos_background_tasks_concurrent_capacity`	> 0.9	Consider increasing config
Circuit breaker open	`neos_background_tasks_circuit_breaker_state`	== 2	Check state store availability
Tasks in degraded mode	`neos_background_tasks_degraded_queue_size`	> 0	Verify state store health
State store errors	`rate(neos_background_tasks_state_store_errors_total[5m])`	> 0	Investigate state store errors

Table of Contents

BackgroundService execution mode

Warning

Warning

Architecture

Normal Mode — Task Lifecycle

Task State Machine

Fast-path vs. Polling-path

Lease Heartbeat

Degraded Mode and Circuit Breaker

Overview

Circuit Breaker State Machine

Degraded Mode Behaviour

Note

Configuration Settings

Configuration Example

Note

Configuration Best Practices

Prometheus Metrics for Background Tasks

📊 Exposed Metrics

Counters

Gauges

Histograms

🎯 Alert Use Cases