Receiving webhooks

The NeosWebhooks module lets an external service, such as a telematics platform or a payment provider, call your cluster when an event occurs on its side. The module exposes one URL per receiver, authenticates the caller, discards the deliveries already processed, keeps a log of every receipt, and passes the verified event to a handler written in your business module. The administrator of each tenant configures the receivers, their secrets, and replays a receipt when needed.

Architecture overview

A receiver is identified by a key in its URL and uses one of two authentication modes:

Bearer: the caller is a service account that sends an OAuth 2.0 token. The gateway validates the token, as for any API call.
Hmac: the caller signs each request with a secret shared with the receiver. The route is anonymous for the gateway, and the module checks the signature.

flowchart TD
    A[External service] -->|POST /tenant/api/v1/webhooks/public/key| B[Gateway]
    A2[Service account] -->|POST /tenant/api/v1/webhooks/in/key + Bearer token| B
    B --> C[NeosWebhooks controller]
    C --> D{Receiver active, mode matching the route?}
    D -->|No| X[404]
    D -->|Yes| E{Caller authenticated?}
    E -->|No| R[401 or 403, receipt Rejected]
    E -->|Yes| L{Same delivery being processed?}
    L -->|Yes| C2[409, no receipt]
    L -->|No| F{Delivery already processed?}
    F -->|Yes| G[200, receipt Duplicate]
    F -->|No| H[Your handler]
    H --> I[Status code of the handler, receipt Processed or Failed]

The tenant always comes from the URL. On the public route, the gateway replaces a Neos-Tenant header sent by the caller. On the authenticated route, a header naming another tenant than the URL gets the request refused with 400 or 403: the caller sends no Neos-Tenant header, or the tenant of the URL. The tables of the module are isolated by tenant, so a receiver, its secrets, and its receipts belong to one tenant.

Setup

Add the NeosWebhooks module to your cluster. See Framework modules.
Give the cluster an encryption key before deploying it (see Encryption key).
Write a handler in a business module that references NeosWebhooks, and register it (see Writing a handler).
For receivers in Hmac mode, make the public route anonymous on the gateway (see Exposing the public route).
Place the Webhook receivers and Webhook receipts menu items of the module in the menu of your cluster, and give the functions to the roles concerned.
Create the receivers in the application (see Administration).

Encryption key

The module stores the secrets of the receivers encrypted in the database, with the key read from the EncryptionKey configuration of the cluster: a random value of 32 bytes, encoded in base64.

In development, a cluster without encryption key of its own uses the default key of Neos, which is enough to try the module.
In deployment, each cluster needs its own key in the secret of its backend. See Encryption key of a business cluster.

Warning

The default key of Neos is the same for every installation and is not a secret: a secret encrypted with it can be read by anyone with access to the database. Once secrets are stored, never change or lose the key of the cluster: with another key, the receivers can no longer read their secrets and refuse every signed request, until an administrator adds their secrets again.

Writing a handler

A handler implements IWebhookHandler<TPayload>. The module deserializes the JSON body to TPayload with the web defaults of System.Text.Json (camelCase, case-insensitive names, [JsonPropertyName] honored), then calls HandleAsync once the caller is authenticated and the delivery is new.

public sealed class SamsaraEvent
{
    public string? EventId { get; init; }

    public long EventMs { get; init; }

    public string? EventType { get; init; }

    public JsonElement? Event { get; init; }
}

public sealed class SamsaraEventsHandler : IWebhookHandler<SamsaraEvent>
{
    private readonly ITelematicsEventRepository _events;
    private readonly IUnitOfWork _unitOfWork;

    public SamsaraEventsHandler(ITelematicsEventRepository events, IUnitOfWork unitOfWork)
    {
        _events = events;
        _unitOfWork = unitOfWork;
    }

    public async Task<WebhookResult> HandleAsync(WebhookRequest<SamsaraEvent> request, CancellationToken cancellationToken)
    {
        SamsaraEvent? samsaraEvent = request.Payload;
        if (string.IsNullOrEmpty(samsaraEvent?.EventId))
        {
            return WebhookResult.BadRequest("The event has no identifier.");
        }

        // The delivery identifier of Samsara is not signed: the signed event identifier rules out duplicates,
        // and its unique index rules out the ones two concurrent deliveries record at the same time.
        if (await _events.GetQuery().Where(e => e.EventId == samsaraEvent.EventId).AnyAsync(cancellationToken))
        {
            return WebhookResult.NoContent();
        }

        _events.Add(new TelematicsEvent
        {
            EventId = samsaraEvent.EventId,
            EventType = samsaraEvent.EventType,
            OccurredAt = DateTimeOffset.FromUnixTimeMilliseconds(samsaraEvent.EventMs).UtcDateTime,
            Details = samsaraEvent.Event?.GetRawText(),
            ReceiverKey = request.ReceiverKey,
        });
        try
        {
            await _unitOfWork.SaveOrThrowAsync(cancellationToken);
        }
        catch (DuplicateKeyException)
        {
            return WebhookResult.NoContent();
        }

        return WebhookResult.NoContent();
    }
}

The cancellationToken of the handler does not depend on the request: when the caller gives up, the handler goes on and its result is recorded, so that a handler that has saved its changes never runs again for the next delivery of the event. The handler is cancelled after 1 minute, so that its response reaches the caller within the 100-second timeout of the gateway, and its receipt is then Failed with a 500. Pass the cancellationToken to every call that accepts one: a handler that ignores it keeps running after the 5-minute lock of the delivery expires, and a resent delivery can then reach it twice.

Register the handler in the Startup class of the business assembly, under a name that is unique in the application. The receivers reference the handler by this name.

public static void ConfigureServices(IServiceCollection services)
{
    services.AddWebhookHandler<SamsaraEventsHandler, SamsaraEvent>("samsara-events");
}

WebhookRequest<TPayload> gives the handler:

Property	Content
`Payload`	The deserialized body, `null` for an empty body
`RawBody`	The raw body, for a payload that needs its own deserialization options
`Headers`	The request headers, with case-insensitive names
`DeliveryId`	The identifier given by the caller to the delivery, if any
`ReceiverKey`	The key of the receiver that received the request
`IsReplay`	`true` when an administrator replays a stored receipt, the headers are then the kept headers

The handler returns a WebhookResult: Ok() (200), Accepted() (202), NoContent() (204), BadRequest(message) (400) or UnprocessableEntity(message) (422). A body that is not valid JSON for TPayload is answered with 400 without calling the handler.

Important

The handler runs while the caller waits for the response. Callers resend an event when the response is not 2xx or comes too late, so keep the handler short and start a background server method for a long process.

Note

An exception thrown by the handler is answered with 500 and a generic message, its details only going to the logs and to the receipt, recorded as Failed with the body so that the caller resends the event or an administrator replays it. The changes the handler did not save are never saved with the receipt, and the receipt is recorded even when the caller stops waiting for the response.

Authentication modes

Bearer

The caller is an application registered with the identity provider of your cluster: it obtains a token with the client credentials flow and calls /{tenant}/api/v1/webhooks/in/{key} with Authorization: Bearer <token>.

Create a service account whose login is the identifier of the calling application, and give its role the Sending by a service account function, under Webhooks.
Configure the identity provider so that the gateway accepts the token, see ServiceAccountClaimNames and the additional JWT authorities in the same article.
The delivery identifier is read in the webhook-id header when the caller sends it.

Hmac

The caller signs each request with a secret of the receiver and calls /{tenant}/api/v1/webhooks/public/{key}. Every provider computes an HMAC-SHA256 of the request with the shared secret, but each one signs its own text and sends the result in its own headers: the signature scheme of the receiver names this format. The module provides the StandardWebhooks scheme, which follows the Standard Webhooks specification used by several providers:

Signed content	Signature header	Timestamp header	Secret	Delivery identifier
`{webhook-id}.{webhook-timestamp}.{body}`	`webhook-signature: v1,{base64}`, several values allowed	`webhook-timestamp`	`whsec_` followed by base64	`webhook-id`

The scheme compares the signature in constant time, and refuses a timestamp (Unix seconds) further than the tolerance of the receiver from the reception time, 5 minutes by default. A receiver without a valid secret refuses every request.

Adding a signature scheme

For a provider with its own format, implement IWebhookSignatureScheme in a business module: IsValid checks the signature against the secrets of the receiver, GetDeliveryId reads the delivery identifier used to detect duplicates, and IsSecretValid refuses an entered secret the scheme cannot sign with, such as a value that is not base64 or that decodes to an empty key. When the provider does not sign this identifier, someone who intercepts a request can resend it with another identifier within the timestamp tolerance: the handler must then also rule out duplicates from an identifier of the signed body, kept under a unique index so that two concurrent deliveries cannot both record it, as the Samsara handler above does. When the provider sends its signature in a header whose name holds none of the words left out of the receipts (see Receipts), such as X-Digest, also return its name from SensitiveHeaderNames, so that it is never kept in a receipt. The HmacSignature class gives the steps shared by the HMAC schemes: computing the signature, comparing it in constant time, checking the timestamp, decoding a base64 or hexadecimal value.

The demo cluster TechnicalDemos adds the scheme of Samsara, which signs v1:{timestamp}:{body} and sends the signature in hexadecimal:

public sealed class SamsaraSignatureScheme : IWebhookSignatureScheme
{
    public string Name => "Samsara";

    public bool IsValid(WebhookSignatureContext context)
    {
        if (!context.Headers.TryGetValue("X-Samsara-Signature", out string? signatureHeader)
            || !signatureHeader.StartsWith("v1=", StringComparison.Ordinal)
            || !HmacSignature.IsTimestampValid(context.Headers, "X-Samsara-Timestamp", context.Now, context.TimestampTolerance))
        {
            return false;
        }

        byte[]? signature = HmacSignature.FromHex(signatureHeader["v1=".Length..]);
        if (signature == null)
        {
            return false;
        }

        string signedPrefix = $"v1:{context.Headers["X-Samsara-Timestamp"]}:";
        return context.Secrets
            .Select(HmacSignature.FromBase64)
            .OfType<byte[]>()
            .Any(key => HmacSignature.AreEqual(HmacSignature.Compute(key, signedPrefix, context.RawBody), signature));
    }

    public bool IsSecretValid(string secret)
    {
        return HmacSignature.FromBase64(secret) != null;
    }

    public string? GetDeliveryId(IReadOnlyDictionary<string, string> headers)
    {
        return headers.TryGetValue("X-Samsara-Request", out string? id) ? id : null;
    }
}

Register it in the Startup class of the business assembly. Its Name then appears in the list of schemes of the receiver form.

services.AddWebhookSignatureScheme<SamsaraSignatureScheme>();

Exposing the public route

The gateway requires authentication on every API route except the ones listed in its anonymous routes. Declare the public route of the webhooks with * segments, which stand for the tenant and the key:

gateway:
  anonymousRoutes:
    - /*/api/v1/webhooks/public/*

For a cluster reached under a prefix, as a nested cluster at /{prefix}/{tenant}/api/..., add the prefix before the tenant segment, for example /tracking/*/api/v1/webhooks/public/*. For a single-tenant cluster, list /api/v1/webhooks/public/* instead. Never list this route for a multitenant cluster: on a path without tenant segment, the gateway takes the tenant from the Neos-Tenant header chosen by the caller. In development, neos run reads the same AnonymousRoutes list from the cluster configuration files. See the gateway section of Prerequisites.

The /webapi/ prefix of the web client reaches the same endpoints. Give the callers the /api/v1/ address that the receiver form shows, and list a webapi pattern only for a caller that already uses it.

Caution

Never declare the webhooks/in route as anonymous: the gateway would no longer validate the token of the Bearer callers, and the module would refuse them.

Responses

Situation	Response	Receipt
Unknown tenant, unknown or inactive key, mode not matching route	404	None
Bearer route without a valid token, refused by the gateway	401	None
Bearer route with a `Neos-Tenant` header naming another tenant	400 or 403	None
Body larger than 1 MB	413	None
Delivery identifier longer than 200 characters	400	None
Missing or invalid signature, timestamp out of tolerance	401	`Rejected`
Bearer caller without the Sending by a service account function	403	`Rejected`
Delivery identifier being processed by another request	409	None
Delivery identifier already processed	200, the handler is not called	`Duplicate`
Handler response 2xx	Status code of the handler	`Processed`
Handler response 4xx, unknown handler, or exception	Status code of the handler, or 500	`Failed`

Only a Processed receipt makes a delivery a duplicate, including the receipt of a successful replay: when the caller resends an event after a failure that no replay has fixed, the handler is called again. A delivery without identifier is never considered a duplicate.

A Rejected receipt is recorded at most once a minute per receiver: the public route is anonymous, so the requests without a valid signature that follow within the minute are refused without a receipt.

The deliveries of the same identifier to a receiver are processed one at a time, even on several instances of the cluster: the module holds a distributed lock from the duplicate check to the recording of the receipt. A delivery arriving meanwhile gets a 409, which the caller treats as a failure and sends again later, by then as a duplicate. The lock expires after 5 minutes when the instance processing the delivery stops without releasing it. It uses the Dapr lock store of the cluster: a cluster run without Dapr takes no lock.

Administration

The module adds two menu items to the application of each tenant: Webhook receivers and Webhook receipts.

Receiver form

A receiver is the entry point of one caller: New receiver in the list of receivers opens an empty form.

Field	Content
Key	The last segment of the address of the receiver. Lowercase letters and digits, separated by single hyphens, such as `samsara` or `fleet-events`, unique in the tenant. Changing it changes the address, so the caller must then be configured again.
Label	A free name that tells the administrators what the receiver is for.
Handler	The processing the receiver calls, chosen among the handlers registered by the business modules of the application (see Writing a handler). The list is empty when no module registers a handler.
Authentication mode	Hmac for a caller that signs its requests with a shared secret, the case of most webhook providers. Bearer for a caller that is a service account of your identity provider (see Authentication modes).
Signature scheme	Shown in Hmac mode only: the way the caller signs its requests, chosen among `StandardWebhooks` and the schemes registered by the business modules (see Hmac). Choose the scheme that matches the documentation of the caller.
Timestamp tolerance (seconds)	Shown in Hmac mode only: the largest gap accepted between the time the caller signed a request and the time it is received, 300 seconds by default. A captured request replayed later than that is refused. Raise it only when the clocks are far apart.
Active	An inactive receiver answers 404 to every request, without recording a receipt: use it to stop a caller without deleting its configuration. Deleting a receiver also deletes its secrets and its receipts.

Under the fields, the form shows the address to give to the caller: /api/v1/webhooks/public/{key} in Hmac mode, /api/v1/webhooks/in/{key} in Bearer mode. In a multitenant cluster, the address starts with the identifier of the current tenant, such as /acme/api/v1/webhooks/public/samsara. Prefix it with the address of the cluster, for example https://erp.example.com/acme/api/v1/webhooks/public/samsara.

On save, the form checks that the key has the expected format and that the handler and the signature scheme are registered.

Secrets of a receiver

The Secrets list is shown in Hmac mode only: a receiver in Hmac mode refuses every request until it has a valid secret. Adding or revoking a secret requires the Receiver secrets function, see Permissions. Add a secret saves the receiver, then opens a window. Validity of the current secret (hours) sets how long the current secret stays valid once the new one is added: 24 hours by default, from 0 to 720 hours. Then two choices are offered:

Save: the secret given by the caller. Most providers create the secret when the webhook is declared on their side, as Samsara does with a base64 secret key: paste it in the field. A secret holds at most 500 bytes, and an accented or special character takes several of them. A secret without the format of the signature scheme, such as whsec_ alone, is refused and the current secret stays unchanged.
Generate: the module creates a random secret, prefixed with whsec_ for the Standard Webhooks scheme. It is shown once in the window and is never readable afterwards: Copy puts it in the clipboard, to give it to the caller before closing the window with Done.

The list shows the secrets of the receiver, never their value. Valid until is empty for the current secret. Adding a secret while the receiver has one gives the previous secret a validity end after the number of hours entered in the window: the caller has this time to switch to the new secret, and the requests signed with either secret are accepted meanwhile. With 0 hours, the previous secret stops being valid at once, which suits a secret that may have leaked. At most two secrets are valid at once, so a third secret can only be added once the previous one has expired or has been revoked. When the current secret is revoked while the previous one is still valid, a new secret can be added at once, and the previous one keeps its validity end. Revoke deletes a secret at once: revoke the previous secret as soon as the caller uses the new one, or when it may have leaked.

Receipts

Webhook receipts lists the requests received, most recent first, with their reception time, receiver key, delivery identifier, status, response code and duration. An icon gives the status at a glance: green for Processed, red for Failed, orange for Rejected, grey for Duplicate.

The detail of a receipt shows its status, receiver and reception time at the top, then:

The error, for a rejected or failed receipt, in a red frame.
The delivery identifier, the response code and the duration, and the original receipt for a replay.
The Body tab, kept except for a rejected or duplicate receipt.
The Headers tab, without the ones that may carry a credential: the headers whose name holds auth, token, secret, key, password, credential, session, cookie, signature or hmac, and the headers named by the signature scheme of the receiver.

Replay calls the handler again with the kept body and records the result as a new receipt linked to the original one. It is disabled for a receipt without body. A replay takes the lock of its delivery identifier: while the caller's delivery of the same identifier is being processed, it is refused with a message, and can be run again a moment later.

Permissions

The functions of the module are grouped under the Webhooks function: giving it to a role gives all of them, unless a function under it has its own permission.

Function	Type	Gives access to
Receivers	CRUD	The receivers, and the list of their secrets
Receiver secrets	Allow/deny	The Add a secret and Revoke actions of a receiver
Receipt log	CRUD	The receipt log, and the `PurgeWebhookReceipts` server method
Receipt replay	Allow/deny	The Replay action of a receipt
Sending by a service account	Allow/deny	The `webhooks/in` route, for the service accounts of the Bearer callers

Purging the receipts

The receipts keep the bodies, which may hold personal data. Schedule the background server method PurgeWebhookReceipts with the Task Scheduler: it deletes the receipts received more than retentionDays days ago, by batches of 100 saved one after the other.

To create this scheduled task for every tenant, without asking the administrators to do it, register a tenant database migration interceptor in a business module that references the TaskSchedulerClient module. After each migration of a tenant, it creates the task when the tenant has none:

public class WebhookReceiptPurgeTaskInterceptor : ITenantDatabaseMigrationInterceptor
{
    private readonly IGetScheduledTasks _getScheduledTasks;
    private readonly IScheduleTask _scheduleTask;

    public WebhookReceiptPurgeTaskInterceptor(IGetScheduledTasks getScheduledTasks, IScheduleTask scheduleTask)
    {
        _getScheduledTasks = getScheduledTasks;
        _scheduleTask = scheduleTask;
    }

    public async Task<Result> OnTenantDatabaseMigratedAsync(TenantDatabaseMigratedEventData data, CancellationToken cancellationToken)
    {
        try
        {
            IPagedList<RemoteScheduledTask> purgeTasks = await _getScheduledTasks.ExecuteAsync(
                filter: "TargetServerMethodName eq 'PurgeWebhookReceipts'", top: 1, cancellationToken: cancellationToken);
            if (purgeTasks.Count == 0)
            {
                await _scheduleTask.ExecuteAsync(
                    new RemoteScheduledTask
                    {
                        Id = Guid.NewGuid(),
                        Description = "Webhook receipts purge",
                        Enabled = true,
                        ServerMethodName = "PurgeWebhookReceipts",
                        Arguments = new Dictionary<string, object> { ["retentionDays"] = 30 },
                        Trigger = new SchedulerTriggerProperties { CronStringExpression = "0 0 3 ? * *", StartAt = DateTime.UtcNow.Date.AddHours(3) },
                    },
                    cancellationToken);
            }
        }
        catch (Exception exception) when (exception is BusinessException or HttpRequestException)
        {
            // The Task Scheduler is unavailable: the next migration creates the task.
        }

        return Result.Ok();
    }
}

Register it in the Startup class of the module with services.AddTenantDatabaseMigrationInterceptor<WebhookReceiptPurgeTaskInterceptor>();. The task then runs every night at 3:00 (UTC) and keeps 30 days of receipts. An administrator can change the retention in its arguments or disable it in the scheduled tasks. A deleted task is created again at the next migration.

Limits

The body of a request is limited to 1 MB.
The module has no rate limiting: protect the public route at the ingress or firewall level when needed.
A handler is cancelled after 1 minute: start a background server method for a longer process.
Duplicates are detected by the delivery identifier only: a caller that sends no identifier or an unsigned one, or a handler that ignores its cancellationToken, can reach the handler twice for the same event. A handler that must never process an event twice also checks its own data.

Table of Contents