Table of Contents

SignalR metrics

Neos exposes metrics about the health of the SignalR notification Hub through the same Prometheus/Grafana circuit used by the rest of the framework's metrics. They cover connection churn by transport (to catch a fallback to long polling), negotiation failures by cause (for example a shared Data Protection key producing 401 responses across instances), and the size of the caches kept for the 3.1 compatibility fallback.

Where the metrics are exposed

The three groups of metrics below are not all registered on the same host: a negotiation can be rejected before it ever reaches the process that maps the Hub, and the tenant caches belong to whichever process resolves the tenant for a connection, not to the Hub's own process.

  • In Kubernetes, the Hub runs inside each backend replica of the cluster (see SignalR with a Redis backplane), and the gateway sits in front of it, authenticating every request -- including a negotiation -- before routing it there.
  • In the local development environment, ServerProxy plays both of these roles at once: it terminates SignalR connections and hosts the Hub directly, and it authenticates requests the same way the gateway does in production. The backend business-cluster app's own Hub mapping is never reached through it.
Host Endpoint What it reports
Backend business-cluster app (Kubernetes) /metrics Native Hub connection metrics, and negotiation failures for requests that reached this replica's own Hub mapping.
Gateway (Kubernetes, production) /metrics Negotiation failures rejected by the gateway's own authentication layer -- before a request ever reaches a backend replica -- and the tenant cache gauges.
ServerProxy (local development) /admin/metrics All of the above at once: native Hub connection metrics, negotiation failures (whether rejected locally or reaching the Hub), and the tenant cache gauges.
Note

ServerProxy's own metrics are placed under /admin so they are excluded from YARP forwarding -- a bare /metrics request is always forwarded to the backend, which would make a second endpoint at that same path unreachable. Locally, both endpoints exist side by side and are scraped together (see Dynamic scrape targets for running clusters).

Tip

A negotiation failure is never counted twice: the gateway only counts a rejection it produces itself, before any request is forwarded. A negotiation that reaches a backend replica is counted there instead, whatever status code the replica returns for it.

Metrics reference

Native connection metrics

Neos bridges the native ASP.NET Core SignalR instruments (meter Microsoft.AspNetCore.Http.Connections, built into the framework since .NET 8) through the same prometheus-net allow-list as the rest of the Neos metrics:

Metric Type Unit Semantics
microsoft_aspnetcore_http_connections_signalr_server_active_connections UpDownCounter connections Current number of active connections, split by signalr_transport
microsoft_aspnetcore_http_connections_signalr_server_connection_duration Histogram seconds Duration of completed connections, split by signalr_transport and signalr_connection_status

These metrics use the following labels:

Name Description Type Values / examples
signalr_transport Transport used by the connection string web_sockets, long_polling, server_sent_events
signalr_connection_status Reason the connection ended (duration metric only) string normal_closure, timeout, app_shutdown, ...

A sustained shift of signalr_transport away from web_sockets is the signal for a fallback storm (proxies, load balancers, or corporate networks blocking WebSocket upgrades).

Negotiation failures

ASP.NET Core exposes no native instrument for a negotiation failure: negotiation is handled by the connection-handling middleware, before any Hub code runs, so nothing at the Hub level can observe it. Neos fills that gap with its own meter (Neos.SignalR), fed from two places that never observe the same request: a small increment inside the gateway's authentication layer for a rejection it produces itself, and a middleware next to the Hub mapping for everything that reaches that far (including a rejection forwarded there instead, or an exception raised while negotiating).

Metric Type Unit Semantics
neos_signalr_negotiation_failures Counter {failure} Count of negotiation requests to /notifications/negotiate that did not succeed, split by neos_signalr_negotiation_cause

This metric uses the following label:

Name Description Type Values
neos_signalr_negotiation_cause Reason the negotiation request failed string unauthorized, error

unauthorized corresponds to a 401 response -- typically a Data Protection key that is not shared consistently across instances, making one instance unable to validate a token issued by another. error covers any other 4xx/5xx response.

Tenant cache sizes

The tenant accessor keeps two in-process caches for the 3.1 compatibility fallback (connections and connection tokens associated with a tenant). Their sizes are published as observable gauges, on the same Neos.SignalR meter, wherever the tenant accessor itself is hosted -- the gateway in production, ServerProxy locally; the backend business-cluster app does not host it and does not export these two gauges:

Metric Type Unit Semantics
neos_signalr_tenant_cache_connections Gauge {entry} Current number of SignalR connections associated with a tenant
neos_signalr_tenant_cache_connection_tokens Gauge {entry} Current number of connection tokens kept for the 3.1 compatibility fallback

These gauges carry no tenant dimension: only the total size of each cache is exported, consistent with the rest of the Neos SignalR metrics.

Example Grafana dashboard

An example dashboard is available at server/monitoring/grafana/neos-signalr.json. It is a starting point, expected to be adjusted by the team once a scrape job is set up for each app's own metrics endpoint. It groups panels in three rows:

  • Connexions & Transport: active connections by transport, P95 connection duration by transport, and completed connections by closure status.
  • Négociation: negotiation failure rate by cause.
  • Cache connectionToken/Tenant: current size of the two 3.1-compatibility caches.

See also