Monitoring
Metrics
Cluster backend server Labels
Neos cluster backend servers expose the following labels for their metrics :
| Name | Description | Type | Scope |
|---|---|---|---|
| neos_fmk_version | Neos framework version used by the cluster | string | Global |
| neos_cluster_name | Name of the cluster | string | Global |
| neos_cluster_version | Version of the cluster | string | Global |
| neos_cluster_multitenancy_enabled | Value indicating whether the cluster is running in multitenancy mode | boolean | Global |
| neos_tenant | Tenant identifier of an http request related metric | string | Http request metrics |
Cluster referenced modules
Each backend server of a cluster exposes the neos_cluster_referenced_module (Gauge) with the following labels (in addition to the global labels) :
| Name | Description | Type | Scope |
|---|---|---|---|
| neos_cluster_referenced_module_name | Name of the referenced module | string | neos_cluster_referenced_module |
| neos_cluster_referenced_module_version | Version of the referenced module | string | neos_cluster_referenced_module |
Database connection pools
Each backend server and task runner of a cluster exposes the state of its database connection pools. The metrics are normalized by Neos rather than taken from the database provider, so the series identity does not change when a provider changes its own instruments.
Background server methods (BackgroundExecution: true) run in the task runner process, not in the backend server. Each process scrapes separately, so a workload started from the UI shows up on the task runner's own /metrics endpoint.
| Metric | Type | Unit | Semantics |
|---|---|---|---|
| neos_db_pool_connections | Gauge | connections | Current connection count, split by connection_state |
| neos_db_pool_max_connections | Gauge | connections | Currently configured maximum, summed over the pools of the provider. Absent when the provider cannot report it |
These metrics use the following labels, in addition to the global labels:
| Name | Description | Type | Values |
|---|---|---|---|
| neos_db_provider | Neos provider identity, and the only provider label | string | PostgreSQL, SQLServer |
| connection_state | Split of the connection count | string | active, idle |
connection_state applies to neos_db_pool_connections only.
An absent series is not a zero. When a provider cannot report a value, the metric is absent rather than published as zero. Alerting on an absent series is correct; treating it as zero is not.
Each provider exposes one provider-wide aggregate for the process, never one series per pool. The provider capabilities are static properties, so they are stated here rather than exported as runtime metrics. Read more about pool and process scope.
| neos_db_provider | Native provider reporting scope | neos_db_pool_connections | neos_db_pool_max_connections |
|---|---|---|---|
PostgreSQL, through Npgsql |
Individual pools, aggregated by Neos | yes | yes |
SQLServer, through SqlClient |
Process-wide counters | yes | never |
A query can still find the providers reporting counts without a maximum:
neos_db_pool_connections{connection_state="active"}
unless ignoring(connection_state) neos_db_pool_max_connections
Cardinality: one series per provider, never per pool
The counts are provider-wide aggregates. A process holding several pools — one per tenant database in a multitenant cluster — reports their sum, and neos_db_pool_max_connections is likewise the sum of their configured maximums.
No pool identity is exported. A per-pool label would be a per-tenant label, which multiplies the series count by the tenant count, and the provider-supplied pool name is unusable as a label anyway: Npgsql names a data source built without an explicit name after its connection string. Correlating a saturated backend with a specific database stays an operator step against the cluster configuration and the DBA tooling.
Deriving ratios and peaks
Neos does not export a saturation ratio or a high-water mark. Both are computed by the time-series database, which can do it over any window and can aggregate across processes — neither of which is possible once a ratio has been collapsed into a single exported number.
# Current saturation.
neos_db_pool_connections{connection_state="active"}
/ ignoring(connection_state) neos_db_pool_max_connections
# Highest active count over the last hour.
max_over_time(neos_db_pool_connections{connection_state="active"}[1h])
# Highest saturation over the last ten minutes.
max_over_time(
(neos_db_pool_connections{connection_state="active"}
/ ignoring(connection_state) neos_db_pool_max_connections)[10m:]
)
An exhaustion alert takes the same shape:
- alert: NeosDbPoolNearExhaustion
expr: |
neos_db_pool_connections{connection_state="active"}
/ ignoring(connection_state) neos_db_pool_max_connections > 0.8
for: 5m
Note that a pool running at full occupancy is not necessarily a pool in trouble — it may simply be busy. Pending acquisitions and acquisition timeouts are the signals that distinguish the two, and they are not part of this contract yet.
Provider metrics are not exposed
Neos restricts the prometheus-net meter bridge to an allow-list of meters, so the raw instruments published by database providers — for example Npgsql's db.client.connection.* — do not appear on /metrics. This is deliberate and load-bearing: those instruments are labelled with the provider's own data-source name, which in a multitenant cluster is derived from the tenant, and republishing them would both expose tenant identifiers and multiply the series count by the tenant count.
Adding a meter to that allow-list is a deliberate decision to publish its series and to own its label shape. It is not a configuration setting.
Gateway unavailable tenants
The gateway exposes the neos_unavailable_tenant (Gauge) to represent tenant blocking state enforced by the unavailable-tenants protection in tenant resolution.
This metric uses the following labels:
| Name | Description | Type | Values / examples |
|---|---|---|---|
| cluster_name | Cluster name for which tenant blocking is tracked | string | manager, technicaldemos |
| tenant_identifier | Tenant identifier currently tracked | string | TDX, NW1 |
| data_persistence_state | Current data persistence state from TM snapshot | string | Running, Stopped, Error |
| tenant_status | Tenant status from TM snapshot | string | Active, Inactive |
Interpretation:
1: the tenant is currently blocked by the gateway for the given cluster/state label set.0: the tenant is not currently blocked for the given cluster/state label set.
This metric is useful to diagnose unexpected 503 responses returned by gateway unavailable-tenants protection.
If Redis pub/sub synchronization is degraded, gateway continues cache convergence through fallback periodic pull refresh from Tenant Management.
A tenant_status of Inactive with data_persistence_state of Running is expected to be emitted as unavailable by Tenant Management.
Reporting
The reporting server exposes the neos_report_pdf_generation_duration_seconds (Histogram) measuring the duration of each pdf generation from the reporting server with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_tenant | Tenant identifier of an http request related metric | string |
| neos_fmk_version | Neos framework version used by the cluster | string |
| neos_report_name | The name of the generated report | string |
The reporting server exposes the neos_report_viewer_generation_duration_seconds (Histogram) measuring the duration of each report generation displayed in a viewer from the reporting server with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_tenant | Tenant identifier of an http request related metric | string |
| neos_fmk_version | Neos framework version used by the cluster | string |
| neos_report_name | The name of the generated report | string |
Tenants state stores
The tenant-management backend exposes the neos_tenants_state_store_rebuild_in_progress (Gauge) metric indicating whether a tenants state store rebuild is currently in progress.
The value is binary:
1: a rebuild is currently running (lock acquired).0: no rebuild is currently running.
This metric is exposed with the following labels:
| Name | Description | Type |
|---|---|---|
| store | Name of the tenants state store | string |
The store label can be one of the following values:
Tenants_ClusterVersionsByTenantTenants_ClustersByUserTenants_ResolutionByTenant
The tenant-management backend also exposes the neos_tenants_state_store_rebuild_duration_seconds (Histogram) metric measuring the duration of tenants state store rebuild executions.
This metric is exposed with the following labels:
| Name | Description | Type |
|---|---|---|
| store | Name of the tenants state store | string |
Tenants accessor
The tenants accessor components expose metrics for inter-service communication requiring tenant resolution. The following metrics are exposed:
The neos_requested_tenant_cache_type label indicates the type of cache source used to retrieve tenant data, which can be one of the following values: local-cache (in-memory local cache), state-store (distributed state store), or database (Tenant Management backend service invocation).
The neos_tenants_accessor_get_clusters_duration_seconds (Histogram) measures the duration of GetClustersAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_tenant_cache_type | Type of cache used for tenant cluster retrieval | string |
The neos_tenants_accessor_get_clusters_calls_total (Counter) counts total GetClustersAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_tenant | Tenant identifier | string |
| neos_requested_tenant_cache_type | Type of cache used for tenant cluster retrieval | string |
The neos_tenants_accessor_get_authorized_tenants_duration_seconds (Histogram) measures the duration of GetAuthorizedTenantsAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_tenant_cache_type | Type of cache used for authorized tenants retrieval | string |
The neos_tenants_accessor_get_authorized_tenants_calls_total (Counter) counts total GetAuthorizedTenantsAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_cluster_name | Name of the requested cluster | string |
| neos_requested_tenant_cache_type | Type of cache used for authorized tenants retrieval | string |
The neos_tenants_accessor_get_tenant_info_duration_seconds (Histogram) measures the duration of GetTenantInfoAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_tenant_cache_type | Type of cache used for tenant info retrieval | string |
The neos_tenants_accessor_get_tenant_info_calls_total (Counter) counts total GetTenantInfoAsync calls with the following labels :
| Name | Description | Type |
|---|---|---|
| neos_requested_tenant | Tenant identifier | string |
| neos_requested_cluster_name | Name of the requested cluster | string |
| neos_requested_cluster_version | Version of the requested cluster | string |
| neos_requested_tenant_cache_type | Type of cache used for tenant info retrieval | string |
Tenant Management
The tenant management backend service exposes metrics related to cache invalidation and rebuild operations.
The neos_tenant_cache_rebuild_requests_total (Counter) counts the total number of tenant cache rebuild requests with the following labels :
| Name | Description | Type | Values |
|---|---|---|---|
| neos_tenant_cache_rebuild_service | Service handling the rebuild request | string | taskrunner, backend |
| neos_tenant_cache_rebuild_scope | Scope of the rebuild operation | string | full, partial |
Service breakdown:
taskrunner: Cache rebuild executed asynchronously via the task runner servicebackend: Cache rebuild executed synchronously in the current backend process
Scope breakdown:
full: Complete cache invalidation of all tenant-related stores (resolution, clusters, users)partial: Partial cache invalidation affecting only specific tenants or users