Table of Contents

Tenant Resolution Caching

Overview

Tenant resolution uses two cache layers that work together:

  1. A distributed cache shared by all replicas (main source)
  2. A short-lived local memory cache (temporary fallback)

For framework users, this mechanism is transparent: tenant lookup APIs keep working the same way whether data comes from the distributed cache or the fallback cache.

The main objective is to keep tenant resolution fast and available during startup, rolling deployments, and cache rebuild phases.

Functional Behavior

Normal Behavior

In steady state, tenant data is read from distributed caches.

These caches cover three functional views of tenant data:

Cache Functional role
Tenants_ResolutionByTenant Resolve tenant technical information (connection, environment, metadata)
Tenants_ClusterVersionsByTenant Resolve cluster/version assignment for a tenant
Tenants_ClustersByUser Resolve which tenants a user can access by cluster

During Rebuild Windows

When distributed caches are not ready yet, the system temporarily falls back to local memory cache to keep requests responsive.

Read priority is:

  1. Distributed cache (when ready)
  2. Local fallback cache (while distributed cache is rebuilding)
  3. Remote tenant-management call (when needed)

This means users may continue to resolve tenants while rebuild is ongoing, without changing calling code.

Startup and Self-Healing

At startup, caches are rebuilt in the background with distributed coordination so only one replica rebuilds the same store at a time.

If a read path detects an incomplete distributed cache state, the system can request a global rebuild automatically (self-heal behavior).

Rebuild Triggers

The following operations can refresh tenant-resolution caches:

Trigger Tenants_ResolutionByTenant Tenants_ClusterVersionsByTenant Tenants_ClustersByUser
Platform startup Full Full Full
Read-path self-heal Full Full Full
Manual full rebuild Full Full Full
POST /api/v1/methods/resetalltenantcache Full Full Full
Manual rebuild by tenant IDs Partial Partial —
POST /api/v1/methods/resettenantcache (tenantId) Partial Partial —
Manual rebuild by user logins — — Partial
Tenant changed Partial Partial Partial
Data persistence changed Partial Partial Partial
Database server changed Partial Partial —
User-tenant assignment changed Partial Partial Partial
Cluster version changed Full Full Full
POST /api/v1/methods/changedatabaseclusterversion Full Full Full
POST /api/v1/methods/setdatabaseclusterversion Full Full Full
Additional tenant metadata changed Full — —
Additional tenant property changed Partial — —

Definition reminder:

  • Full rebuild: rebuilds the full store content.
  • Partial rebuild: updates only impacted tenants or users.

Execution Modes for Full Rebuild

Full rebuild can run in two execution modes:

  • Background mode: rebuild is enqueued and completed asynchronously.
  • In-process mode: rebuild executes synchronously in the current flow.

From a functional standpoint:

  • Background mode favors availability and shorter request latency.
  • In-process mode is used when the caller needs immediate confirmation of rebuild outcome.

When lock contention occurs during rebuild, one store can succeed while another is postponed. This is handled as a partial completion scenario. Critical operations can enforce stricter behavior by failing the operation if rebuild completeness is required before continuing.

Monitoring

Use method GET /api/v1/methods/tenantscacherebuildingstatus to monitor rebuild progress.

Note

This method requires the TenantAdministration permission.

It returns three booleans:

Property Meaning
IsTenantResolutionCacheRebuilding Resolution cache is rebuilding
IsTenantClustersCacheRebuilding Cluster/version cache is rebuilding
IsTenantUsersCacheRebuilding User access cache is rebuilding

Typical operational use cases:

  • Validate startup readiness
  • Diagnose slow rebuilds
  • Explain temporary fallback-cache usage

Configuration

{
  "TenantsResolutionCacheTtlInMinutes": 5,
  "TenantsResolutionCacheFullRebuildUseTaskRunnerByDefault": true,
  "TenantsResolutionCachePartialRebuildUseTaskRunnerByDefault": false,
  "DistributedStores": {
    "PartialRebuildLockWaitTimeout": 30
  }
}
Setting Default Functional impact
TenantsResolutionCacheTtlInMinutes 5 Local fallback cache lifetime
TenantsResolutionCacheFullRebuildUseTaskRunnerByDefault true Default mode for full rebuilds
TenantsResolutionCachePartialRebuildUseTaskRunnerByDefault false Default mode for partial rebuilds
DistributedStores:PartialRebuildLockWaitTimeout 30 How many seconds a partial rebuild waits for a store lock held by a concurrent full rebuild before degrading (0 = no wait, single attempt)
Tip

Higher TTL values reduce remote calls during long rebuild windows. Lower TTL values favor fresher local entries but may increase remote calls during rebuild.

What Framework Users Should Expect

  • Tenant resolution remains available during cache rebuild phases.
  • Most operations do not require manual intervention.
  • Cache rebuilds are normally transparent, but can be monitored when needed.
  • Some admin or migration operations may enforce stronger consistency and can fail fast if rebuild cannot be completed safely.

Troubleshooting

Rebuild seems stuck

  1. Query GET /api/v1/methods/tenantscacherebuildingstatus.
  2. Identify which cache remains in rebuilding state.
  3. Check distributed store lock settings and logs.
  4. Confirm the rebuild eventually completes after lock release.

Troubleshooting technique (state store and framework internals)

Administrators can investigate and recover directly from the distributed state store.

Checks to perform quickly:

  1. Verify completion markers exist for each store after rebuild:
  • Tenants_ResolutionByTenant_completed
  • Tenants_ClusterVersionsByTenant_completed
  • Tenants_ClustersByUser_completed
  1. Verify no stale rebuilding marker remains set unexpectedly:
  • Tenants_ResolutionByTenant_rebuilding
  • Tenants_ClusterVersionsByTenant_rebuilding
  • Tenants_ClustersByUser_rebuilding
  1. Verify the global rebuild marker is coherent with current state:
  • Tenants_GlobalRebuilding
  1. Check lock duration configuration for each distributed store (DistributedStores:{StoreName}MaxLockDuration, default 120s).
  2. Inspect Tenant Management logs for lock-acquire failures and partial rebuild completion warnings.

Fast recovery actions available to administrators:

  1. If a stale rebuilding marker remains after a crash/interruption, remove the stale key in state store.
  2. If completion markers are missing while stores are expected to be ready, trigger a manual full rebuild.
  3. If repeated lock contention occurs during version operations, re-run the operation during a quieter window.
Warning

Altering state store keys is an administrative operation. Apply only after checking logs and current rebuild status, and prefer forcing a clean full rebuild when in doubt.

API methods tied to database version operations

Two exposed APIs trigger full tenant cache rebuilds and are relevant for diagnostics:

  • POST /api/v1/methods/changedatabaseclusterversion
  • POST /api/v1/methods/setdatabaseclusterversion

Operational note:

  • setdatabaseclusterversion runs the full rebuild synchronously and waits for each store lock instead of failing immediately when another rebuild (for example a replica startup rebuild) already holds it. The wait is bounded by the buildStoreTimeoutInSeconds request parameter (see below); if a lock is still unavailable after that wait, the call fails with a business error so the caller can retry.
  • changedatabaseclusterversion also triggers full rebuild but runs it asynchronously (task runner) and does not wait on store locks.

Synchronous rebuild lock wait (setdatabaseclusterversion)

setdatabaseclusterversion forces a synchronous full rebuild so tenant caches reflect the new cluster version before the call returns. To avoid failing immediately when a concurrent rebuild already holds a store lock (typically a replica that just restarted and rebuilds caches at startup, right after a deployment), the rebuild now waits for each store lock up to a bounded budget instead of attempting it once and giving up.

Request parameter:

  • buildStoreTimeoutInSeconds: maximum time to wait for each full store lock.
    • > 0: wait up to this duration per store (WaitForLockAsync), then rebuild while holding the lock.
    • 0: no wait — a single non-blocking attempt (TryLockAsync), preserving the historical behavior.
    • < 0: rejected with a business error before anything is persisted.

If a lock is still unavailable once the wait elapses, the call fails with a business error (HTTP 400) and the caller can retry. The three full stores are rebuilt sequentially, so the overall request duration is bounded by (buildStoreTimeoutInSeconds × number of full stores) + execution margin.

Setting Default Functional impact
DistributedStores:SynchronousRebuildExecutionMargin 360 Extra time, added on top of the per-store lock waits, allowed for the rebuild execution itself before the overall request is cancelled. Falls back to DistributedStores:GlobalRebuildMaxLockDuration when unset.
Note

For full rebuilds, only the forced synchronous path (setdatabaseclusterversion) waits on locks. Startup, read-path self-heal, asynchronous, and changedatabaseclusterversion full rebuilds keep the non-blocking behavior: a store is simply skipped when its lock is held by another replica. Partial rebuilds have their own bounded wait (see below). While a store is rebuilding, its *_rebuilding / *_completed markers are updated only by the replica that holds the lock.

Partial rebuild lock wait (save flows)

Saving a tenant, a database server, a data persistence, or a user-tenant assignment triggers a partial rebuild of the impacted stores — synchronously, inside the save request, when TenantsResolutionCachePartialRebuildUseTaskRunnerByDefault is false (the default).

A partial rebuild never takes the store lock itself: it only refuses to write while a full rebuild holds it. A full rebuild dispatched just before (for example by a cluster or database creation, or a replica startup rebuild) is therefore a normal transient state. Instead of failing the save immediately in that case, the partial rebuild retries once per second until the lock is released, up to DistributedStores:PartialRebuildLockWaitTimeout seconds (default 30; 0 disables the wait and restores the historical single-attempt behavior).

  • If the lock is released within the budget, the save completes normally — the only visible effect is the added wait.
  • If the budget elapses while the lock is still held, the behavior is unchanged from before: the store is reported as FailedToAcquireLock and the synchronous save fails with a business error (HTTP 400) so the caller can retry, since committing without updating the cache would leave the just-saved data unresolvable.
  • Each impacted store has its own budget, and cancelling the request interrupts the wait immediately.

Manual cache reset endpoints

Two API methods are available for direct cache reset operations:

  • POST /api/v1/methods/resetalltenantcache
  • POST /api/v1/methods/resettenantcache (with tenantId)

resetalltenantcache accepts forceUsingBackend (bool, default false):

  • In UI flows, the default behavior goes through task runner (forceUsingBackend = false).
  • For urgent troubleshooting, administrators can force in-process backend execution with forceUsingBackend = true to trigger immediate full rebuild in the current request flow.

Practical recommendation:

  • Start with task runner mode for normal operations.
  • Use backend-forced mode only when fast, synchronous rebuild feedback is required for incident resolution.
  • In Tenant Management UI, the ResetAllCache action currently also triggers an explicit RepublishUnavailableTenantsSnapshots call after reset completion.

Unavailable-tenants republication after reset

Tenant resolution cache rebuild and unavailable-tenants synchronization are operationally linked.

When these endpoints are used:

  • POST /api/v1/methods/resetalltenantcache
  • POST /api/v1/methods/resettenantcache

Tenant Management republishes unavailable-tenants snapshots after rebuild execution.

This keeps gateway blocking state aligned with rebuilt tenant-resolution data.

Unavailable snapshots include tenants whose data persistence is not Running, and also tenants whose TenantStatus is Inactive even if their persistence is still Running.

Manual republish endpoint

For explicit operator control, Tenant Management exposes:

  • POST /api/v1/methods/republishunavailabletenantssnapshots

Use this endpoint when:

  1. You need immediate re-synchronization without running a full cache reset.
  2. You suspect pub/sub message loss or transient Redis disruption.
  3. You want to validate end-to-end snapshot propagation during incident handling.

Operational note:

  • Gateway unavailable-tenants synchronization uses Redis pub/sub in normal mode and automatically falls back to periodic pull refresh from Tenant Management when pub/sub synchronization is unavailable.

If gateway returns a 503 page for blocked tenants, note that this page can be customized in production. See Custom status code pages in production.

Operational runbook: cache rebuild and snapshot re-sync

  1. Trigger resetalltenantcache (or resettenantcache for targeted impact).
  2. Wait until tenantscacherebuildingstatus no longer reports rebuilding for impacted stores.
  3. Check gateway metric neos_unavailable_tenant for expected tenant/cluster labels, including data_persistence_state and tenant_status.
  4. If state mismatch persists, call republishunavailabletenantssnapshots.
  5. Re-test request behavior (503 for blocked tenants, success for available tenants).

Request Flow (Functional View)

flowchart TD
    A["Tenant Resolution Request"] --> B{"Distributed Cache Ready?"}
    B -->|Yes| C["Read Distributed Cache"]
    B -->|No| D{"Local Fallback Has Valid Entry?"}
    D -->|Yes| E["Return Local Cached Value"]
    D -->|No| F["Call Tenant Management"]
    F --> G["Populate Local Fallback Cache"]
    G --> H["Return Value"]
    C --> H
    E --> H

    style A fill:#0f766e,stroke:#0d9488,color:#fff
    style B fill:#1d4ed8,stroke:#2563eb,color:#fff
    style D fill:#1d4ed8,stroke:#2563eb,color:#fff
    style C fill:#15803d,stroke:#16a34a,color:#fff
    style E fill:#15803d,stroke:#16a34a,color:#fff
    style F fill:#b45309,stroke:#d97706,color:#fff
    style G fill:#b45309,stroke:#d97706,color:#fff
    style H fill:#15803d,stroke:#16a34a,color:#fff

See Also