Table of Contents

Intercluster communication architecture : State store for tenants

StateStoreTenants.svg

Tenant resolution state store

This state store contains an entry by tenant. The value contains the information needed to resolve the database connection for the tenant for all the clusters that the tenant is part of.

The database connection information contains the database type and the connection string. Database type is an integer that represents the type of the database (PostgreSQL, SQL Server, etc.).

The tenantId is the internal identifier of the tenant referenced in all multi-tenant tables. The tenant identifier is the unique identifier of the tenant.

The tenant resolution state store contains some others information like the license id, the input languages, the tenant name, the cluster name and the cluster version.

Creation and update

Created by Tenant Management cluster when started and updated when a tenant is added or removed from a cluster or when the database connection information is updated.

Use by who

Used by the business cluster to resolve the database connection information for a tenant. The business cluster reads the state store when a tenant is requested to obtain the database connection information and the tenant internal id.

State store Key

The key is Tenants_ResolutionByTenant_[Tenant_Identifier] where Tenant_Identifier is the identifier of the tenant.

State store Value

Example of the value for the tenant Acme (key = Tenants_ResolutionByTenant_(Acme)):

[
  {
    "tenantInfo": {
      "databaseType": 1,
      "connectionString": "Server=localhost;Port=5432;Database=TechnicalDemos;User Id=*******;Password=********;",
      "licenseId": null,
      "inputLanguages": [],
      "id": "13",
      "identifier": "Acme",
      "name": "Acme Corp."
    },
    "clusterName": "TechnicalDemos",
    "clusterVersion": "0.0.1"
  },
  {
    "tenantInfo": {
      "databaseType": 1,
      "connectionString": "Server=localhost;Port=5432;Database=Northwind;User Id=*******;Password=*******;\n",
      "licenseId": null,
      "inputLanguages": [],
      "id": "13",
      "identifier": "Acme",
      "name": "Acme Corp."
    },
    "clusterVersion": "0.0.1",
    "clusterName": "Northwind"
  }
]

State store persistence

This state store does not need to be persisted. It is created when the Tenant Management is started. If the state store is lost the business cluster will invoke the Tenant Management to get the information. Meanwhile the Tenant Management will recreate the state store.

A special key named Tenants_ResolutionByTenant_completed is used to check if the state store is valid. If the key is not found, the business cluster will call the Tenant Management to rebuild the state store.

Flow diagram

sequenceDiagram
    participant TM as Tenant Management
    participant TR as State Store
    participant BC as Business Cluster

    Note left of TM: Start
    TM->>TR: Build state store
    BC->>TR: Get tenant resolution
    alt valid store
        TR->>BC: return tenant info
    else invalid store
        BC->>+TM: Call methods/GetTenantInfo
        Note left of TM: Rebuild state store
        TM->>-BC: return tenant info
    end

Tenant clusters and versions state store

This state store contains an entry by tenant. The value contains the cluster on which the tenant is part of. It provides the cluster version associated with the tenant. It can only have one version of a cluster for a tenant, the tenant is redirected to this cluster version after the authentication.

Creation and update

Created by the Tenant Management when the cluster is started and updated when a tenant is added or removed from a cluster or when the tenant is migrated to a new version.

State store Key

The key is Tenants_ClusterVersionsByTenant_[Tenant_Identifier] where Tenant_Identifier is the identifier of the tenant.

State store Value

Example of the value for the tenant Acme (key = Tenants_ClusterVersionsByTenant_(Acme)):

[
  {
    "clusterName": "TechnicalDemos",
    "clusterVersion": "0.0.1"
  },
  {
    "clusterName": "Northwind",
    "clusterVersion": "0.0.1"
  }
]

State store persistence

This state store does not need to be persisted. It is created when the Tenant Management is started. If the state store is lost the businessCluster will invoke the Tenant Management to get the information. Meanwhile the Tenant Management will recreate the state store.

A special key named Tenants_ClusterVersionsByTenant_completed is used to check if the state store is valid. If the key is not found, the business cluster will call the Tenant Management to rebuild the state store.

Use by who

This state store is used by the business cluster when a remote invocation is made to another cluster to get the cluster version.
It is also used by the business cluster to check if the cluster is the receiver of a pub/sub message according to the cluster name, the cluster version and the tenant identifier information in the message.
Finally, It is used by the userpermission module to allow to assign role to a user on all its clusters.

Flow diagram

sequenceDiagram
    participant TM as Tenant Management
    participant ST as State Store
    participant BC as Business Cluster

    Note left of TM: Start
    TM->>ST: Build state store

    rect rgb(191, 223, 255)
        note right of ST: Remote cluster invocation
        BC->>ST: Get tenant clusters
        alt valid store
            ST->>BC: return tenant clusters versions
        else invalid store OK
            BC->>+TM: Call methods/GetClusterVersions
            Note left of TM: Rebuild state store
            TM->>-BC: return tenant clusters versions
        end
    end

    rect rgb(191, 255, 244)
        note right of ST: Endpoint Pub/Sub : Receive message
        BC->>ST: Get tenant clusters
        alt valid store
            ST->>BC:return tenant clusters versions
        else invalid store
            BC->>+TM: Call methods/GetClusterVersions
            Note left of TM: Rebuild state store
            TM->>-BC: return tenant clusters versions
        end
    end

User clusters state store

This state store contains an entry by user. The value contains the tenants on which the user is part of and for each tenant the clusters on which the user is part of.

Creation and update

Created by the Tenant Management when the cluster is started and updated when a user is added or removed from a tenant or when the user is migrated to a new version.

State store Key

The key is Tenants_ClustersByUser_[User_Identifier] where User_Identifier is the identifier of the user.

State store Value

Example of the value for the user John (key = Tenants_ClustersByUser_(John)):

 [
      {
        "clusters": [
          {
            "clusterName": "TechnicalDemos",
            "clusterVersion": "0.0.1"
          },
          {
            "clusterName": "Northwind",
            "clusterVersion": "0.0.1"
          },
        ],
        "identifier": "Acme",
        "name": "Acme Corp."
      }
    ],

State store persistence

This state store does not need to be persisted. It is created when the Tenant Management is started. If the state store is lost the business cluster will invoke the Tenant Management to get the information. Meanwhile the Tenant Management will recreate the state store.

A special key named Tenants_ClustersByUser_completed is used to check if the state store is valid. If the key is not found, the business cluster will call the Tenant Management to rebuild the state store.

Use by who

This state store is used by the gateway to get the tenants allowed for a user. In fact, when a user is authenticated, if the user is part of several tenants, the user is redirected to the tenant selection page built with the information of this state store.

Inactive tenants are intentionally kept in this authorized-tenants store. Availability is enforced separately by the gateway unavailable-tenants cache, so an inactive tenant can still appear in authorized tenants but will be blocked at request time with a 503.

Flow diagram

sequenceDiagram
    participant TM as Tenant Management
    participant TR as State Store
    participant GT as Gateway

    Note left of TM: Start
    TM->>TR: Build state store
    GT->>TR: Get authorized tenants for user
    alt valid store
        TR->>GT: return authorized tenants for user
    else invalid store
        GT->>+TM: Call methods/GetAuthorizedTenants
        Note left of TM: Rebuild state store
        TM->>-GT: return authorized tenants for user
    end

Tenant visual identity state store

This state store publishes the tenant visual identity. It contains a manifest by tenant and cluster, and the CSS bodies referenced by the manifests. It also contains every style on its own, under its identifier, so that the code of a cluster can apply any style by its name (see Choose the style from the cluster code). The images of the styles are not stored here.

A CSS body is the published CSS of one color scheme of a style: the generated CSS of the theme of that slot, a line break, then the free CSS of the slot. A part that does not exist is skipped, together with the line break, so a style without a theme publishes exactly its free CSS.

Creation and update

Created by Tenant Management when it starts (unless DistributedStores:RebuildOnStart is false) and by the RebuildTenantStyles server method. The manifest of a tenant and cluster is republished each time an assignment of a style to the tenant is created, modified or deleted, each time the default style of the tenant changes, each time a style that the tenant uses or one of its images is modified, and each time a theme or a CSS that such a style uses is modified, because the body of the style then changes. A CSS body is written before the manifest that references it, and a body that no manifest references any more is given a time to live of one day after the new manifest is saved, instead of being deleted, because the sessions that are open still hold its hash and can ask for the stylesheet of a color scheme that they have not loaded yet. The cluster that serves a hash reads the body first, then it only asks Tenant Management when the manifest lists the hash.

Every style is also published on its own, whether a tenant uses it or not, each time it is created, modified or deleted, and each time its theme, its CSS or one of its images changes. A full rebuild publishes them once the store is completed, so their number never delays the styles of the tenants: a style that cannot be published is logged, the others are published, and the marker TenantStyles:IdentitiesPublished is only written when all of them are. Its manifest and its bodies are stored under its identifier, and a key gives the identifier of its name, so a rename only moves that key. The key of a name is always written from the database, under a distributed lock named TenantStyles:PublishIdentityName:({name_hash}): it gets the identifier of the style that has this name at that moment, or 0 with a time to live of one day when no style has it. The key is saved under the etag read before the database, so a write that read an older owner, after its lock expired, fails and reads the database again. Two styles that exchange a name, or a publication that resumes late, therefore never leave a name to the wrong style. The bodies that the manifest of a style no longer lists get the same time to live of one day.

Use by who

Used by the business clusters. The on-authenticated call reads the manifest to tell the client which hashes to load, and the stylesheet route reads the body of a hash. When a tenant resolved interceptor chooses a style by its name, the on-authenticated call reads the key of the name, then the manifest of that style. The cluster never writes this store.

State store Key

The keys are lowercase, and the colon separates the levels so that a Redis browser groups them in a tree. The tenant visual identity article explains how to check them:

  • TenantStyles:Manifest:({tenant_identifier})_({cluster_name}): the manifest of a tenant for a cluster.
  • TenantStyles:Body:({tenant_identifier})_({cluster_name})_({hash}): a CSS body, the theme CSS followed by the free CSS of a slot, where hash is the SHA-256 of its UTF-8 bytes, as 64 lowercase hexadecimal characters. The same CSS used for two clusters of a tenant is stored once for each of them, so that the cleanup of one manifest never expires a body that another manifest lists.
  • TenantStyles:Index: the tenants and clusters whose manifest was published, in lowercase. Only Tenant Management reads and writes it, the clusters never do. The state store cannot list its keys, so a full rebuild uses this index to find the manifests that the database no longer has a style for, replaces them by empty manifests and gives their bodies a limited life.
  • TenantStyles:completed: the marker that tells the store is valid.
  • TenantStyles:Identity:({id}): the manifest of the style whose identifier is id, for every tenant and cluster.
  • TenantStyles:IdentityBody:({id})_({hash}): a CSS body of the style id.
  • TenantStyles:IdentityName:({name_hash}): the identifier of the style whose exact name, compared with its case, has the SHA-256 name_hash of its UTF-8 bytes. The name is hashed because it can hold any character.
  • TenantStyles:IdentitiesPublished: the marker that tells every style was published on its own by the last rebuild. Its value is a date. A rebuild removes it at its start and writes it again once every style is published. A publication of a style that fails after a save removes it too, until the next rebuild, and a rebuild during which such a failure happens does not write it. A cluster only trusts the keys of the names and the manifests of the styles when the store is completed and this marker exists: it then answers at once that a name without key, or released, is unknown. Otherwise it asks Tenant Management.
  • TenantStyles:IdentityPublicationFailure: a value that each failed publication of a style changes. Only Tenant Management reads and writes it, to know whether a publication failed while a rebuild ran. Without it, for example when a style could not be published or when the store was built by an earlier version of Tenant Management, the cluster asks Tenant Management.
  • TenantStyles:IdentityIndex: the identifiers of the styles published on their own. Only Tenant Management reads and writes it, so that a full rebuild finds the styles that the database no longer has and removes them.

State store Value

Example of the value for the tenant Acme and the cluster Northwind (key = TenantStyles:Manifest:(acme)_(northwind)):

{
  "versions": [
    {
      "version": "1.0.0",
      "lightHash": "<SHA-256 of the light body>",
      "darkHash": "<SHA-256 of the dark body, or null>",
      "isDefault": false,
      "styleId": 12,
      "styleName": "Acme",
      "assets": [
        {
          "name": "logo.png",
          "hash": "<SHA-256 of the image>",
          "mimeType": "image/png"
        }
      ]
    }
  ],
  "hasStyleNames": true
}

The body behind lightHash or darkHash is the published CSS of the slot, theme CSS first, then free CSS, so the contract of the manifest is the same with or without themes. darkHash is null when the dark slot has neither a theme nor a CSS: a cluster then uses the light hash in the dark mode. isDefault is true for the entry that holds the default style of the tenant, published under the version 0.0.0: a cluster takes it when no version of the manifest is lower than or equal to its own, and it is told by this flag, never by its version, so that a real style of the version 0.0.0 is not taken for it. styleId and styleName tell the style of the entry, which is the name that a tenant resolved interceptor reads in VisualIdentity. They are null for an assignment without style when the tenant has no default style. hasStyleNames is false in a manifest published by a version of Tenant Management that did not publish the names: a cluster then asks Tenant Management with GetTenantStyle, which gives the name too, until the store is rebuilt.

The manifest of a style on its own (key = TenantStyles:Identity:(12)) has the same hashes, without versions:

{
  "id": 12,
  "name": "Acme",
  "lightHash": "<SHA-256 of the light body>",
  "darkHash": "<SHA-256 of the dark body, or null>",
  "assets": [
    {
      "name": "logo.png",
      "hash": "<SHA-256 of the image>",
      "mimeType": "image/png"
    }
  ]
}

The contract is additive: a field is never removed or renamed, because Tenant Management is always the greatest version deployed and the clusters read the store in any version.

State store persistence

This state store does not need to be persisted. If the key TenantStyles_completed is not found, the business cluster calls the GetTenantStyle, GetTenantStyleBody and GetTenantStyleAsset methods of Tenant Management instead of reading the manifest. For a CSS body it still reads the key of the body first, because the key is the hash of the content and the database no longer finds the hash of a body that is only kept for the open sessions. A tenant without a style has no manifest, or an empty one (no version) when its last style was removed: a cluster reads both as "no style". A cluster answers at once that a hash is unknown when it has no body for it and its manifest of the tenant does not list it. A cluster that does not find a body that its manifest lists, in a completed store, asks Tenant Management for it, with its cluster name: Tenant Management only searches the styles that apply to that cluster. Tenant Management publishes the manifest and the bodies of a tenant and a cluster one publication at a time, under a distributed lock named TenantStyles:Publish:({tenant_identifier})_({cluster_name}). For a style chosen by its name, the cluster calls GetVisualIdentity, GetVisualIdentityBody and GetVisualIdentityAsset in the same way. It also calls them in a completed store that has no key for the name or no manifest for the style, because a store built by an earlier version of Tenant Management has none. A key of a name that holds 0 is a released name, read as an unknown name without calling Tenant Management. When the style of a key has no manifest yet, or a manifest under another name, the cluster asks Tenant Management too, because the key is written from the database and can precede the publication of the style. Tenant Management publishes each style one publication at a time, under a distributed lock named TenantStyles:PublishIdentity:({id}).

Flow diagram

sequenceDiagram
    participant TM as Tenant Management
    participant TS as State Store
    participant BC as Business Cluster

    Note left of TM: Start or style saved
    TM->>TS: Write the bodies, then the manifest
    BC->>TS: Get the manifest of the tenant
    alt valid store
        TS->>BC: return the manifest
    else invalid store
        BC->>+TM: Call methods/GetTenantStyle
        TM->>-BC: return the hashes
    end