Table of Contents

OpenTelemetry collector configuration

This article explains how to configure OpenTelemetry Collector routing for Neos .NET backend processes.

For the platform flow and architecture diagram, see Observability architecture.

Important

The Neos frontend still uses the Application Insights JavaScript library. This article focuses on backend .NET telemetry routing through OpenTelemetry Collector.

Collector routing model

Neos .NET backend processes export telemetry through OTLP. Helm supports two main routing modes:

  • Internal collector mode: the chart deploys and manages the internal collector (observability.collector.mode=enabled).
  • Direct OTLP mode: workloads send telemetry directly to the endpoint defined in observability.otlp.global.endpoint, or to per-signal endpoints (observability.otlp.traces.endpoint, observability.otlp.metrics.endpoint, observability.otlp.logs.endpoint), with observability.collector.mode=disabled.

Within either mode, individual signals can be routed to dedicated endpoints using per-signal overrides (see Per-signal endpoint routing).

observability.collector.mode=auto (default) keeps legacy behavior and activates internal collector routing when legacy tracing backends are enabled (interClusterCommunication.appInsights.enabled, interClusterCommunication.zipkin.enabled, or jaeger.enabled).

For Dapr inter-cluster traces (Zipkin protocol), Helm can route sidecars to an explicit external Zipkin-compatible endpoint through observability.dapr.zipkinEndpoint. When the chart-generated collector configuration is active (no config.inline or existingConfigMap override), zipkinEndpoint is ignored for all legacy backends — including Zipkin-only mode — and Dapr is automatically routed to the internal collector Zipkin receiver at neos-otel-collector:9411. When a custom collector configuration is in use, the chart cannot guarantee the custom config exposes a Zipkin receiver; observability.dapr.zipkinEndpoint must be set explicitly when Application Insights or Jaeger is also enabled — validation fails otherwise. When only Zipkin is enabled with a custom or disabled collector, Dapr defaults to neos-zipkin:9411; setting observability.dapr.zipkinEndpoint overrides this fallback and routes to the explicit endpoint.

By default, the chart injects the OTLP endpoint, protocol, and timeout into .NET workloads both as Kubernetes pod environment variables (so the OpenTelemetry SDK picks them up via Environment.GetEnvironmentVariable()) and as entries in appsettings.json (for the Serilog sink and other IConfiguration-based reads). The Serilog OpenTelemetry sink configuration is also injected so that logs are routed to the same endpoint. Signal routing can be controlled with observability.otlp.tracesEnabled, observability.otlp.metricsEnabled, and observability.otlp.logsEnabled. Advanced OTEL SDK keys can be added through observability.otlp.additionalConfiguration (injected as pod environment variables, applies to the native OTLP SDK only — traces and metrics). Advanced Serilog sink arguments can be added through observability.otlp.additionalSerilogConfiguration (applies to Serilog.WriteTo[0].Args only — logs). Credential keys and other sensitive OTEL settings can be supplied via observability.otlp.envFromSecretName to mount a Kubernetes Secret as envFrom on all backend pods.

By default, backend telemetry instance metadata (service.instance.id) is resolved from the machine name (Environment.MachineName). In Kubernetes, this value is typically the pod name. The Neos runtime automatically injects service.instance.id as an OTLP resource attribute on every configured Serilog OpenTelemetry sink at startup, keeping instance-level correlation consistent across logs, traces, and metrics.

With the default internal collector configuration, traces are exported to the enabled trace backends. Metrics and logs are exported when Application Insights is configured or when the collector debug exporter is enabled for troubleshooting. The debug exporter is disabled by default.

Signal flags and truth table

The chart supports independent control of the three OTLP signal families:

  • observability.otlp.tracesEnabled
  • observability.otlp.metricsEnabled
  • observability.otlp.logsEnabled

Each property accepts true, false, or null.

  • true: force the signal on. This always takes priority over auto-detection, including for the generated collector — even when the generated configuration has no pipeline for that signal.
  • false: force the signal off. The chart injects OTEL_<SIGNAL>_EXPORTER=none.
  • null: auto mode. The chart derives the behavior from the active routing mode.

When observability.otlp.enabled=true, at least one effective signal must remain enabled after evaluation.

Auto mode behavior

The following table describes auto mode (null flag) behavior per routing mode. Explicit true always overrides the auto-detected value regardless of routing mode.

Routing mode tracesEnabled: null metricsEnabled: null logsEnabled: null Result
Chart-generated collector config Enabled Enabled only when Application Insights or collector debug exporter is enabled Enabled only when Application Insights or collector debug exporter is enabled Traces always routed; metrics/logs depend on generated collector pipelines
Custom collector config (config.inline / config.existingConfigMap) Enabled Disabled Disabled Safe default for traces-only custom pipelines
Direct OTLP mode (otlp.enabled=true, global.endpoint, or per-signal endpoint) Enabled Enabled Enabled All signals routed unless explicitly disabled

Effective rendering rules

Traces Metrics Logs Rendered OTEL settings Serilog OpenTelemetry sink
On On On No OTEL_TRACES_EXPORTER=none, no OTEL_METRICS_EXPORTER=none, no OTEL_LOGS_EXPORTER=none Present
On On Off OTEL_LOGS_EXPORTER=none Absent
On Off On OTEL_METRICS_EXPORTER=none Present
On Off Off OTEL_METRICS_EXPORTER=none, OTEL_LOGS_EXPORTER=none Absent
Off On On OTEL_TRACES_EXPORTER=none Present
Off On Off OTEL_TRACES_EXPORTER=none, OTEL_LOGS_EXPORTER=none Absent
Off Off On OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=none Present
Off Off Off Invalid when OTLP is enabled Not rendered

Example — custom collector with traces only:

observability:
  collector:
    mode: enabled
    config:
      inline: |
        receivers:
          otlp:
            protocols:
              grpc:
              http:
        processors:
          batch:
        exporters:
          debug:
            verbosity: basic
        service:
          pipelines:
            traces:
              receivers: [otlp]
              processors: [batch]
              exporters: [debug]
  otlp:
    enabled: true
    tracesEnabled: true
    metricsEnabled: false
    logsEnabled: false
    global:
      endpoint: "http://neos-otel-collector:4317"

In this configuration, the chart keeps traces enabled, injects OTEL_METRICS_EXPORTER=none and OTEL_LOGS_EXPORTER=none, and does not render Serilog.WriteTo[0].

Important

observability.collector.mode=enabled alone is not a valid generated collector configuration. When using the chart-generated collector config, also enable at least one exporter backend (interClusterCommunication.appInsights.enabled, interClusterCommunication.zipkin.enabled, jaeger.enabled, or observability.collector.config.debugExporterEnabled=true), or provide observability.collector.config.inline / observability.collector.config.existingConfigMap.

Note

With observability.collector.mode=auto (default), legacy backends (interClusterCommunication.appInsights.enabled, interClusterCommunication.zipkin.enabled, jaeger.enabled) implicitly activate internal collector routing. Set observability.collector.mode=disabled to force opt-out.

Note

Custom collector config and fixed Service ports. When using config.inline or config.existingConfigMap, the Neos chart still deploys its own neos-otel-collector Kubernetes Service with fixed ports: Zipkin 9411, OTLP gRPC 4317, OTLP HTTP 4318. This Service is generated from the chart and its ports are not configurable. If your custom collector configuration exposes receivers on different ports, the neos-otel-collector Service will not route traffic to them correctly. In that case, deploy your collector and its Service independently (outside the Neos chart) and use observability.collector.mode=disabled with observability.otlp.global.endpoint or per-signal endpoints to point workloads at your collector.

Note

Collector resources can be tuned through observability.collector.resources.*. For backward compatibility, if these keys are not set, the chart falls back to interClusterCommunication.appInsights.* resource settings.

Per-signal endpoint routing

By default, all three signals (traces, metrics, logs) are sent to the same OTLP endpoint — either the internal collector URL (when the collector is active) or observability.otlp.global.endpoint (in direct OTLP mode). When the internal collector is active, global.endpoint is ignored; workloads always target the collector. global.endpoint applies exclusively in direct OTLP mode (mode=disabled or mode=auto without any legacy backend). To route individual signals to dedicated endpoints, use the per-signal endpoint overrides:

  • observability.otlp.traces.endpoint — injects OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.
  • observability.otlp.metrics.endpoint — injects OTEL_EXPORTER_OTLP_METRICS_ENDPOINT.
  • observability.otlp.logs.endpoint — overrides the Serilog sink endpoint and injects OTEL_EXPORTER_OTLP_LOGS_ENDPOINT.

Per-signal endpoints take priority over observability.otlp.global.endpoint for their respective signal. Optionally, a per-signal protocol (observability.otlp.traces.protocol, observability.otlp.metrics.protocol, observability.otlp.logs.protocol) can also be set independently from the global protocol.

When a per-signal protocol is set but no per-signal endpoint is given, the chart derives the fallback endpoint from global.endpoint (in direct OTLP mode) or from the internal collector URL. For the internal collector the URL is always rewritten with the correct port (4318 for http/protobuf, 4317 for grpc). For global.endpoint, the behavior depends on whether the effective signal protocol differs from the global protocol:

  • Same protocol: global.endpoint is used as-is. Any URL format is accepted, including non-standard ports (:443) or paths.
  • Different protocol: the chart rewrites the port in global.endpoint. In this case the URL must end with :4317 or :4318; any other format causes a chart error. Set a per-signal endpoint explicitly to avoid this constraint.

The corrected per-signal endpoint env var and, for logs, Serilog.WriteTo[0].Args.endpoint are injected automatically.

Important

In direct OTLP mode (no active collector), every auto-enabled signal must have a resolvable endpoint. Setting observability.otlp.enabled: true with only a subset of per-signal endpoints while leaving other signals on auto is invalid — those signals would be enabled but have no endpoint. Provide observability.otlp.global.endpoint to cover all signals, or explicitly disable unused signals (tracesEnabled: false / logsEnabled: false), or enable the internal collector.

Example — internal collector for traces (via Jaeger), dedicated external collectors for metrics and logs:

jaeger:
  enabled: true

observability:
  otlp:
    enabled: true
    metricsEnabled: true  # explicit: route metrics to external endpoint
    logsEnabled: true     # explicit: route logs to external endpoint
    metrics:
      endpoint: "http://external-metrics.example.com:4317"
    logs:
      endpoint: "http://external-logs.example.com:4318"
      protocol: "http/protobuf"

In this configuration the chart renders:

  • OTEL_EXPORTER_OTLP_ENDPOINT pointing to the internal collector (for traces).
  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT pointing to the external metrics collector.
  • OTEL_EXPORTER_OTLP_LOGS_ENDPOINT pointing to the external logs collector.
  • Serilog.WriteTo[0].Args.endpoint pointing to the external logs endpoint with HttpProtobuf protocol.
Important

When per-signal endpoints are used alongside a generated internal collector, the generated collector config must include a pipeline for the signals that target it. For example, a Jaeger-backed generated config only has a traces pipeline. If you override metrics.endpoint or logs.endpoint to point elsewhere, those signals bypass the internal collector entirely — they are never received by it.

Note

If only per-signal endpoints are configured (no global.endpoint, no active collector), OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL are not injected. Only the per-signal env vars and the Serilog endpoint are rendered. OTEL_EXPORTER_OTLP_TIMEOUT is still injected as a global timeout.

Basic Helm configuration

Internal collector via Jaeger (implicit auto mode):

jaeger:
  enabled: true

Internal collector with the debug exporter enabled for local inspection:

observability:
  collector:
    mode: enabled
    config:
      debugExporterEnabled: true

Direct OTLP mode:

observability:
  collector:
    mode: disabled
  otlp:
    enabled: true
    global:
      endpoint: https://otel-gateway.example.com:4317
      protocol: grpc

Dapr inter-cluster tracing in direct OTLP mode:

observability:
  collector:
    mode: disabled
  dapr:
    zipkinEndpoint: https://external-zipkin.example.com/api/v2/spans

Advanced OTEL configuration passthrough:

observability:
  otlp:
    enabled: true
    global:
      endpoint: https://otel-gateway.example.com:4317
      protocol: grpc
      timeoutMs: 15000
    additionalConfiguration:
      OTEL_EXPORTER_OTLP_COMPRESSION: gzip
      OTEL_EXPORTER_OTLP_HEADERS: x-tenant-id=neos-devtest
Note

Only keys supported by the OpenTelemetry OTLP exporter SDK are accepted.

Migration from Application Insights SDK configuration

Historically, Application Insights SDK options were configured directly in .NET backend processes. With the observability model, routing and export are now centralized in the OpenTelemetry Collector.

What changes

  • Before: each backend process configured Application Insights SDK parameters (ApplicationInsights__*).
  • Now: backend processes emit OTLP, and collector exporters define the final backend target (Application Insights, Jaeger, Zipkin, or others).
  • Advanced OTEL exporter options are no longer modeled one by one in Helm values. When needed, they are passed explicitly through observability.otlp.additionalConfiguration.

Equivalent configuration mapping

Previous SDK-oriented setup Collector-oriented setup
ApplicationInsights__ConnectionString on each process APPINSIGHTS_CONNECTION_STRING in collector exporter settings
Per-process transport and runtime tuning Shared OTLP endpoint, protocol, and timeout in observability.otlp.global.*
ApplicationInsights__* Serilog sink per process Serilog.WriteTo[0] (OpenTelemetry sink) injected automatically by the chart
Destination chosen in each process Destination chosen once in collector exporters
Mixed service-level settings across processes Centralized governance in the collector pipeline

Log routing through Serilog

Traces and metrics are emitted through the native OpenTelemetry SDK. Logs follow a different path: all Neos .NET processes use Serilog.Sinks.OpenTelemetry to send logs over OTLP.

When log export is enabled (resolved logsEnabled is true), the chart automatically injects Serilog.WriteTo[0] in every workload appsettings.json with Name: OpenTelemetry. The endpoint and protocol are taken from observability.otlp.logs.endpoint/logs.protocol if set, otherwise they fall back to global.endpoint/global.protocol:

"Serilog": {
  "WriteTo": [
    {
      "Name": "OpenTelemetry",
      "Args": {
        "endpoint": "<otlpEndpoint>",
        "protocol": "<Grpc or HttpProtobuf>"
      }
    }
  ]
}
Important

When log export is active (resolved logsEnabled is true), index WriteTo[0] is reserved by the chart for the OpenTelemetry sink. Custom sinks (e.g. Sentry) must use index 1 or higher.

Note

When WriteTo already contains an entry named OpenTelemetry, the Neos runtime automatically suppresses the programmatic Application Insights Serilog sink. This prevents duplicate log emission when both the OTLP sink and a TelemetryClient (Application Insights SDK) are present in the same process.

Note

The chart also injects a resourceAttributes block with the correct service.name for each process so that logs appear under the expected service name in the observability backend instead of unknown_service:dotnet. See Service names for the full mapping table.

Scope of additionalConfiguration vs additionalSerilogConfiguration

Property Where it is injected Signals affected
additionalConfiguration Pod env: (Kubernetes environment variables) Traces, metrics (OTLP SDK)
additionalSerilogConfiguration Serilog.WriteTo[0].Args Logs (Serilog sink)
envFromSecretName Pod envFrom: (Kubernetes secret) All signals (SDK env vars)
Note

The two maps are independent. A header required on both pipelines must be configured in both additionalConfiguration (as OTEL_EXPORTER_OTLP_HEADERS) and additionalSerilogConfiguration (as headers).

Example — adding a non-sensitive routing header for all signals:

observability:
  otlp:
    enabled: true
    global:
      endpoint: https://otel-gateway.example.com:4317
      protocol: grpc
    additionalConfiguration:
      OTEL_EXPORTER_OTLP_HEADERS: x-tenant-id=neos-devtest
    additionalSerilogConfiguration:
      headers: "x-tenant-id=neos-devtest"
Note

Do not place credentials or tokens directly in values files. For authentication headers, prefer secret-backed environment variables in backend secrets.

Warning

The keys endpoint and protocol are reserved in additionalSerilogConfiguration and cannot be overridden through this map — they are always generated by the chart.

Note

The key resourceAttributes receives special treatment: instead of replacing the chart-generated resourceAttributes block (which carries service.name), the map is merged. User-supplied sub-keys are added alongside the chart-generated service.name. When the same sub-key is present in both, the chart-generated value takes precedence, so service.name cannot be overridden through this map. Use this to inject additional OpenTelemetry resource attributes such as deployment.environment:

additionalSerilogConfiguration:
  resourceAttributes:
    deployment.environment: production

Secret-backed OTEL environment variables

Some OTEL SDK keys — notably OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_COMPRESSION — carry credentials or vary between environments. These should not be inlined in Helm values. Use observability.otlp.envFromSecretName to mount a Kubernetes Secret as envFrom on every .NET backend pod:

observability:
  otlp:
    enabled: true
    global:
      endpoint: https://otel-gateway.example.com:4317
      protocol: grpc
    envFromSecretName: neos-otel-credentials

The referenced Secret can then contain any OTEL SDK key:

apiVersion: v1
kind: Secret
metadata:
  name: neos-otel-credentials
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "authorization=Bearer <token>"
  OTEL_EXPORTER_OTLP_COMPRESSION: "gzip"
Note

Explicit env: entries generated by the chart (endpoint, protocol, timeout, and signal-specific overrides) take precedence over envFrom values according to Kubernetes precedence rules. The secret is additive and is best used for keys the chart does not manage directly.

For the full Serilog sink configuration reference, see Serilog OpenTelemetry sink.

Default trace filters

When the chart-generated internal collector configuration is active (no config.inline or existingConfigMap override), a filter/noise processor is automatically inserted in the traces pipeline. It drops high-frequency, low-value spans before they reach any exporter. The table below lists all filtered spans.

Span name pattern Type Reason
SignalR.Redis.MessageReceived Exact Redis pub/sub heartbeat emitted for every SignalR message, ~180 000–230 000 spans/day
SignalR.Context.NotificationContext.SendToUserConnectionsAsync Exact Per-user connection fan-out, same order of magnitude
Validation rules for NeosUserNotification Exact Lightweight validation span generated for each SignalR notification
opentelemetry.proto.collector.* Prefix (regex) OTLP meta-telemetry: the collector reporting on its own export calls
taskhubsidecarservice/GetWorkItems Suffix (regex, case-insensitive) Dapr TaskHub sidecar polling, fires every few seconds with no business value
taskhubsidecarservice/Hello Suffix (regex, case-insensitive) Dapr TaskHub sidecar health handshake
Note

These filters apply only to the chart-generated collector configuration. They have no effect on custom configurations supplied via observability.collector.config.inline or observability.collector.config.existingConfigMap. When using a custom configuration, add the filter/noise processor manually if you want the same behavior.

Note

The filter uses error_mode: ignore, so evaluation errors on any span attribute (e.g. missing fields) are silently skipped rather than causing the span to be dropped.

Note

Span conditions use the OpenTelemetry Collector filter processor syntax, which is based on the OpenTelemetry Transformation Language (OTTL).

Send telemetry to Application Insights through the collector

When using the chart-generated internal collector configuration, enable Application Insights by setting interClusterCommunication.appInsights.enabled=true and providing connectionStringSecret. The chart automatically includes the azuremonitor exporter and the traces pipeline. The metrics and logs pipelines are also included when Application Insights is enabled or when observability.collector.config.debugExporterEnabled=true.

When using a custom inline collector configuration (observability.collector.config.inline or existingConfigMap), connectionStringSecret is not required — the chart does not inject it as a secret reference in the collector pod's envFrom, so an absent or empty value does not cause a deployment error. Supply the Application Insights connection string through any mechanism appropriate for your collector (e.g. a Kubernetes Secret mounted on the collector pod via observability.collector.envFromSecretName, or observability.otlp.envFromSecretName to make the secret available as envFrom on backend pods for SDK-level use).

When using a custom inline collector configuration (observability.collector.config.inline or existingConfigMap), add the exporter and pipelines manually. Include the zipkin receiver in the traces pipeline so Dapr sidecars can deliver inter-cluster spans, and set observability.dapr.zipkinEndpoint to http://neos-otel-collector:9411/api/v2/spans:

observability:
  collector:
    config:
      inline: |
        receivers:
          zipkin:
            endpoint: "0.0.0.0:9411"
          otlp:
            protocols:
              grpc:
                endpoint: "0.0.0.0:4317"
        exporters:
          azuremonitor:
            connection_string: "${APPINSIGHTS_CONNECTION_STRING}"
        service:
          pipelines:
            traces:
              receivers: [zipkin, otlp]
              exporters: [azuremonitor]
            metrics:
              receivers: [otlp]
              exporters: [azuremonitor]
            logs:
              receivers: [otlp]
              exporters: [azuremonitor]
  dapr:
    zipkinEndpoint: "http://neos-otel-collector:9411/api/v2/spans"

For chart-level properties and examples, see Helm configuration - observability.

Migrating to Neos 3.1

This section summarizes the steps required when upgrading an existing deployment to Neos 3.1.

Breaking changes checklist

logMetricsAndHealthCheck renamed to logMetricsAndHealthChecks

A typo has been corrected. Rename the key in all Helm values files:

# Before
clusters:
  - versions:
      - backend:
          logMetricsAndHealthCheck: true

# After
clusters:
  - versions:
      - backend:
          logMetricsAndHealthChecks: true

instrumentationKeySecret removed

The interClusterCommunication.appInsights.instrumentationKeySecret property has been removed (it was deprecated since Neos 2.2). Replace it with connectionStringSecret if not already done:

# Before
interClusterCommunication:
  appInsights:
    enabled: true
    instrumentationKeySecret: my-secret   # removed

# After
interClusterCommunication:
  appInsights:
    enabled: true
    connectionStringSecret: my-secret

The secret must expose the APPINSIGHTS_CONNECTION_STRING environment variable.

Internal OpenTelemetry Collector renamed

The Kubernetes resources previously named neos-dapr-otel-collector have been replaced by neos-otel-collector. Update any hardcoded references in custom appsettings.json overrides or external Dapr configuration:

Old value New value
http://neos-dapr-otel-collector:4317 http://neos-otel-collector:4317
http://neos-dapr-otel-collector:9411 http://neos-otel-collector:9411
neos-dapr-otel-collector-configmap neos-otel-collector-configmap

Standard Helm-managed deployments are updated automatically.

Disable Application Insights SDK secrets in Kubernetes

When deploying on Kubernetes, Application Insights SDK settings are typically injected as environment variables in Kubernetes secrets, under keys prefixed with ApplicationInsights__ (e.g. ApplicationInsights__ConnectionString, ApplicationInsights__RoleName).

Once OTLP is configured, the Neos runtime automatically guards against double telemetry emission at two levels:

  • Serilog logs: when Serilog:WriteTo already contains an OpenTelemetry sink, the programmatic Application Insights Serilog sink is not registered. This is handled by SerilogConfiguration.Configure at startup.
  • Traces and metrics SDK: ConfigureApplicationInsights skips all SDK registration when OTLP is active for at least one of the legacy signals (traces or metrics). OTLP is considered active when OTEL_EXPORTER_OTLP_ENDPOINT is set, or when OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set, provided the corresponding signal is not explicitly disabled via OTEL_TRACES_EXPORTER=none / OTEL_METRICS_EXPORTER=none. A logs-only configuration (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT without a traces or metrics endpoint) does not block Application Insights SDK registration. To re-enable the Application Insights SDK alongside an active OTLP endpoint, set both OTEL_TRACES_EXPORTER=none and OTEL_METRICS_EXPORTER=none in the pod environment.

Despite these automatic guards, cleaning up ApplicationInsights__* keys from Kubernetes secrets is still recommended to avoid unnecessary SDK initialization overhead and to prevent confusion during troubleshooting.

The following PowerShell script renames all ApplicationInsights__* keys in a namespace by prefixing them with _ (e.g. ApplicationInsights__ConnectionString becomes _ApplicationInsights__ConnectionString). The values are preserved, allowing a rollback by reversing the rename. The script is idempotent: if a destination key already exists, the corresponding source key is skipped with a warning.

<#
.SYNOPSIS
  Disables Application Insights SDK configuration in Kubernetes secrets.
.DESCRIPTION
  Renames all secret keys starting with 'ApplicationInsights__' by prefixing them with '_'
  (e.g. ApplicationInsights__ConnectionString becomes _ApplicationInsights__ConnectionString).
  This effectively disables the Application Insights SDK without deleting the values, allowing
  a rollback if needed.
  Useful when migrating workloads from the Application Insights SDK to OpenTelemetry.
.PARAMETER Namespace
  The namespace where the secrets are located.
.PARAMETER DryRun
  If specified, the script will only display the changes that would be made without applying them.
.PARAMETER Rollback
  If specified, performs the reverse operation by renaming keys from '_ApplicationInsights__*'
  back to 'ApplicationInsights__*'.
.EXAMPLE
  ./disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest
  Renames all 'ApplicationInsights__*' keys in secrets in the neos-neos-devtest namespace.
.EXAMPLE
  ./disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest -DryRun
  Displays the changes that would be made in the neos-neos-devtest namespace without applying them.
.EXAMPLE
  ./disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest -Rollback
  Restores all '_ApplicationInsights__*' keys to 'ApplicationInsights__*' in secrets in the neos-neos-devtest namespace.
#>

[CmdletBinding()]
param (
  [Parameter(Mandatory = $true)]
  [String]
  $Namespace,

  [Parameter(Mandatory = $false)]
  [Switch]
  $DryRun,

  [Parameter(Mandatory = $false)]
  [Switch]
  $Rollback
)

$keyPrefix = "ApplicationInsights__"
$disabledKeyPrefix = "_"

if ($Rollback) {
  $searchPrefix = "$disabledKeyPrefix$keyPrefix"
  $fromPrefix = "$disabledKeyPrefix$keyPrefix"
  $operationLabel = "rollback"
}
else {
  $searchPrefix = $keyPrefix
  $fromPrefix = $keyPrefix
  $operationLabel = "disable"
}

Write-Host "Searching for secrets with '$searchPrefix*' keys in namespace '$Namespace' ($operationLabel mode)..."

$allSecrets = (kubectl get secrets -n $Namespace -o json | ConvertFrom-Json).items

$matchingSecrets = $allSecrets | Where-Object {
  $_.data -and ($_.data.PSObject.Properties.Name | Where-Object { $_.StartsWith($searchPrefix) })
}

if (-not $matchingSecrets) {
  Write-Host "No secrets with '$searchPrefix*' keys found in namespace '$Namespace'."
  exit 0
}

foreach ($secret in $matchingSecrets) {
  $secretName = $secret.metadata.name
  $keysToRename = @($secret.data.PSObject.Properties.Name | Where-Object { $_.StartsWith($fromPrefix) })

  Write-Host ""
  Write-Host "Secret: $secretName"

  $operations = [System.Collections.Generic.List[object]]::new()

  foreach ($key in $keysToRename) {
    if ($Rollback) {
      $newKey = $key.Substring(1)
    }
    else {
      $newKey = "$disabledKeyPrefix$key"
    }

    # Skip if destination key already exists to avoid overwriting values
    if ($secret.data.PSObject.Properties.Name -contains $newKey) {
      Write-Host "  WARNING: Key '$newKey' already exists. Skipping '$key'." -ForegroundColor Yellow
      continue
    }

    Write-Host "  $key -> $newKey"

    $value = $secret.data.$key

    # Add the renamed key first, then remove the old one — order matters for atomicity
    $operations.Add([pscustomobject]@{ op = "add"; path = "/data/$newKey"; value = $value })
    $operations.Add([pscustomobject]@{ op = "remove"; path = "/data/$key" })
  }

  if ($operations.Count -eq 0) {
    Write-Host "  Nothing to do."
    continue
  }

  if ($DryRun) {
    continue
  }

  $patchJson = ConvertTo-Json @($operations) -Compress

  kubectl patch secret $secretName -n $Namespace --type='json' -p $patchJson | Out-Null

  if ($LASTEXITCODE -eq 0) {
    Write-Host "  Patched successfully."
  }
  else {
    Write-Host "  ERROR: Failed to patch secret '$secretName'." -ForegroundColor Red
  }
}

if ($DryRun) {
  Write-Host ""
  Write-Host "Dry run completed. No changes were applied."
}
else {
  Write-Host ""
  Write-Host "Done."
}

  1. Preview the changes:

    .\disable-secrets-applicationinsights.ps1 -Namespace <your-namespace> -DryRun

  2. Apply the changes:

    .\disable-secrets-applicationinsights.ps1 -Namespace <your-namespace>

  3. Restart the affected pods so the renamed environment variables take effect.

  4. If the result is not satisfactory, preview the rollback:

    .\disable-secrets-applicationinsights.ps1 -Namespace <your-namespace> -Rollback -DryRun

  5. Apply the rollback:

    .\disable-secrets-applicationinsights.ps1 -Namespace <your-namespace> -Rollback

  6. Restart the affected pods again after rollback.

Example for neos-neos-devtest:

.\disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest -DryRun
.\disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest
.\disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest -Rollback -DryRun
.\disable-secrets-applicationinsights.ps1 -Namespace neos-neos-devtest -Rollback
Note

The script is idempotent: if a destination key already exists, the corresponding source key is skipped with a warning.

  1. Decide whether to use internal collector mode (observability.collector.mode=enabled) or direct OTLP mode (observability.collector.mode=disabled).
  2. In direct OTLP mode, set observability.otlp.enabled=true and configure the endpoint, protocol, and timeout.
  3. If you use the chart-generated internal collector configuration, enable at least one exporter backend or provide observability.collector.config.inline / observability.collector.config.existingConfigMap.
  4. Add advanced OTEL SDK keys only when needed through observability.otlp.additionalConfiguration (applies to traces and metrics — use observability.otlp.additionalSerilogConfiguration for logs), using the SDK documentation as reference.
  5. Configure collector exporters for your target backend, including Application Insights when required.
  6. Validate telemetry in the target backend.
  7. Keep the Application Insights JavaScript library configuration for the frontend unchanged — the new OTLP pipeline is backend-only.