Send data from Kubernetes

A DaemonSet collector for node and pod telemetry, where the license key goes, and the two collectors most clusters end up needing.

aiAxonIQ has no Kubernetes agent of its own. It accepts OTLP, so the thing you install is the upstream OpenTelemetry Collector, pointed at your ingest endpoint. Everything below is stock chart configuration.

Before you start, you need two values.

1. Your base endpoint — shown on Get Started in the dashboard. These pages write it as $OIQ_ENDPOINT.
2. A license key — created in Settings → License Keys, starting oiq_. Requires the Admin role. See Create a license key. These pages write it as $OIQ_LICENSE_KEY.

Export both before running anything below:

export OIQ_ENDPOINT="https://app.aiaxoniq.com/otlp"   # or your own
export OIQ_LICENSE_KEY="oiq_..."

You also need a cluster you can install a Helm chart into, with kubectl and helm configured against it.

The shape most clusters need

There are two jobs, and they want different deployment modes.

Per-node telemetry

Container logs, host metrics, and OTLP from workloads on that node. One collector per node: mode: daemonset.

Cluster-level telemetry

Object counts, deployment replica status, events. Exactly one per cluster: mode: deployment with replicaCount: 1.

Running the cluster receiver as a DaemonSet is the usual first mistake. Every node reports the same cluster-wide numbers, and the result is not wrong so much as multiplied by your node count.

Start with the DaemonSet. Add the second collector when you want workload status alongside the telemetry.

Store the key as a Secret

kubectl create namespace observability

kubectl create secret generic aiaxoniq \
  --namespace observability \
  --from-literal=license-key="$OIQ_LICENSE_KEY"

The key is an ingest credential for the whole workspace. It belongs in a Secret referenced by the collector, never in a values file committed to a repository.

Use --from-literal, not --from-file. A file created by a shell heredoc carries a trailing newline, and that newline becomes part of the key value — producing a 401 that looks nothing like a formatting problem.

Install the node collector

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install aiaxoniq-node open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values node-values.yaml

node-values.yaml

mode: daemonset
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: aiaxoniq
        key: license-key

presets:
  # Container logs from the node, with pod/namespace/container metadata attached.
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu: {}
        memory: {}
        disk: {}
        filesystem: {}
        network: {}

  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch:
      timeout: 5s
      send_batch_size: 1000

  exporters:
    otlphttp/aiaxoniq:
      # The base URL shown on Get Started in the dashboard. A self-hosted
      # deployment behind the bundled nginx serves OTLP under /otlp; a hosted
      # one does not. Use the value the dashboard gives you.
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]
      metrics:
        receivers: [otlp, hostmetrics]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]

memory_limiter goes first, batch goes last. The order of processors is the order data passes through them. memory_limiter can only shed load it sees before anything has buffered it, and batch should group records after every other processor has finished changing them. A pipeline that batches first and limits afterwards will still OOM under the load the limiter was added for.

Add the cluster collector

helm install aiaxoniq-cluster open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values cluster-values.yaml

cluster-values.yaml

mode: deployment
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: aiaxoniq
        key: license-key

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true

config:
  exporters:
    otlphttp/aiaxoniq:
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip

The clusterMetrics preset is what populates the Kubernetes pages in the product — clusters, nodes, namespaces, pods and containers are all derived from k8s.* metrics. Without it those pages stay empty however much application telemetry is flowing, because they are describing the cluster rather than the workloads.

Point applications at the node collector

Inside the cluster, applications export to their own node’s collector rather than to aiAxonIQ directly. The DaemonSet service address is stable:

env:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4318"

This keeps the license key in one place — the collector — instead of in every workload’s environment, and it means the batching and retry behaviour is configured once.

Verify

First confirm the pods are up — one per node, plus one cluster collector:

kubectl -n observability get pods

Expected output:

NAME                                 READY   STATUS    RESTARTS   AGE
aiaxoniq-cluster-6b9f7c8d4-x2kqp     1/1     Running   0          62s
aiaxoniq-node-4tzrw                  1/1     Running   0          71s
aiaxoniq-node-9mp2s                  1/1     Running   0          71s

A DaemonSet pod count that does not match your node count means a node is tainted or short of resources — kubectl describe on the DaemonSet says which.

Then read the collector’s own metrics:

kubectl -n observability port-forward svc/aiaxoniq-node 8888:8888 &
curl -s localhost:8888/metrics \
  | grep -E 'otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)'

Expected output:

otelcol_receiver_accepted_log_records{receiver="filelog"} 2841
otelcol_exporter_sent_log_records{exporter="otlphttp/aiaxoniq"} 2841
otelcol_exporter_send_failed_log_records{exporter="otlphttp/aiaxoniq"} 0

send_failed at zero with sent climbing is a working install. Then open Kubernetes in the product, set the range to the last 15 minutes, and confirm your nodes and pods are listed.

Troubleshooting

401 in the collector logs

The key did not reach the pod, or reached it with a trailing newline.

kubectl -n observability get secret aiaxoniq -o jsonpath='{.data.license-key}' \
  | base64 -d | wc -c

A key is 36 characters. 37 means a trailing newline — recreate the Secret with --from-literal, which does not add one, rather than --from-file.

Then confirm extraEnvs names the same Secret and key.

Kubernetes pages are empty while application telemetry flows

Those pages are built from k8s.* metrics, which come from the cluster collector, not from your services or the DaemonSet. Confirm the clusterMetrics preset is enabled and the cluster collector pod is running.

This is the most common Kubernetes install outcome that looks like a bug and is not.

Every cluster metric appears multiplied by the node count

The cluster collector is running as a DaemonSet. It must be mode: deployment with replicaCount: 1 — every node is reporting the same cluster-wide numbers.

A DaemonSet pod is CrashLoopBackOff

Almost always the config. The collector refuses to start on an invalid configuration rather than running degraded.

kubectl -n observability logs -l app.kubernetes.io/name=opentelemetry-collector \
  --tail=50 --previous

--previous is the important flag — without it you read the logs of the container that is about to fail rather than the one that already did.

Applications cannot reach the node collector

Confirm HOST_IP is being injected. The fieldRef on status.hostIP must be declared before the variable that references it in the same env list, because Kubernetes resolves $(VAR) in declaration order.

The collector starts, then logs 'Exporting failed. Will retry'

Read the error field on that line — it names the cause exactly.

Exporting failed. Will retry the request after interval.
  {"kind": "exporter", "name": "otlphttp/aiaxoniq",
   "error": "failed to make an HTTP request: Post \"…/v1/metrics\":
             dial tcp: lookup … : no such host", "interval": "5.2s"}

• no such host — the endpoint hostname does not resolve. Check $OIQ_ENDPOINT against the value on Get Started.
• connection refused — the host resolves but nothing is listening on that port.
• 401 Unauthorized — the key is missing, malformed or revoked.
• 404 — the endpoint already ends in /v1/…. It must be the base URL; the collector appends the signal path itself.

The collector retries with backoff and does not drop data while it retries, so a transient failure here is not a loss.

Nothing at all in the logs after 'Everything is ready'

The collector is running and receiving nothing. That is an application-side problem, not a collector one — your services are not exporting to it.

Check that your application’s OTEL_EXPORTER_OTLP_ENDPOINT points at the collector’s OTLP port (4318 for HTTP, 4317 for gRPC), not at aiAxonIQ.

401 with a key you know is correct

Two causes that are not about the key’s value:

• A trailing newline. A key read from a file created by a shell heredoc, or a Kubernetes Secret made with --from-file, carries the newline as part of the value. Use --from-literal, or printf rather than echo.
• A validation outage. If the receiver cannot reach the service that validates keys it fails closed and returns the same 401. A sudden 401 across every service at once, with a key you have not changed, is far more likely to be this. Check $OIQ_ENDPOINT/health first.

Config changes appear to do nothing

The collector reads its configuration only at startup. Restart it after any edit, and confirm the file you edited is the one mounted into the process — a bind mount pointing at a path that does not exist silently yields the image’s default config rather than an error.

Next

Verify your data

Confirm it arrived.

Instrument the applications

Zero-code setup per language.

Endpoints and errors

Every status code and limit.