Prometheus remote-write

Ship metrics from an existing Prometheus server without re-instrumenting anything, and the limits a write request must stay inside.

If you already run Prometheus, remote-write is the shortest path to getting those metrics into aiAxonIQ: it is a configuration change on the Prometheus side and nothing at all on your services.

Before you start. You need a license key and a running Prometheus server, version 2.x or later.

Configure Prometheus

Add a remote_write block to prometheus.yml. The base endpoint differs between a hosted deployment and a self-hosted one, so no single URL is correct here:

Deployment	Base endpoint
aiAxonIQ Cloud	https://app.aiaxoniq.com/otlp
Self-hosted behind the bundled nginx	https://app.<your-domain>/otlp

Both forms carry the /otlp prefix because nginx serves OTLP under it and strips it before forwarding, so the receiver still sees /v1/logs. Dropping the prefix is the most common setup mistake: the request reaches the dashboard instead of the receiver and comes back as an HTML 404 rather than an ingest error.

prometheus.yml

remote_write:
  - url: <base-endpoint>/api/v1/prom/write
    headers:
      X-License-Key: oiq_4f3c2b1a9e8d7c6b5a4f3e2d1c0b9a8e

The endpoint is POST /api/v1/prom/write. Prometheus sends snappy-compressed protobuf by default, which is what this endpoint expects — no encoding options need setting.

A successful write returns 202 Accepted.

This endpoint does not accept gzip. Every other ingest endpoint accepts gzip-compressed bodies. This one does not: it expects the snappy-compressed protobuf that remote-write already produces, and gzip on top of that is rejected. Leave Prometheus’s compression settings alone and it will do the right thing.

Your account must have metrics enabled. If metrics are not part of your plan, writes are rejected with 403 rather than silently dropped.

Limits per request

A remote-write request is rejected outright if it exceeds any of these. The defaults are generous enough that a normally-configured Prometheus will not approach them, but a large federated setup can.

Limit	Value
Compressed body	8 MB
Decompressed body	64 MB
Timeseries per request	10,000
Samples per request	100,000
Labels per series	128
Bytes per label name or value	1,024

Every series must carry a __name__ label. Prometheus always sets it; a custom client might not.

If you hit these, the fix is to shorten Prometheus’s remote-write flush interval or reduce the batch size rather than to send fewer metrics — smaller, more frequent requests carry the same data.

Samples outside the accepted window

Samples are accepted if their timestamp falls within the last 30 days and no more than 1 hour in the future.

Samples outside that window are dropped silently — the request still succeeds with a 202 and the out-of-range samples simply do not appear. This is deliberate, so that one bad timestamp does not fail an otherwise good batch, but it does mean a backfill of older data will appear to succeed while writing nothing.

The future-dated allowance exists for clock skew between your Prometheus host and ingest. A host whose clock is more than an hour fast will lose every sample it sends, and the symptom is a healthy-looking Prometheus writing into a void — check clock sync before anything else if metrics from one host are missing entirely.

Rate limiting

Remote-write is subject to a per-account limit of 10,000 requests per minute, shared with your other ingest traffic. Exceeding it returns 429 with a Retry-After header.

Prometheus’s remote-write client handles 429 responses by backing off and retrying, so a brief overshoot is absorbed without data loss.

Verifying

Prometheus exposes its own remote-write metrics, and they are the fastest way to tell whether the problem is on your side or ours:

# Samples successfully written.
rate(prometheus_remote_storage_samples_total[5m])

# Samples that failed permanently — should be flat at zero.
rate(prometheus_remote_storage_samples_failed_total[5m])

# How far behind the write path is falling.
prometheus_remote_storage_highest_timestamp_in_seconds
  - ignoring(remote_name, url) prometheus_remote_storage_queue_highest_sent_timestamp_seconds

A rising failure rate with a growing lag points at rejected requests — check the Prometheus server log, which records the status code and response body from the failed write.

Troubleshooting

Writes succeed but the metrics never appear

Almost always the timestamp window. Samples older than 30 days or more than an hour in the future are dropped silently — the request still returns 202. A backfill of historical data therefore looks like it worked and wrote nothing.

Check clock sync on the Prometheus host first. A host running more than an hour fast loses every sample it sends, and nothing in the response says so.

400 Bad Request on every write

Either a cap was exceeded — 10,000 series, 100,000 samples, 128 labels per series — or a series is missing its __name__ label. Prometheus always sets __name__; a custom remote-write client might not.

Shorten the flush interval rather than sending fewer metrics: smaller, more frequent requests carry the same data.

415 or a decompression error

Something is gzipping the body. This endpoint expects the snappy-compressed protobuf that remote-write already produces and has HTTP-level decompression disabled, so gzip on top of it fails.

Leave Prometheus’s compression settings at their defaults.

403 Forbidden

The key is valid but metrics are not enabled on your plan. This is a plan entitlement rather than a key property — see Create a license key.

Next

Alerting

Alert on the metrics you are now sending.

Endpoints and errors

Every status code and limit.