Observability Storage & Retention

Traces, request logs, and metrics are the fastest-growing data on a Temps server — a single busy, fully auto-instrumented app can write more telemetry in a day than every other table combined in a year. This page explains what each record costs on disk, what the defaults do for you, how to measure where your disk went, and the levers for storing a lot of records with as little storage as possible.

What a record costs

Approximate on-disk cost per record, including index overhead. Exact numbers depend on attribute sizes, URL lengths, and how compressible your workload is — measure yours with the queries below.

Two things follow from this table:

• On the default backend, your most recent data is the expensive data. TimescaleDB only compresses chunks after they age past the compression window (7 days by default), so the hot window is always stored at full row-store cost with full B-tree indexes.
• ClickHouse is roughly 10× cheaper per record and compresses data minutes after insert, not days. If "store a lot of records with the least storage" is the goal, ClickHouse is the intended backend — see below.

Retention & compression defaults

On the default PostgreSQL/TimescaleDB backend, Temps creates these policies automatically — no configuration needed:

On the ClickHouse backend, retention is enforced per row via a retention_days column (same defaults: 90 days for spans and metrics, 30 days for request logs), and compression happens automatically as parts merge — typically within minutes of ingest.

Measure your usage

All of these run in psql against the Temps database.

Total size and bytes per record for the big tables:

SELECT hypertable_name,
       pg_size_pretty(hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)) AS total,
       approximate_row_count(format('%I.%I', hypertable_schema, hypertable_name)::regclass) AS rows,
       (hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)::float8
         / NULLIF(approximate_row_count(format('%I.%I', hypertable_schema, hypertable_name)::regclass), 0))::int
         AS bytes_per_row
FROM timescaledb_information.hypertables
WHERE hypertable_name IN ('otel_spans', 'otel_metrics', 'otel_log_events', 'proxy_logs');

Growth per day (chunk sizes double as a daily ingest report):

SELECT c.range_start::date AS day,
       pg_size_pretty(sum(d.total_bytes)) AS size
FROM timescaledb_information.chunks c
JOIN chunks_detailed_size('otel_spans') d
  ON d.chunk_name = c.chunk_name AND d.chunk_schema = c.chunk_schema
WHERE c.hypertable_name = 'otel_spans'
GROUP BY 1 ORDER BY 1;

Who is writing all of it — sample a recent chunk instead of scanning the whole table (a full GROUP BY over hundreds of GB will run for minutes; a 0.5% block sample answers in seconds — take the chunk name from the previous query and multiply counts by ~200):

SELECT project_id, service_name, name,
       count(*) AS sampled,
       avg(pg_column_size(t.*))::int AS avg_bytes
FROM _timescaledb_internal._hyper_NN_NNNN_chunk t TABLESAMPLE SYSTEM (0.5)
GROUP BY 1, 2, 3
ORDER BY sampled DESC
LIMIT 20;

If the top rows are named middleware - jsonParser, middleware - corsMiddleware, tcp.connect, or similar, an app is running default auto-instrumentation — that is your disk, and the next section is your fix.

Reduce span volume at the source

Auto-instrumentation defaults are tuned for debugging, not for production volume. Two changes typically cut span volume by 95–99% with no loss of useful signal:

1. Disable per-middleware and socket-level spans. For Node.js (@opentelemetry/auto-instrumentations-node):

getNodeAutoInstrumentations({
  '@opentelemetry/instrumentation-express': {
    // keep the request span, drop the ~10 middleware/router spans per request
    ignoreLayersType: ['middleware', 'request_handler'],
  },
  // drop tcp.connect / tls.connect spans
  '@opentelemetry/instrumentation-net': { enabled: false },
})

2. Head-sample. Most services do not need every trace — 5–10% is plenty to see latency distributions and errors:

const { TraceIdRatioBasedSampler } = require('@opentelemetry/sdk-trace-base')

new NodeSDK({
  sampler: new TraceIdRatioBasedSampler(0.05), // keep 5% of traces
  // ...
})

Equivalent knobs exist in every OTel SDK (OTEL_TRACES_SAMPLER=traceidratio and OTEL_TRACES_SAMPLER_ARG=0.05 work language-independently). Also skip tracing health checks and static assets — they are the highest-volume, lowest-value spans in most apps.

Cap ingest with storage quotas

Quotas are the platform-side safety net for instrumentation you don't control. They are disabled by default. Enable them by setting a per-project cap (in GB) on temps serve:

TEMPS_OTEL_QUOTA_GB=10

Once a project's combined traces + metrics + logs usage reaches the cap, further OTLP ingest for that project is rejected with 413 Storage Quota Exceeded until retention frees space. Other projects are unaffected. Setting the variable to 0 (or unsetting it) disables quotas again.

OTLP ingest is also rate-limited per project (TEMPS_OTEL_RATE_LIMIT, requests per window; TEMPS_OTEL_RATE_LIMIT_WINDOW_SECS), which bounds burst behavior where the quota bounds accumulation.

Shorten retention

If you rarely look at traces older than a couple of weeks, 90 days of spans is pure disk cost. Shorten it:

SELECT remove_retention_policy('otel_spans');
SELECT add_retention_policy('otel_spans', drop_after => INTERVAL '30 days');

And to reclaim space immediately instead of waiting for the policy's next run — for example after an instrumentation flood — drop whole chunks (instant, no vacuum required):

SELECT drop_chunks('otel_spans', older_than => now() - INTERVAL '30 days');

Trace summaries (the aggregate view used for the traces list) are stored separately and much more compactly, so shortening raw-span retention does not erase your long-horizon picture.

Switch to ClickHouse for high volume

For sustained high-volume telemetry, ClickHouse is the recommended backend. It stores the same spans, metrics, and request logs roughly 10× smaller than the row store, and — unlike TimescaleDB's age-based compression — compresses data within minutes of insert, so a traffic spike costs columnar prices immediately.

Use ClickHouse when any of these hold:

• Sustained ingest above roughly 100–500 spans/second (a single auto-instrumented app at moderate traffic can exceed this).
• Observability tables growing by more than about 1 GB/day.
• You host projects whose instrumentation you don't control (the flood-protection combination is quota + ClickHouse).
• Trace-list and wide-time-window queries are getting slow on the default backend.

Below those thresholds, stay on the default: TimescaleDB needs no extra service and keeps the single-binary install true to its name.

Switching is a runtime backend change — set the TEMPS_CLICKHOUSE_* variables and restart; no rebuild, no data loss, reversible. The full procedure is in Migrate Observability to ClickHouse.

Sizing rules of thumb

For capacity planning, per 1 million requests/day through an instrumented app:

Request logs add roughly 0.5–2 KB per request on the default backend (30-day retention) and a small fraction of that on ClickHouse.

If a disk fills faster than these numbers predict, run the measurement queries — the answer is almost always one service emitting middleware-level spans at full sampling.

Related pages

• Migrate Observability to ClickHouse — the step-by-step backend switch.
• OpenTelemetry (OTLP) Ingest & Query — how traces, metrics, and logs are ingested.
• Observe — the unified view over whichever backend is configured.
• Environment Variables Reference — TEMPS_OTEL_QUOTA_GB, rate limits, and the TEMPS_CLICKHOUSE_* family.
• Production Checklist — disk and retention checks before going live.