Observability Without a Platform Team: OpenTelemetry That Pays for Itself

There's a specific moment every small team hits: a client messages "checkout was slow yesterday around 4pm," and you have nothing. Maybe some access logs. Maybe a hosting dashboard that says CPU was fine. You end up guessing, shipping a speculative fix, and hoping. That moment is when observability stops being a platform-team luxury and starts being table stakes — and the good news is that in the OpenTelemetry era, a team of two to ten engineers can get 90% of the value of a dedicated observability org with a few days of deliberate setup. We run web products for clients with a small senior team, no SREs, no platform group, and we can answer "what happened at 4pm" in about ninety seconds. This post is the setup that gets you there, and — just as important — the parts you should skip.

Logs, metrics, traces — and why traces win for request-shaped systems

The classic "three pillars" framing treats logs, metrics, and traces as peers. They aren't. They have different shapes, different costs, and wildly different leverage depending on what your system looks like.

• Logs are timestamped events. Infinitely flexible, easy to emit, expensive to store, and nearly useless for answering why questions unless you already know what you're looking for. Grep is a search tool, not a debugging strategy.
• Metrics are pre-aggregated numbers over time. Cheap, fast to query, great for "is the system healthy," terrible for "why is this one user's request slow" — the aggregation threw away exactly the context you need.
• Traces are the causal story of a single request: every service hop, database query, cache miss, and external API call, with timing, as a tree of spans.

If your system is request-shaped — an HTTP request comes in, fans out to a database, a cache, a third-party API, maybe a queue, and a response goes out — traces are the highest-leverage signal you can collect, full stop. A single trace of a slow checkout tells you it spent 40ms in your handler, 1,800ms waiting on a payment provider, and 300ms in an N+1 query you didn't know you had. Getting the same answer from logs means correlating a dozen lines across three services by timestamp and prayer. Getting it from metrics is impossible.

Most web products a small team builds — Next.js apps, API backends, worker queues — are request-shaped. So the pragmatic ordering is: traces first, structured logs that link to traces second, metrics last (and mostly derived from the traces you already have). This is roughly the opposite of how most teams stumble into observability, which is why most teams have a Grafana instance nobody opens and still can't debug a slow endpoint.

OpenTelemetry in one page

OpenTelemetry (OTel) won. It's the vendor-neutral standard for generating and shipping telemetry, it's the second-most-active CNCF project after Kubernetes, and every serious backend — Datadog, Honeycomb, Grafana, New Relic, Axiom, self-hosted Jaeger — ingests its wire format (OTLP) natively. Instrument once, point the pipe wherever you want, and switching vendors becomes a config change instead of a rewrite. That last property is the entire reason a small team should care: you are not locked in, so you can start cheap and move later.

The project has three parts people constantly conflate:

• The API (@opentelemetry/api) — the interfaces your application code calls: trace.getTracer(), span.setAttribute(). Deliberately tiny and dependency-safe; libraries instrument against the API so they work whether or not an SDK is installed.
• The SDK (@opentelemetry/sdk-node and friends) — the concrete implementation you configure once at process startup: what to sample, where to export, what resource attributes (service name, version, environment) to stamp on everything.
• The Collector — a standalone binary that receives telemetry, processes it (sampling, redaction, batching), and exports it to one or more backends. More on this below; it's the piece small teams skip and shouldn't.

The fourth piece, quieter but load-bearing, is semantic conventions: standardized attribute names like http.response.status_code, db.system.name, and gen_ai.usage.input_tokens. Follow them even when it feels bureaucratic. Every backend's prebuilt dashboards, sample queries, and anomaly detection assume them, and your future self grepping for "which attribute did we call the customer tier" will thank you.

Instrumenting a Node/Next.js service, realistically

Auto-instrumentation does most of the work

The Node SDK ships auto-instrumentation for HTTP, fetch/undici, Express, Fastify, pg, mysql2, Redis, gRPC, and dozens more. You get spans for every inbound request and every outbound call without touching application code. Startup file:

// instrumentation.node.ts — must load before anything else
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "storefront-api",
    [ATTR_SERVICE_VERSION]: process.env.GIT_SHA ?? "dev",
    "deployment.environment.name": process.env.NODE_ENV ?? "development",
  }),
  traceExporter: new OTLPTraceExporter({
    // OTEL_EXPORTER_OTLP_ENDPOINT env var also works; default is localhost:4318
    url: "http://localhost:4318/v1/traces",
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // fs instrumentation is noisy and rarely useful in web apps
      "@opentelemetry/instrumentation-fs": { enabled: false },
    }),
  ],
});

sdk.start();

In Next.js this slots into the instrumentation.ts hook, which the framework calls once per server process:

// instrumentation.ts (project root)
export async function register() {
  if (process.env.NEXT_RUNTIME === "nodejs") {
    await import("./instrumentation.node");
  }
}

(If you deploy on Vercel, @vercel/otel's registerOTel() wraps this same setup in one call and plays nicely with the runtime. Same OTLP output, same portability.)

Point the exporter at a Collector, deploy, and you already have per-route latency, status codes, DB query spans, and external call timing. This is a legitimate half-day of work and it's the single biggest step change in the whole journey.

Custom spans around business logic

Auto-instrumentation tells you a POST to /api/orders was slow. It doesn't tell you whether the slowness was inventory reservation, payment capture, or the email send. That's what custom spans are for — and the discipline is to wrap business operations, not functions:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("checkout");

export async function processOrder(order: Order) {
  return tracer.startActiveSpan("checkout.process_order", async (span) => {
    span.setAttribute("order.id", order.id);
    span.setAttribute("order.item_count", order.items.length);
    span.setAttribute("order.payment_method", order.paymentMethod);
    try {
      await reserveInventory(order);     // child spans nest automatically
      const receipt = await capturePayment(order);
      span.setAttribute("payment.provider_latency_ms", receipt.latencyMs);
      return receipt;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Two rules of thumb. First, attributes are the payoff — a span named checkout.process_order with no attributes is barely better than a log line, while one carrying order.item_count and order.payment_method lets you ask "are 3DS payments the slow ones?" without shipping code. Second, don't over-span: five meaningful spans per request beat fifty trivial ones. If a function is pure CPU and takes microseconds, it doesn't need a span.

Propagating context across queues

Auto-instrumentation propagates trace context over HTTP automatically (via the W3C traceparent header). Queues break the chain — the consumer runs in a different process, later, with no headers. Fix it by injecting context into the message and extracting it on the other side:

import { context, propagation, trace } from "@opentelemetry/api";

// Producer: stash context in the message payload
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
await queue.send({ body: job, otel: carrier });

// Consumer: restore it before doing work
const parentCtx = propagation.extract(context.active(), message.otel);
context.with(parentCtx, () => {
  tracer.startActiveSpan("email.send_receipt", (span) => {
    /* ... */
    span.end();
  });
});

Ten lines, and now a trace shows the full story: request → enqueue → dequeue → email sent, including how long the job sat in the queue. Queue-wait time is one of the most common invisible latency sources in small systems, and this makes it a first-class measurement.

The Collector is your control plane

The tempting shortcut is to export straight from your app to a SaaS backend. It works, and it's fine for week one. But routing everything through an OpenTelemetry Collector — one small container next to your app — buys you three things that matter enormously when you're small:

1. Vendor mobility. Your app exports OTLP to one address forever. Trying a new backend, or sending traces to two backends during a migration, is a Collector config change and a redeploy of one container. No application release.
2. Redaction before egress. Auth headers, emails, anything PII-shaped gets scrubbed or hashed before it leaves your infrastructure. This is the difference between "we should audit what we send to the vendor" and "we provably don't send it."
3. Cost control at one choke point. Sampling, filtering health-check spam, dropping noisy attributes — all in one place, owned by config, not scattered across services.

A production-shaped config, including tail sampling (explained next):

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

  # Drop health checks before they cost anything
  filter/healthchecks:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/healthz"'

  # Scrub PII before egress
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash

  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

  batch: {}

exporters:
  otlphttp/backend:
    endpoint: ${env:BACKEND_OTLP_ENDPOINT}
    headers:
      authorization: ${env:BACKEND_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/healthchecks, attributes/redact, tail_sampling, batch]
      exporters: [otlphttp/backend]

That's the whole thing. One YAML file is your observability control plane.

Head vs tail sampling, and keeping costs sane

Nobody needs every trace. At even modest traffic, storing 100% of spans is how observability bills become a line item the CFO asks about. The question is which traces to keep, and there are two strategies:

Head sampling decides at the start of a trace — "keep 10% of requests, chosen randomly" — configured in the SDK. It's simple and cheap because unsampled traces are never even generated. The fatal flaw: the decision is made before anything interesting has happened, so it keeps 10% of your errors and 10% of your slow requests too. The traces you most need are exactly as likely to be discarded as the boring ones.

Tail sampling decides at the end, in the Collector, after the whole trace is assembled. Now the policy can be intelligent, like the one above: keep every error, keep everything slower than 2 seconds, and keep a 10% random baseline so you still understand normal behavior. This is the right default for small teams — you get every trace that matters at a fraction of full-volume cost. The trade-offs are real but manageable: the Collector must buffer traces in memory during the decision_wait window, and if you scale to multiple Collector replicas you need load-balancing so all spans of a trace land on the same instance. At small-team scale, one adequately-sized Collector handles it without ceremony.

A hybrid worth knowing: light head sampling at very high volume endpoints plus tail sampling for everything else. But don't start there. Start with tail sampling and the three policies above; tune when the bill or the gaps tell you to.

Choosing a backend

Honest trade-offs, no pricing tables (pricing changes faster than blog posts):

Our take for a team without dedicated ops: start with a managed backend — the entire premise is that you don't have a platform team, so don't create the job. Because everything flows through your Collector speaking OTLP, this decision is reversible in an afternoon, which means you should spend a day on it, not a quarter. The mistake isn't picking the "wrong" backend; it's instrumenting against a vendor's proprietary SDK and making the decision permanent.

Structured logs that correlate with traces

Logs don't disappear in a trace-first world; they get demoted to supporting evidence — and promoted in usefulness by carrying the trace ID. When every log line includes trace_id, "find the logs for this slow request" becomes a click instead of a timestamp archaeology session, in both directions: trace → logs, log → trace.

With pino, it's a five-line mixin:

import pino from "pino";
import { trace } from "@opentelemetry/api";

export const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};
    const { traceId, spanId } = span.spanContext();
    return { trace_id: traceId, span_id: spanId };
  },
});

While you're here, adopt the rest of structured-logging hygiene: JSON output, one event per line, attributes as fields rather than interpolated into the message (logger.info({ orderId }, "payment captured"), not `payment captured for ${orderId}`), and consistent field names across services. And log less: if a span already records the operation, its duration, and its outcome, the log line saying "starting operation X" is pure storage cost.

SLOs and alerting that doesn't page you at 3am for nothing

Alerting is where observability setups go to die. The failure mode is always the same: alert on causes (CPU at 80%! memory climbing! one pod restarted!) and wake up for things users never noticed, until everyone mutes the channel and misses the real one.

The fix is two ideas from the SRE canon, both of which work fine without an SRE:

Alert on symptoms, not causes. Users experience exactly two things: errors and slowness. So alert on error rate and latency at the edges of your system — things that are definitionally user-impacting. CPU, memory, queue depth, restarts: those are dashboard material for the person debugging, never pages.

Use error budgets and burn rates instead of static thresholds. Pick a target — say 99.9% of requests succeed over 30 days. That gives you an error budget of 0.1% of requests. Instead of "page when error rate > 1% for 5 minutes" (which fires on every blip), alert on how fast you're spending the budget. The multiwindow numbers from Google's SRE Workbook are a fine starting point: page when the burn rate is ~14x budget over the last hour (you'd exhaust a month's budget in ~2 days — this is a fire), and open a ticket at ~1–2x sustained over days (a slow leak; fix it this week, sleep tonight). Requiring the elevated rate over both a long and a short window keeps a 30-second blip from paging anyone.

For a small team, the honest version is: two or three SLOs, total. Availability and latency on your main user flow, maybe one for a critical async job. Every page should mean "a user is having a bad time right now, and a human can do something about it." Anything that fails that test gets deleted, not tuned.

Dashboards people actually look at

Most dashboards are built once, screenshotted for a slide, and never opened again. The ones that survive share a shape:

• One overview per service, four to six panels, questions not vibes. Request rate, error rate, p50/p95/p99 latency, and saturation of the one resource that actually constrains you (DB connections, queue depth). If a panel doesn't change a decision, it's decoration.
• Percentiles, never averages. An average latency of 200ms happily hides a p99 of 8 seconds, and your unhappiest users all live in the tail.
• Exemplars — the killer feature to insist on from your backend: click a spike on the latency chart, land on an actual trace from that spike. This closes the loop between "something is wrong" and "here is why" without writing a single query.
• Delete dashboards. If nobody has opened it in three months, it's not documentation, it's clutter that makes the useful ones harder to find.

Observability for AI features

If you're shipping LLM-backed features — we build chat assistants and structured-output endpoints into client products routinely — model calls are the most expensive, slowest, and least deterministic spans in your system. They deserve first-class instrumentation, and OTel's generative-AI semantic conventions (gen_ai.*) give you standard attribute names to do it:

return tracer.startActiveSpan("chat completion", async (span) => {
  span.setAttribute("gen_ai.operation.name", "chat");
  span.setAttribute("gen_ai.request.model", model);
  const res = await client.messages.create({ model, max_tokens: 1024, messages });
  span.setAttribute("gen_ai.usage.input_tokens", res.usage.input_tokens);
  span.setAttribute("gen_ai.usage.output_tokens", res.usage.output_tokens);
  span.setAttribute("gen_ai.response.model", res.model);
  span.end();
  return res;
});

With token counts and model names on every span, questions that are otherwise guesswork become queries: token spend per feature per day; p95 latency per model (p99 for LLM calls is dramatically worse than p50 — cold paths, long generations, retries — and that's the number to design timeouts and streaming UX around); whether last week's prompt change moved output length; which retried calls are silently doubling cost. When you swap models, you get a before/after on latency and token usage for free instead of anecdotes. One caution: never put raw prompts or completions in span attributes by default — they're PII-dense and enormous. Sample them deliberately if you need them for evals, and scrub them in the Collector like everything else.

The minimal viable setup

The whole thing, as a checklist. Steps 1–4 are a focused week for one engineer; nothing below requires a platform team to build or to keep running:

1. Node SDK with auto-instrumentation in every service; service.name, service.version (git SHA), and environment on the resource. (half a day)
2. One Collector container receiving OTLP, with memory_limiter, health-check filtering, PII redaction, tail_sampling (errors + slow + 10% baseline), and batch. (half a day)
3. A managed backend receiving from the Collector. Spend a day choosing, not a quarter — the Collector makes it reversible.
4. Custom spans + attributes on your three most important business operations; context propagation across any queue hop. (a day)
5. Trace IDs in structured logs via a logger mixin; logs shipped somewhere that can link them to traces. (an hour)
6. Two or three SLOs on user-facing symptoms with burn-rate alerts; delete every cause-based page. (half a day, plus the political will)
7. One overview dashboard per service with exemplars wired up. (half a day)
8. gen_ai.* attributes on every model call if you ship AI features. (an hour)

Takeaways

• For request-shaped systems, traces are the highest-leverage signal — instrument them first, and derive metrics and log-correlation from them, not the other way around.
• OpenTelemetry's value is reversibility: instrument against the vendor-neutral API once, and every backend decision becomes a Collector config change.
• Auto-instrumentation is a half-day that gives you per-route latency, DB timing, and external-call visibility. Do it this week.
• Custom spans should wrap business operations with rich attributes, not every function. Attributes are what make traces queryable.
• Run a Collector even though you could skip it — it's your one choke point for redaction, sampling, and vendor routing.
• Tail sampling (all errors + all slow + small random baseline) keeps the traces that matter at a fraction of the cost. Head sampling throws away your errors at the same rate as your noise.
• Alert on symptoms with burn rates, not causes with thresholds. Two or three SLOs. A page means a user is hurting now.
• LLM calls are spans too: token usage and per-model latency percentiles via gen_ai.* attributes turn AI cost and performance from anecdotes into queries.
• The whole setup is about a week of one engineer's time — and it pays for itself the first time someone asks what happened at 4pm.