Application monitoring is working only when it changes production outcomes during incidents. I take a stricter position than most teams: instrumentation coverage, dashboard polish, and trace volume are weak evidence because they can improve while on-call engineers still guess. The real test is whether the system shortens the path from symptom to safe action under load, deploy churn, and partial failure.
Coverage is a vanity metric unless it predicts incident behavior
Top Application Monitoring Metrics for Better Observability correctly prioritizes latency, error rate, saturation, and traffic, but I would demote any metric that cannot change a deploy, rollback, capacity action, or alert route because production engineers pay for every emitted series with cardinality, storage, and attention.
The measurement I trust first is empirical: review the last 25 production pages and count how many alerts named the failing user-visible capability before a human opened a dashboard; my initial pass/fail line is 70%, a threshold to tune because a platform with many batch jobs will have different symptoms than an API gateway. If the number is lower, more dashboards will not fix it because the first responder is already starting in the wrong room.
I would not score success by “percentage of services with dashboards,” because a service can have a Grafana 10.4 dashboard, a Prometheus 2.50 scrape target, and OpenTelemetry spans while still hiding the customer-impacting failure behind labels like handler=”/v1/{id}”. I would also not celebrate “100% trace coverage” unless spans carry useful resource attributes, because a trace without service.name, deployment.environment, k8s.namespace.name, and cloud.region usually proves that something was slow without proving who owns it.
The production scorecard should use a small set of outcome metrics:
- Alert precision: the share of pages that required action; target 85% as a value to tune, because below that the team learns to distrust Alertmanager, PagerDuty, Opsgenie, or Splunk On-Call.
- First useful clue time: the measured time from page creation to a correct hypothesis in Slack, incident notes, or Jira; a practical starting target is under 5 minutes for Sev-2 incidents because longer delays usually mean the alert lacked service, dependency, or blast-radius context.
- Diagnostic join rate: the percentage of incidents where the engineer moved cleanly from metric to log to trace using the same correlation ID; 80% is an aggressive tuneable goal because W3C Trace Context only helps when traceparent survives proxies, queues, and background workers.
- Rollback confidence: the fraction of deploy-related incidents where the monitoring proved improvement within one Prometheus evaluation cycle; use 2 evaluation intervals as a working limit because longer waits make canary and rollback decisions political instead of observable.
Those numbers are deliberately operational rather than architectural because DevOps engineers support failure, not diagrams. A Kubernetes 1.29 cluster with kube-state-metrics, cAdvisor, node_exporter, and Envoy 1.29 sidecars can still be poorly observed if the responder cannot answer “which tenant, which version, which dependency, and whether rollback helped.”
The useful score is the responder’s path through one real failure
Application Monitoring and Observability Best Practices recommends practices that are reasonable as defaults, yet production support should judge them by incident evidence rather than adoption because a perfect-looking OpenTelemetry rollout can still leave the responder without the broken tenant, pod, or dependency.
Pick one common production failure and walk it end to end. For an HTTP API, inject a 5xx spike, a downstream timeout, or a latency regression in one zone. The test passes only if the alert fires, the dashboard shows blast radius, logs expose the error class, traces identify the dependency, and the runbook points to a reversible action. This is more reliable than a static checklist because it tests the handoff between tools, which is where most observability programs fail.
Use concrete production artifacts. Prometheus should expose RED metrics such as http_requests_total, http_request_duration_seconds_bucket, and http_requests_in_flight. Infrastructure views should include USE signals such as CPU utilization, memory pressure, disk saturation, and network errors from node_exporter. Traces should arrive through OTLP/gRPC on port 4317 or OTLP/HTTP on 4318, and they should follow OpenTelemetry semantic conventions 1.25.0 closely enough that http.request.method, http.response.status_code, db.system, and messaging.system mean the same thing across languages.
The score is not “did the graphs move”; the score is “did the engineer know what to do.” In a game day, time the responder without giving hints. If they jump from Grafana to Loki 2.9 to Tempo 2.4 or Jaeger 1.53 using a trace ID copied from a log line, the correlation path works. If they paste a pod name into five tabs and still cannot tell whether the error is caused by Redis, PostgreSQL, Kafka, or an upstream REST API, the instrumentation is decorative because it increases visible data without reducing uncertainty.
Prometheus has a documented default global.scrape_interval of 1 minute, a vendor-published starting point that is often too slow for short canaries because a bad deploy can finish harming users before two samples confirm the slope. For rollout safety, I would tune scrape intervals for edge and API SLO metrics to 15 seconds where cardinality is controlled, because faster signal reduces rollback hesitation; I would leave low-value host metrics at 60 seconds because high-frequency CPU graphs rarely change the incident decision.
A runnable Prometheus rule makes the measurement enforceable instead of aspirational:
groups:
- name: api-slo
rules:
- alert: ApiHighErrorBudgetBurn
expr: |
(sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="api"}[5m]))) > (14.4 * 0.001)
for: 2m
labels:
severity: page
annotations:
summary: "API is burning the 30-day 99.9% error budget too fast"
The 99.9% objective over 30 days is a value to tune, not a moral standard, because an internal admin API and a revenue-critical checkout path do not deserve the same error budget. The 14.4 burn multiplier is also adjustable, because a short-window page should catch fast user harm while avoiding a page for one bad scrape sample.
Metrics should page you; traces should prove the cause
The most defensible disagreement in production observability is that metrics, not traces, should own most paging. Metrics win for paging because they are cheaper to aggregate, easier to alert on with PromQL, and less sensitive to sampling. Traces win for proof because a single request path can show that a 900 ms p95 latency jump came from payments-db, a Kafka rebalance, or an Envoy retry storm.
This division gives you a measurable design rule. Every page must include one metric expression, one owning service, one user-visible symptom, and one suggested first query. If an alert cannot point to a PromQL expression or SLO burn calculation, it is probably a symptom from logs or traces that should be downgraded to investigation, because paging on high-cardinality events creates noisy incidents that cannot be budgeted.
Use histograms carefully. Native histograms in Prometheus are useful when your client library supports them, but classic buckets still work with histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) because they are portable across older exporters and remote storage. A tuneable latency rule such as “p95 under 300 ms for read endpoints” is valid only when it reflects user tolerance, because a background reconciliation job may be healthy at 3 seconds while a login endpoint is broken at 800 ms.
Traces need their own success measures. With OpenTelemetry Java agent 1.32.0, Micrometer 1.12, Spring Boot 3.x, Go otelhttp, or .NET OpenTelemetry SDK, check whether spans contain route names rather than raw URLs, because raw IDs in span names explode cardinality and make service maps expensive. Check whether tail sampling in the OpenTelemetry Collector keeps errors and slow requests, because head sampling at 1% can drop the only trace that explains a rare failure. Treat 10% trace sampling for critical APIs as a starting sample rate to tune, because storage cost and incident rarity move in opposite directions.
Logs should be judged by joinability, not volume. Loki, Elasticsearch, OpenSearch, Vector, Fluent Bit 3.x, and OpenTelemetry Collector filelog receivers can all move lines, but production value appears when logs include trace_id, span_id, service.name, version, and a stable error code. A measured log quality target I like is 90% of error logs carrying a trace ID, because below that the incident path breaks exactly when the responder needs detail.
Cardinality is the hidden tax. A metric label such as user_id, request_id, or raw path can multiply time series until remote_write, Thanos, Cortex, Mimir, or VictoriaMetrics becomes the incident. Limit high-cardinality dimensions to traces and logs because those stores are designed for search and sampling, while metrics databases are designed for repeated aggregation over bounded label sets.
Your tooling choice is valid only if its failure modes fit your team
Prometheus plus Alertmanager and Datadog APM are both legitimate, but they win in different operating models. Prometheus plus Alertmanager wins when you need control, Kubernetes-native discovery, PromQL transparency, and predictable open-source behavior; it costs engineering time, storage planning, rule maintenance, and on-call ownership of the monitoring stack. Datadog APM wins when you need fast onboarding, managed correlation, hosted retention, and broad integrations; it costs recurring license spend, ingestion discipline, and some vendor-specific query habits.
The price comparison is not abstract. Datadog has published APM host pricing around $31 per host per month on annual billing, a vendor-listed figure that becomes meaningful when multiplied by autoscaling nodes, sidecars, and environments. Prometheus has no license fee, but “free” is misleading because long retention with Thanos object storage, Mimir blocks, or VictoriaMetrics clusters requires people who understand compaction, retention, relabeling, and query limits.
Use the same acceptance test for both options. Can the platform show a deploy marker, error-budget burn, p95 latency, exemplar-linked trace, and logs for the same request without a responder inventing a correlation path? Grafana exemplars with Tempo can do this in an open stack, and Datadog service pages can do it in a managed stack. The cheaper choice is the one your team can operate during an outage, because a tool that needs a specialist at 03:00 has transferred cost from procurement to incident duration.
I would not run two full observability platforms indefinitely, because duplicate agents, duplicate alerts, and mismatched retention create contradictory evidence during incidents. A short migration overlap is fine, because you need parity checks, but a permanent split makes every Sev-1 start with a debate about which graph is true. If compliance or acquisition history forces two systems, define one source of truth for paging and one for forensic search, because responders need hierarchy more than optionality.
The final measurement is resilience of the monitoring stack itself. Alertmanager 0.27 should be clustered or backed by a managed paging route, because a single alerting process is a silent single point of failure. The OpenTelemetry Collector should expose its own otelcol_receiver_refused_spans, otelcol_exporter_queue_size, and otelcol_exporter_send_failed_spans metrics, because a saturated collector makes applications look healthy by dropping the evidence. Grafana dashboards should have saved variables and pinned time ranges for incident views, because a responder should not debug the dashboard before debugging production.
Start by deleting one noisy signal
Pick one service that pages often and audit the last month of alerts before adding anything new. Delete or downgrade one alert that did not lead to action, add one SLO burn rule for a user-visible path, and require trace-log correlation for the next incident review. That small change proves whether observability is working where it matters: the next tired engineer making a production decision.
