Your hybrid app can survive a traffic spike while its monitoring gives you the wrong answer. Backend developers often expect the first scaling problem to be an overloaded collector or a slow dashboard. I would fix the measurement boundary first: as retries, WebViews, native code, and offline devices multiply, “requests served” stops meaning “users succeeded.” More telemetry cannot repair a denominator that was wrong at the source.
The success denominator breaks before the telemetry pipeline does
A backend HTTP 200 is a poor proxy for a completed mobile action because the response may arrive after the user abandons the screen, or the WebView may fail while rendering it. Conversely, an HTTP 503 can precede a successful retry, so counting every failed attempt as a failed user action can overstate the damage. The distinction gets harder to ignore at scale because retries and intermittent connections create more transport events per intended action.
Is your hybrid app SLO-ready in production, and how do you know makes production readiness the question; I would settle what counts as one eligible user action before choosing its SLO threshold. Define a small event contract for a named action, such as “open account summary”: an action ID is created at the tap, a terminal success requires usable content on screen, and a terminal failure requires an error or a timeout that the user actually encounters. A server span can support that account of events, but it cannot establish the screen outcome by itself.
Separate logical actions from network attempts in storage. Generate one operation ID per action, carry it across retries where possible, and retain attempt IDs beneath it. Do not put either ID in a Prometheus label: per-action label values would create a new time series for nearly every interaction. Put IDs in traces or bounded-retention events instead, and label aggregate metrics with a deliberately small set of dimensions such as action, platform, and release.
Offline behavior needs an explicit rule because a device that queues work can otherwise report an apparent success long after the user needed it. For an illustrative policy, treat a result arriving more than 30 seconds after the tap as outside the foreground “open summary” objective; that deadline is a value to tune against the action’s UX contract, not a universal mobile standard. Record queued background synchronization separately, since merging it with foreground opens lets a healthy queue hide a broken screen.
Instrument both sides of a hybrid boundary. In a WebView, the JavaScript PerformanceObserver API can record rendering-related events, while native lifecycle callbacks can tell you whether the screen was backgrounded before completion. Where requests cross from WebView code into a backend, propagate W3C Trace Context Level 1’s traceparent header when feasible; a trace without a parent across that boundary looks like two unrelated operations. If browser CORS applies, allow the header deliberately rather than assuming propagation works. Keep user identifiers out of W3C Baggage, since baggage travels with requests and may reach services that do not need that data.
Cardinality, not dashboard count, consumes the first surprise budget
The next failure is often a metric that was cheap in a test environment and expensive across real releases. Suppose an application emits a latency histogram by endpoint, platform, release, region, and device model. Each additional label multiplies the number of series, while each histogram bucket adds another series per combination. Prometheus and Grafana can display the resulting data, but neither can make an unbounded device-model or URL-path label inexpensive.
Top Application Monitoring Metrics for Better Observability is a useful checklist; I would reject any metric on that checklist whose labels cannot be bounded before rollout. For an adjustable initial limit, allow at most 20 named user actions in the SLO metric and route unknown action names into an “other” value; the exact cap should follow your product, but a cap makes a new release’s telemetry cost predictable. Normalize URLs to route templates, and never label by account ID, operation ID, raw exception message, or arbitrary query string.
Use different instruments for different questions. A Prometheus counter for eligible actions and a counter for failed actions give an unsampled error ratio. A histogram of action duration supports latency calculations with PromQL functions such as rate() and histogram_quantile(), provided its buckets match durations users notice. OpenTelemetry spans exported over OTLP/HTTP to an OpenTelemetry Collector can explain a particular failure; they should not replace the counters, because sampled traces cannot be assumed to represent every action.
Prometheus histograms versus trace-only latency: histograms win when you need a stable fleet-level alert, at the cost of bucket series and less detail about any one request. Trace-only latency wins when investigating a specific slow path, at the cost of sampling bias and incomplete coverage. I would not compute an SLO from sampled Jaeger or Grafana Tempo traces, because changing the sampling policy could change the reported SLO without changing the user experience.
Put an explicit memory budget on the Collector as well. The OpenTelemetry Collector’s memory_limiter processor, including limit_mib and check_interval, can bound memory use, while the batch processor’s send_batch_size and timeout trade export efficiency against delay. Those controls protect the pipeline, but dropped spans still reduce diagnostic coverage; watch exporter failures and refused data alongside application metrics so a quiet trace view is not mistaken for a quiet incident.
Fleet-wide averages hide the release that is failing
A hybrid rollout rarely reaches every device at once. An aggregate success rate can stay healthy while a new Android release fails on a narrow WebView version, because the older healthy population supplies most of the denominator. Slicing everything by every device attribute is not the answer, however, because that recreates the cardinality problem and leaves many slices too small to interpret.
Start with dimensions that correspond to a decision someone can make: platform, app release, action, and perhaps a coarse WebView or OS cohort. Keep detailed device attributes in events or traces with controlled retention. Android Vitals and iOS MetricKit provide platform-side signals for crashes and responsiveness, but neither substitutes for your named action outcome because a screen can fail without crashing or producing a platform-defined hang. Web Vitals metrics such as Interaction to Next Paint can help within web content, yet they do not measure a native navigation step that happens before the WebView receives input.
Small cohorts also make alarming percentages easy to produce. The following Python 3 snippet runs without dependencies; its counts are illustrative, and its minimum-volume guard is an operational choice to tune. It reports burn against the same success target while refusing to call a tiny cohort conclusive:
samples = {"android/new": (9, 180), "ios/new": (12, 2400)}
target = 0.995
minimum_actions = 1000
for cohort, (failures, actions) in samples.items():
error_budget = actions * (1 - target)
burn = failures / error_budget
if actions < minimum_actions:
print(f"{cohort}: investigate ({actions} actions, burn {burn:.1f}x)")
else:
print(f"{cohort}: burn {burn:.1f}x")
Here, 99.5% is an example target to choose with the team, not a measured property of either cohort. The 1,000-action guard is likewise a starting threshold for this example, not proof that a cohort above it is statistically safe. The Android slice deserves investigation because its failures are concentrated, but an automatic rollback based on that small count alone could mistake a brief cluster for a sustained regression. Pair cohort burn with volume, time window, and a release comparison.
Alert design should preserve that distinction. An aggregate alert can catch widespread failures; a release-cohort alert can catch concentrated ones if it also requires enough eligible actions. Add a separate signal for missing client reports, because a broken SDK or blocked upload path can make a failing release appear unusually successful. Compare client action counts with backend operation counts where the action requires a request, while allowing for offline actions and delayed uploads rather than demanding exact equality.
Peak traffic turns diagnostic detail into an incident of its own
During a failure, error logs and spans often grow faster than successful traffic because each retry produces more attempts and each exception produces more attributes. A collector sized against normal request volume can therefore lose the evidence you most want during an incident. Fixed-probability head sampling helps bound cost, but it can miss a rare failing cohort; tail sampling can retain error traces, but it must hold trace data until its decision window closes and so needs memory sized for peak concurrent traces.
Make the failure mode explicit before enabling either policy. For head sampling, retain unsampled counters for outcomes and consider a higher trace probability for a bounded set of critical actions. For OpenTelemetry Collector tail_sampling, set decision_wait and num_traces against a load test that includes retries and long mobile calls; otherwise the policy can exhaust its own buffer precisely when traces become useful. Exporter queues and backpressure should be visible to on-call staff, because application errors and observability loss can happen together.
Do not respond to missing traces by logging full HTTP bodies. That raises storage and privacy exposure while often adding little to a timeout investigation. Log a bounded failure category, route template, release, attempt ID, and operation ID where policy permits; use a trace for timing across services and a client event for the screen outcome. Sentry can help group client exceptions, and Loki can help search bounded logs, but neither should be the sole source of the action denominator because ingestion gaps would then look like changes in user behavior.
Test this under load with two simultaneous faults: degrade a backend dependency and delay telemetry export. Watch whether the user-action counters remain available, whether Collector refused-data metrics rise, and whether the affected release still appears in your cohort view. This test is more valuable than proving a dashboard loads at peak traffic, because it checks whether the evidence remains trustworthy when the application and its diagnostic path are both stressed.
The first fix should be one action contract
Pick one high-traffic action this week and write down its tap, terminal success, terminal failure, retry, and offline rules. Implement its bounded counters and compare a client-observed outcome with the corresponding backend attempts during a load test. If those numbers disagree, investigate the boundary before adding another dashboard: at scale, a reliable account of one action is more useful than a comprehensive view of the wrong events.
