Skip to main content

> saas_observability_ingestion_budgets_&_span_indexing_quotas

SaaS Observability Ingestion Budgets & Span Indexing Quotas

How do unbounded APM spans, custom metric tags, and log indexing in SaaS observability tools (like Datadog, Dynatrace, New Relic) easily surpass your total AWS compute infrastructure bill?

Senior (L5)

THE SHORT ANSWER

Third-party SaaS observability platforms use consumption-based pricing models that heavily penalize unindexed volume: Datadog charges $1.70 per million indexed spans, $0.10 to $0.40 per 100 custom metrics, and $1.06 to $2.50 per million indexed logs. In microservice environments emitting millions of gRPC calls daily, indexing 100% of healthy trace spans and passing high-cardinality tags creates a monitoring bill that can exceed actual cloud hosting costs. Implementing tail-based sampling, APM span retention filters (keeping 100% of errors and 1% of 200 OKs), and log exclusion filters reduces SaaS observability spend by 60% to 80% without losing diagnostic fidelity.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

APM tracing separates ingestion from indexing. Ingestion receives the raw trace stream ($0.10/million spans), while indexing retains the span in the analytics database for 15 days ($1.70/million spans). Without retention filters, Datadog's Intelligent Retention filter indexes huge volumes of routine health checks and HTTP 200 calls. By configuring custom Retention Filters in Datadog APM, teams can enforce rules: `status:error OR @http.status_code:>=500` retains 100% of failures, `duration:>2s` retains 100% of latency anomalies, and `service:* AND status:ok` samples at 1-5%. Log Ingestion vs Indexing applies the same logic: ingest all logs to S3 archives for compliance, but index only warning/error levels for interactive search.

2. Appropriate Use Context

Any enterprise or scale-up utilizing commercial SaaS observability platforms (Datadog, Dynatrace, New Relic, Honeycomb) with over 50 microservices or generating >100 million requests monthly.

3. Production Failure Modes

A load test running overnight with 100% trace sampling, generating 500 million indexed spans and an unexpected $8,500 APM overage bill in one weekend; instrumenting a loop that tags custom metrics with `order_id`, instantly creating 2 million custom metrics and incurring $20,000 in monthly metric surcharges.

4. Diagnostic Signals & Telemetry

Datadog Plan & Usage dashboard showing `Indexed Spans` or `Custom Metrics` exceeding committed plan tiers; overage warning emails from SaaS account managers; monthly observability invoice exceeding 30% of total AWS compute spend.

5. Prevention & Safeguards

Set strict APM Retention Filters retaining 100% errors, 100% slow traces (p99), and 1% of routine 200 OK traces; configure Datadog Metric Summary Rules to drop high-cardinality tags at ingestion; set usage anomaly alerts in Datadog Usage Attribution; implement OpenTelemetry Collector upstream to filter and sample before sending to SaaS.

6. Architectural Trade-offs

Sampling successful traces reduces SaaS observability spend by 70%+, but means individual healthy transactions from hours ago cannot be retrieved by exact trace ID unless linked to an error or caught in the 1-5% sample.

Case Study (TinyCTO In-Field Example)

A fintech startup processing 800 million monthly API requests saw their Datadog bill surge to $28,000/month, dominated by $16,000 in Indexed Spans and $7,000 in Custom Metrics. By implementing Datadog APM Retention Filters (100% errors, 100% >1.5s latency, 2% sample of 200 OKs) and stripping user IDs from custom metrics, indexed span volume dropped by 88% and custom metrics dropped by 95%, reducing the monthly Datadog bill to $6,500 (saving $258,000/year).

Interactive Concept Drills

2 Cards
Q1

What is the difference between Ingested Spans and Indexed Spans in Datadog APM?

Ingested spans are briefly processed for live metrics ($0.10/M); indexed spans are retained for 15 days for interactive searching ($1.70/M).
Q2

What is an optimal APM retention filter strategy for microservice architectures?

Retain 100% of error spans, 100% of high-latency outliers (p99), and sample only 1-5% of normal 200 OK spans.

SaaS Observability Ingestion Budgets & Span Indexing Quotas — Technical FAQ

Can I use OpenTelemetry Collector to control SaaS observability costs?

Yes. OpenTelemetry Collector can apply tail-based sampling, span attribute scrubbing, and log filtering before data ever leaves your VPC, preventing expensive telemetry from hitting SaaS ingestion meters.

What happens if our indexed spans exceed the committed SaaS plan quota?

Most SaaS providers allow data ingestion without dropping traces, but bill on-demand overage rates that are 20% to 50% higher than contracted commitment rates.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸SaaS observability platforms bill heavily for indexed spans and custom metrics.
  • ▸Datadog indexed spans ($1.70/M) cost 17x more than ingested spans ($0.10/M).
  • ▸APM Retention Filters should index 100% of errors and sample 1-5% of successes.
  • ▸OpenTelemetry Collector upstream filtering prevents unwanted telemetry from leaving the VPC.

Common Misconceptions

  • ✗Misconception: You must index 100% of all traces to maintain full system observability (False: Aggregated metrics give full volume metrics; sampling 1-5% of OKs provides sufficient trace examples).
  • ✗Misconception: SaaS observability spend is fixed based on host count (False: Usage-based span, metric, and log indexing drives the majority of the bill).

Decision & Governance Guidance

Configure APM Retention Filters immediately on all production Datadog/SaaS services. Set strict limits on custom metric tag creation in CI/CD pipeline linters.

Authoritative Sources & Standards