Skip to main content

> cloudwatch_custom_metrics_high-cardinality_billing_explosions

CloudWatch Custom Metrics High-Cardinality Billing Explosions

Why does publishing dynamic tags (like User IDs or Transaction IDs) into Amazon CloudWatch custom metrics result in thousands of dollars in unexpected metric billing?

Senior (L5)

THE SHORT ANSWER

AWS CloudWatch bills custom metrics at $0.30 per metric per month (scaling down to $0.02 at extreme scale). In CloudWatch, every unique combination of metric name and dimension key-value pairs is classified and billed as a distinct custom metric stream. If developers emit a metric tagged with a high-cardinality dimension such as `userId`, `orderId`, or `ipAddress`, emitting 100,000 unique user requests creates 100,000 distinct custom metrics, generating a $30,000/month surprise bill. High-cardinality data must be routed to structured logs (queried with Logs Insights or OpenTelemetry) rather than dimensional metric time-series.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

CloudWatch metric identity is defined by the tuple `(Namespace, MetricName, Dimensions)`. Adding `{DimensionName: 'UserId', Value: 'usr_123'}` instantiates an independent time-series that retains storage for 15 months. Even if a user ID is only active for a single second and emits one data point, AWS charges for that metric stream for the entire prorated month. Embedded Metric Format (EMF) allows developers to emit high-cardinality data in JSON logs where specified keys are extracted as low-cardinality CloudWatch metrics while high-cardinality contextual attributes remain free in log fields.

2. Appropriate Use Context

Low-cardinality operational health indicators: HTTP status codes (2xx, 4xx, 5xx), service names, AWS regions, deployment stages (prod, staging), and API route paths (e.g. `/api/v1/checkout`). Never use unbounded identifiers.

3. Production Failure Modes

A developer instrumenting an e-commerce microservice publishes `PaymentLatency` with `Dimension: CustomerId`, creating 500,000 metric streams during a holiday sale and generating a $150,000 monthly CloudWatch metric bill; Lambda functions dynamically generating metrics per invocation ID.

4. Diagnostic Signals & Telemetry

AWS Cost Explorer showing exponential spikes in `CloudWatch:MetricMonitorUsage`; CloudWatch Metrics console showing hundreds of thousands of active custom metric names with single data points.

5. Prevention & Safeguards

Adopt AWS CloudWatch Embedded Metric Format (EMF) to strictly separate dimensional metrics from raw log properties; implement CI/CD linting checks against metric instrumentation code forbidding variable dimensions; set AWS Budgets anomaly alerts on CloudWatch metric count.

6. Architectural Trade-offs

Aggregating metrics to low cardinality enables cheap, real-time CloudWatch dashboarding and alarms, but requires falling back to log analytics (Athena, CloudWatch Logs Insights, or ClickHouse) to drill down into specific user or transaction IDs.

Case Study (TinyCTO In-Field Example)

A SaaS company emitted custom API latency metrics tagged with `tenantId` and `clientIpAddress`. With 250,000 active client IPs monthly, their CloudWatch metric bill reached $75,000/month ($0.30 * 250,000). By migrating to AWS Embedded Metric Format (EMF), they retained `clientIpAddress` as a searchable log property while only indexing `tenantId` and `httpStatus` as metric dimensions, collapsing metric count to 120 streams and slashing the monthly bill to $36.

Interactive Concept Drills

2 Cards
Q1

What determines a unique custom metric stream in AWS CloudWatch?

Every unique combination of Namespace, Metric Name, and Dimension Key-Value pairs.
Q2

How does AWS Embedded Metric Format (EMF) solve high-cardinality metric costs?

It writes high-cardinality attributes into standard JSON logs while extracting only low-cardinality keys as CloudWatch metrics.

CloudWatch Custom Metrics High-Cardinality Billing Explosions — Technical FAQ

How much does a custom CloudWatch metric cost per month?

$0.30 per metric-month for the first 10,000 metrics, scaling down to $0.02 per metric-month for over 1.5 million metrics.

If a custom metric receives only one data point in a month, is it billed for the full month?

Yes. Once created, CloudWatch treats the metric stream as active and charges the pro-rated monthly rate.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸CloudWatch charges $0.30 per custom metric stream per month.
  • ▸Each unique dimension value creates a distinct paid metric stream.
  • ▸Dynamic high-cardinality dimensions (User IDs, UUIDs, IPs) cause exponential billing spikes.
  • ▸AWS Embedded Metric Format (EMF) separates metrics from high-cardinality log attributes.

Common Misconceptions

  • ✗Misconception: Dimensions are free metadata tags on metrics (False: Every unique dimension value combination is a new billed metric).
  • ✗Misconception: CloudWatch automatically deletes metrics with no recent data (False: Metric metadata is stored for up to 15 months).

Decision & Governance Guidance

Never use dynamic entity IDs (user, order, IP) as CloudWatch metric dimensions. Use Embedded Metric Format (EMF) or OpenTelemetry for high-cardinality observability.

Authoritative Sources & Standards