Your AWS Logs Never Expire — and CloudWatch Has Other Meters Running

The second AWS bill nobody budgets for

Ask a team what their AWS bill is made of and they'll say EC2, RDS, S3 — the services they chose. Ask about CloudWatch and you'll usually get a shrug: it's "just monitoring," a background utility that comes with the platform.

Then one month it's the third-largest line item, and nobody can say why.

CloudWatch doesn't bill like a normal AWS service. It bills on about six different meters at once — log ingestion, log storage, custom metrics, metric API calls, alarms, dashboards — and several of its defaults are designed in a way that makes costs grow linearly forever unless someone intervenes. This post is about the meters you probably didn't know were running.

Meter 1: log groups never expire by default

This is the quiet one. When a log group is created — by Lambda automatically, by the console, by your SDK — its retention setting is Never expire. Not 30 days, not 90 days. Forever.

Storage is billed at $0.03 per GB per month. That sounds like nothing until you do the compounding math:

The storage line grows forever. Year five costs five times year one — same app, same traffic, same code. Nothing is wrong; it's just a default nobody revisited.

Check your own account — this lists every log group with its stored bytes and retention (a blank retention column means "never"):

aws logs describe-log-groups \
  --query 'logGroups[].{Name:logGroupName,StoredBytes:storedBytes,Retention:retentionInDays}' \
  --output table

Fix everything missing retention in one pass — 30 days is a sane starting point for operational logs:

aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' \
  --output text | tr '\t' '\n' | while read -r group; do
  aws logs put-retention-policy --log-group-name "$group" --retention-in-days 30
done

If compliance requires long retention, export to S3 instead — S3 storage is cheaper per GB and has lifecycle policies to Glacier. "Logs we might need for an audit" is a real requirement; "logs we might grep someday" usually isn't. (AWS has also added automatic storage tiering for older, untouched log data — helpful, but it discounts storage rather than ending it, and does nothing for ingestion.)

Meter 2: ingestion is the expensive half

Storage is $0.03/GB/mo. Ingestion is $0.50/GB — over 16× the storage rate, paid on every uncompressed byte the moment it arrives.

Which means the real cost lever isn't retention, it's what you choose to log. The classic failure modes:

• A DEBUG level that never got turned back off. One forgotten flag on a busy service can 10× its log volume permanently.
• Health checks and 200 OKs. Load balancer health checks hitting an endpoint that logs every request, generating gigabytes of "all fine" per day.
• Full stack traces per request. Useful in staging; at production QPS it's a billing generator.
• Vended logs nobody sized. VPC Flow Logs, Route 53 resolver logs, etc. get tiered delivery pricing (still starting at $0.50/GB for the first 10 TB) — but a busy VPC with ALL traffic logging produces a lot of bytes.

For logs that exist mainly for forensics — rarely queried, kept "just in case" — the Infrequent Access log class cuts ingestion to ~$0.25/GB. The catch: the class is chosen at log group creation and can't be changed later, and it drops features like metric filters and subscription filters. For new log groups holding cold audit data, it's a straight 50% discount.

Meter 3: custom metrics bill per dimension combination

Custom metrics are $0.30 per metric per month — where "metric" means every unique name + dimension combination. Emit request_latency{endpoint=/users, method=GET} and you have one metric. Emit it with a user_id or request_id dimension and you've just created a billing multiplier: every distinct value is a new metric, forever, whether anyone ever looks at it.

This is how teams discover they're paying for 40,000 metrics — not because they wrote 40,000 metrics, but because one metric had a high-cardinality label.

Count yours:

aws cloudwatch list-metrics --output json | jq 'length'

If the number is in the tens of thousands, go find the dimension that's exploding before looking at anything else.

Meter 4: the polling tax — how third-party monitoring bills happen

Every call to GetMetricData or GetMetricStatistics costs $0.01 per 1,000 metrics requested. Sounds trivial — until you realize this is exactly how Datadog, Grafana Cloud, and every other external observability tool reads your AWS metrics.

A tool polling 5,000 metrics every minute makes ~7.2 million metric-requests a day — about $72/day, $2,160/month, for read-only polling alone. This is the mechanism behind most "our monitoring bill is bigger than our staging environment" stories. The fix is boring: poll less often, poll fewer metrics, or use metric streams (Kinesis-based, priced differently) for high-volume metric export.

Meter 5–6: the small ones that still add up

• Alarms: $0.10/month each at standard resolution — cheap until an autogenerated setup creates hundreds per service.
• Dashboards: first three are free; after that $3/month each.
• Logs Insights: $0.005/GB scanned — a query against a multi-TB log group without a tight time range is a real charge per run.
• Detailed monitoring on EC2: ~$2.10/instance/month — frequently enabled account-wide "just in case" and never revisited.

Why this never shows up in waste scans

CloudWatch is a blind spot for most cost tooling, ours included: GreenOps Scan currently covers compute, storage, databases, and networking — not observability spend. That's the honest gap this post exists to close: the checks above are a handful of CLI commands, and every dollar they find is pure margin.

The pattern underneath is the same one as everywhere else in cloud waste, though: a reasonable default, set once, never revisited, compounding quietly. Logs that never expire. Metrics nobody queries. A monitoring tool polling everything it can see. None of it shows up as a spike — which is exactly why it works.

npx greenops-scan

The CLI is free, read-only, and local. It won't audit your CloudWatch bill yet — but it will find the idle compute and storage sitting next to it, which is usually where the bigger number is hiding anyway.

Tags: #AWS #CloudWatch #FinOps #Observability #CloudCostOptimization #Logging #GreenOps