PostHog durable stream cost investigation — 2026-07-18

Status: mitigated, with the complete Group Analytics model prepared on this branch. Production keeps the complete durable feed. Local and preview exports are disabled by application code, and a reversible staging ingestion transformation protects existing preview deployments while they update.

Decision#

• Keep every non-ephemeral production stream fact in PostHog.
• Associate every production stream fact with its project group. Do not read the project directory merely to derive an organization for the machine feed. Accept identified-event and Group Analytics processing for this feed as an intentional product cost.
• Identify signed-in browser users as people. Use Group Analytics for their UI activity and curated product events, with organization and project as the group types.
• Do not create a user group. The identified person already is the user.
• Do not export the synthetic durable feed from local development or preview deployments.
• Keep the staging Drop Events transformation enabled until every live preview deployment has the application-side production gate.
• Treat the production error and parked-subscription events found during this audit as product defects to follow up, not as analytics noise.

The emergency code change is PR #2119.

Executive summary#

The spike was not caused by event payload size, PostHog retries, or one runaway stream. It was the multiplicative effect of exporting the full synthetic stream history generated by CI across all nine preview workers.

During the fixed 24-hour window from 2026-07-17 07:12 UTC to 2026-07-18 07:12 UTC:

The staging rate projects to 86.9 million events per 30-day month if sustained. Using PostHog's public July 2026 Product Analytics tiers, the base analytics charge alone projects to about $2,332/month, or $77.73/day. Because the stream events currently create person profiles, identified-event processing projects to another roughly $2,902.20/month, or $96.74/day. Group Analytics can add a third charge. The actual invoice depends on organization-wide tier allocation, enabled add-ons, discounts, and the partial month; the API available to this investigation does not expose the invoice.

The post-fix production feed ran at about 42,944 stream events/day, or 1,288,320 events per 30-day month. At PostHog's 2–15 million event marginal rates, that is about $44.19/month for base Product Analytics alone. Treating every row as identified raises the same volume to about $133.99/month; Group Analytics raises it to about $172.63/month. If this feed were the organization's only volume and therefore crossed the first-million-free boundary itself, the corresponding estimates would be $14.42, $71.50, and $91.97.

At exactly 40,000 grouped events/day (1.2 million/month), the combined public price is approximately $63.80/month when the organization's one-million free allowances are otherwise unused. If other traffic has already consumed those allowances and this feed occupies the next million's higher marginal band, the same feed is approximately $345.80/month. If all of it lands in the 2–15 million marginal band, it is approximately $160.80/month. Grouped 40k/day therefore does not intrinsically cost $300; the organization's prior usage determines which tier each row occupies. The preview flood distorted the current month by consuming the cheap bands before production traffic reached them.

For the 41,331 production stream rows in the fixed 24-hour window, the 2–15 million marginal costs are approximately $1.42 base-only, $4.30 with identified event processing, or $5.54 with both identified and group processing. The number “41k” is an event count, not a £41,000 charge.

PostHog bills Product Analytics by captured event count, not JSON bytes. The large GitHub webhook payloads explain storage and query weight, but not the event-count spike.

What “used person profiles” means#

This PostHog invoice line is driven by the number of identified events processed, not by the number of unique people stored. The fixed production window used only 107 stable synthetic project distinct IDs, but all 41,331 stream rows were associated with those profiles. That is why the usage looked like tens of thousands of “profiles” even though there were not tens of thousands of users.

The synthetic project IDs therefore remain intentional identified profiles. Every production stream event uses the same stable synthetic identity for its project and carries only PostHog's project $groups mapping. The exporter does not read the project directory to derive an organization for machine traffic. This is the cost required for the raw durable facts to participate in project-level funnels, cohorts, breakdowns, flags, and experiments.

Signed-in web UI events are different. They should have a real person profile so funnels, cohorts, person properties, flags, and cross-session behavior can use the authenticated user. The browser now calls identify(user.id, …) and then attaches organization and project groups. It resets identity on sign-out and clears/replaces groups when route context changes.

Timeline#

Why staging was so large#

Every preview slot contributed substantial volume. This rules out a single bad worker and points to shared CI behavior.

No individual preview project dominated the staging project. The five largest were:

The largest accounted for only 0.08% of staging stream volume. The largest individual project/stream pairs were likewise recognizable E2E fixtures, and every stored row had a distinct event UUID:

The largest staging event types were lifecycle fan-out:

Root streams, /repos/config, /scheduler/primary, and /integrations/email were the most common paths on multiple workers. The distribution matches preview E2E fixtures creating projects and their standard child streams. It does not match one stream appending the same fact indefinitely.

Development was much smaller: one local worker (os) produced 21,116 stream events across 209 projects and 1,649 project/stream pairs. It is still synthetic data with no production analytics value, so the same production-only gate applies. Its largest project (prj_4bfdd6bb…) contributed 676 events across 19 streams, and its largest individual stream was a different project's root with 420 events. All 420 had distinct event UUIDs, so development also showed fixture fan-out rather than a retry loop.

Production attribution#

After the ephemeral fix, from 2026-07-17 15:10 UTC through 2026-07-18 07:12 UTC, production stored 28,689 stream events. The top projects were:

The production recreation at 2026-07-18 05:31 UTC explains the two Iterate project IDs; they are one logical project across the reset.

The old Iterate GitHub installation stream was the largest individual stream: 7,465 events and 145,898,898 original bytes after the ephemeral fix. The largest individual streams were:

The UUID equality in this table is further evidence that these were distinct durable facts, not duplicate capture retries.

GitHub webhooks were the largest event type by bytes and count:

This is useful production telemetry, so it remains enabled.

Payload size#

For the 28,689 post-fix production rows:

3,231 rows (11.26%) exceeded 16 KiB. This matters for Cloudflare Workers Analytics Engine even though it does not affect PostHog event-count billing.

Nine events exceeded the application's 100 KiB boundary and were deterministically truncated before capture. Their original size and truncation flag remain indexed. Payload size deserves storage/query monitoring, but shrinking these objects would not materially change Product Analytics event-count billing.

Cloudflare ClickHouse alternative#

Workers Analytics Engine is Cloudflare's ClickHouse-backed analytics product, not a general ClickHouse database. At this volume its proposed included write allowance would cover the feed, and its SQL API is useful for aggregate operational telemetry. It is not an exact replacement for this feed: it has a 16 KiB blob budget, three-month retention, and adaptive sampling that can make individual rare records unavailable. 11.26% of the measured events already exceed its blob ceiling.

An exact Cloudflare-native alternative is to stream raw events through R2 Pipelines into R2/Iceberg and query them with R2 SQL or an external ClickHouse. That is likely inexpensive at the observed data volume, but it means building or operating the segmentation UI, joins, funnels, cohorts, retention queries, and identity model that PostHog supplies.

The practical split is therefore:

• PostHog for curated product analytics, browser identity, organization/project Group Analytics in the browser, and project Group Analytics for durable stream activity.
• Grouped stream:append events while their product value justifies the full base, identified-event, and Group Analytics cost. Use the native project group for project slicing; type and path are promoted event properties (stream_event_type, stream_path) rather than encoded into the event name.
• R2/Iceberg as the better future home if exact long-retention raw history is required independently of PostHog.

Runaway-loop and retry checks#

No unbounded ingestion loop was found:

• All 41,331 production stream rows in the fixed 24-hour window had distinct stream_event_uuid values. There were zero duplicate stored UUIDs.
• Every one of the 6,547 post-fix stream/woken events had a distinct (project, stream, incarnationId) tuple. Wake bursts aligned with production deploys and Durable Object incarnations, not repeated delivery of one wake fact.
• Every post-fix GitHub webhook event had a distinct stream idempotency key.
• Staging volume was distributed across nine workers, thousands of projects, and hundreds of thousands of streams. Per-project counts were consistent with normal project birth and E2E fixture fan-out.
• Ephemeral leakage stopped completely after the revised subscription and capture-side filter deployed. It was a bounded rollout defect, not an ongoing source of the spike.

Deploy-driven stream/woken volume is real operational churn and should be monitored per deploy, but it is neither a retry storm nor the main staging cost driver.

Error classification#

The full feed exposed bounded production defects that were previously harder to see:

• 42 stream/error-occurred events belonged to the pre-recreation Iterate project. They record rollout incompatibilities such as missing worker SDK exports, RPC serialization/version errors, poison-event skips, and processor revival backoff. The last was stored at 23:21 UTC.
• Two events in agent-fleet are current contract violations: support and release agents produced agent/runtime-changed without runtime.runningScripts. These need an owner and regression tests.
• 23 stream/subscription-parked facts were bounded and loud: 17 on the old Iterate project and two each on three short-lived production test projects. Causes included the rollout incompatibilities, unseeded repositories, a memory-limit reset, and an unavailable commit.

These events do not explain the cost spike and do not show an unbounded loop. They must still be resolved or explicitly modeled under the repository's no-deviant-behavior principle.

Immediate controls and rollback#

Application gate#

capturePosthogStreamEventBatch now sends only when WORKER_SELF === "os-prd". Non-production deliveries return successfully before calling PostHog, so existing durable subscriptions advance normally instead of retrying or parking.

Preview acceptance verified that behavior against os-preview-3: the E2E project's iterate-platform-posthog subscription acknowledged all 25 offsets with lag: 0, attempt: 0, lastError: null, and parkedAtOffset: null. Cloudflare telemetry for the deployed version contained zero posthog.capture_stream_events spans and zero PostHog-related errors.

Rollback: revert PR #2119. This would restore local/preview export, so do not roll it back while their PostHog keys remain configured unless another ingestion filter is already active.

Staging ingestion transformation#

The enabled staging-only PostHog Drop Events transformation is:

• ID: 019f741a-2dd8-0000-61e9-73abda6505fe
• Name: Emergency: drop preview stream events
• Filter: event = 'stream:append' OR event LIKE 'append:%' (legacy dual match; production capture is now only stream:append)
• Execution order: 0
• Function: return null

PostHog documents that dropped transformation events do not count toward usage or billing. The transformation is reversible, but dropping is permanent for matching events; those staging facts cannot be recovered from PostHog.

Keep it enabled until all preview slots and long-lived PR preview branches have picked up the production-only application gate. Then disable it and run a controlled staging capture check before deletion. Production project 115112 was never modified.

Recommendations#

1. Keep full grouped non-ephemeral production coverage. The current post-fix volume is bounded and has already exposed valuable lifecycle and defect evidence. Every row should join its project group.
2. Keep dev/preview stream export off. Preview E2E data belongs in test logs and traces, not paid organization analytics. Ordinary browser analytics can remain separated in the existing development/staging projects.
3. Create daily event-volume alerts per PostHog project. Alert on both total events and stream:append, with breakdowns by native project group slug, stream_path, and stream_event_type. A production warning at 50,000/day and intervention at 60,000/day leaves headroom below $20/day at the highest first-paid combined marginal rates. Non-production stream volume should be exactly zero.
4. Alert on source integrity. Daily checks should assert zero ephemeral rows, zero duplicate top-level event UUIDs, and bounded delay between the source timestamp and PostHog's created_at.
5. Use created_at for billing investigations. The event timestamp is the durable source commit time and can be old when a backlog is delivered; created_at is the PostHog ingestion time that explains the bill.
6. Use groups on every production event. A browser person is the authenticated user. A machine-authored stream fact uses one stable synthetic identity per project and carries only project group context. Browser events carry both organization and project context where available.
7. Classify the production error rows. Fix the two current runtime.runningScripts contract violations and track the old rollout/parked cases to explicit resolutions. Do not normalize them as expected noise.
8. Keep Group Analytics for all production activity. The product model needs organization- and project-level funnels, cohorts, flags, experiments, and breakdowns in the browser, plus project-level analysis across browser and durable stream events. Accept its identified-event charge, and track the add-on line item so a future volume increase cannot silently change the economics again.
9. Treat billing limits as a last-resort circuit breaker. PostHog limits can permanently drop data. Use a warning below the limit, document ownership and rollback, and keep production headroom rather than setting the limit at the normal run rate.

Reproducibility notes#

All PostHog data and operations in this investigation used posthog-cli api. The environment projects are:

• production: 115112
• development: 116602
• staging/previews: 116603

The standard PostHog event audit also passed its direct-SDK checks: there are no direct posthog.capture() call sites, automatic pageviews were 2.31% of the trailing week's production events, and the production project contained no detectable development or preview workers. Its direct-call checks do not cover the custom batch-capture path investigated here.

Current PostHog references:

• Product Analytics pricing
• Identified Events and Group Analytics add-ons
• Group Analytics billing
• Anonymous versus identified events
• Event filtering
• Drop Events transformation
• Billing limits and alerts
• Workers Analytics Engine
• Workers Analytics Engine limits
• Workers Analytics Engine sampling
• R2 Pipelines pricing
• R2 SQL pricing