Resolved -
One cell of our CH infrastructure encountered replication issues and required re-replication of the data in a shard. That is now complete and all systems are operating as expected. Enjoy your weekend and holidays!
Dec 20, 03:43 UTC
Update -
Still monitoring for bit longer before we raise the API query limits again, but recovery is looking good.
Dec 20, 00:34 UTC
Monitoring -
We've applied some fixes, and are almost fully recovered. We're monitoring closely for a bit longer to be sure.
Dec 19, 22:21 UTC
Update -
We're still working on this. You may see errors about busy query endpoints in your insights and reports in the app. This only impacts loading existing data from PostHog, ingestion and feature flags are unaffected.
Dec 19, 20:46 UTC
Update -
To mitigate the impact from this incident, we have decided to limit API queries temporarily. You may experience increased rate limits on all public endpoints that fetch data from PostHog until we resolve this.
Dec 19, 18:36 UTC
Identified -
We have spotted a load spike in our ClickHouse cluster, we already identified the problem and are working to resolve it as soon as possible.
Query performance may be degraded and the app may not respond as expected.
Dec 19, 13:31 UTC