Resolved -
This incident has been resolved.
Mar 20, 18:05 PDT
Monitoring -
Dataset freshness has been restored to configured goals on customer tenants. We will continue to monitor freshness for the next 2 hours to ensure this incident has been resolved.
Mar 20, 16:05 PDT
Update -
Recovery efforts are ongoing on the Observe side and we are getting closer to meeting data freshness goals on all customer datasets. We expect all remaining datasets freshness latency to be resolved within the next hour. We are continuing to monitor the systems very closely through the recovery process.
Mar 20, 14:52 PDT
Update -
Our service provider, Snowflake, has completed changes to mitigate the networking issue on their end. https://status.snowflake.com/incidents/525rr3btnrz8.
Recovery efforts are ongoing on the Observe side and we are getting closer to meeting our customers' data freshness goals. ETA on dataset freshness catch up is approximately 1 hour. We are continue to monitor the systems very closely through recovery.
Mar 20, 13:27 PDT
Update -
Snowflake has completed changes to mitigate the networking issue on their end. This is the incident they were working https://status.snowflake.com/incidents/525rr3btnrz8.
Recovery efforts are underway on the Observe side and we are steadily catching back up. ETA on restoration is approximately 1-2 hours to be fully restored and caught up.
We currently are monitoring the catch up and the current ingest of this region.
Mar 20, 12:28 PDT
Update -
Snowflake is continuing to apply changes to mitigate the networking issue on their end. Recovery efforts are underway on the Observe side and we are ramping processing back up. https://status.snowflake.com/incidents/525rr3btnrz8. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.
Mar 20, 11:27 PDT
Identified -
We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified a networking issue that is tracked by an incident https://status.snowflake.com/incidents/525rr3btnrz8. Snowflake is actively taking steps to mitigate the identified issue. There is no ETA yet on resolution. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.
Mar 20, 10:28 PDT
Update -
We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified an issue that is affecting more than 3 customers that is tracked by an incident https://status.snowflake.com/incidents/525rr3btnrz8. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.
Mar 20, 09:28 PDT
Update -
We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified an issue that is affecting more than 3 customers. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.
Mar 20, 08:56 PDT
Update -
We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.
Mar 20, 07:56 PDT
Update -
We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.
Mar 20, 06:55 PDT
Update -
We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.
Mar 20, 05:55 PDT
Update -
We are currently still investigating and actively working the incident with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.
Mar 20, 04:55 PDT
Update -
We have identified that the issue is being caused by an issue with our service provider Snowflake. Currently all ingest is failing but we do not expect any data loss. Our engineers are actively working with Snowflake to try to identify the root cause of the issue and mitigate. We will post further updates as they become available.
Mar 20, 03:58 PDT
Investigating -
We’ve identified an issue causing delays in ingest in the following regions: US East (N. Virginia). As a result, some users may experience see delays in ingesting data. Our team is actively investigating the root cause and working to restore normal performance as quickly as possible. We’ll provide updates as we learn more. Thank you for your patience!
Mar 20, 02:55 PDT