All backlogged events have been processed and service is operating as expected.
Posted May 06, 2024 - 17:43 EDT
Update
The service should have all backlogged events processed within the next hour. Next update 18:30 or sooner as information becomes available.
Posted May 06, 2024 - 17:25 EDT
Update
The service team has updated a dynamic property to help speed processing of http events in the backlog. This change caused lag to briefly increase but should improve overall event processing. Latency continues to decrease at this time. Next update 17:30 Eastern
Posted May 06, 2024 - 16:36 EDT
Monitoring
Latency continues on a downward trend. If backlog continues to process at current rate, expect all clear within the next two hours. Next update 16:30 Eastern or sooner as information is available.
Posted May 06, 2024 - 15:21 EDT
Identified
Teams have implemented additional mitigation steps and we have begun to see improvement. Customers may continue to see delays until backlogged events have cleared. Changes to prevent recurrence have been identified and will need to undergo QA. If successful, the improvements will be deployed after hours tonight. Next update in one hour.
Posted May 06, 2024 - 14:25 EDT
Update
Service teams are continuing to investigate root cause of the elevated errors on analytics consumer. There will be continued delayed conversation events until root cause is identified and backlogged events are processed. Next update 30 minutes or as new information becomes available.
Posted May 06, 2024 - 13:49 EDT
Update
Teams are continuing to investigate. Previously implemented mitigation steps slowed growth of the analytics consumer lag but did not prevent further event accumulation. Additional development resources have been added to assist in isolation of root cause. Next update in 30 minutes or sooner as information becomes available.
Posted May 06, 2024 - 13:17 EDT
Investigating
Teams are investigating an issue impacting Realtime Analytics. Customers may see delays in conversation detail records. Teams have taken mitigation steps to slow the analytics consumer delay but users may see delayed events until backlog is processed. Next update in thirty minutes or sooner as information becomes available.
Posted May 06, 2024 - 12:43 EDT
This incident affected: Reporting (Americas (US West)).