Additional mitigation steps have been implemented. Agents should go off queue and back on again to begin receiving interactions. Backlog of events is clear and service is processing new requests.
Posted Oct 11, 2023 - 15:24 EDT
Update
We are seeing significant improvement on processing of backlogged events. Next update 15:15 Eastern.
Posted Oct 11, 2023 - 14:50 EDT
Update
Pending relief of backlog pressure, teams are reviewing additional mitigation steps. Next update 14:50 Eastern
Posted Oct 11, 2023 - 14:39 EDT
Update
Mitigation steps are in progress to help alleviate pressure on the cluster. The teams have been able to control traffic so backlog is slowly being processed. Next update 14:30 Eastern.
Posted Oct 11, 2023 - 14:08 EDT
Update
Processing of backlogged events has stalled. Teams have scaled services out and have implemented additional mitigation steps to relieve pressure on the cluster but have not seen improvement. Additional resources have been engaged and are on bridge actively troubleshooting. Next update 14:05 Eastern.
Posted Oct 11, 2023 - 13:51 EDT
Identified
Starting at approximately 11:35AM Eastern, teams began seeing elevated error rates in US West region. Customers in region would have noticed interactions not being routed to agents. The teams quickly identified root cause and applied mitigation steps, however a backlog of events had accumulated due to the number of retry requests. The backlog is being processed and teams are continuing to monitor the cluster health. Next update 13:45 or sooner as information becomes available.
Posted Oct 11, 2023 - 13:15 EDT
This incident affected: Platform (Americas (US West)).