This incident has been resolved. Root cause was due to an errant application of additional security control policies that led to a temporary failure of API calls across the platform. The change was applied at 14:26 UTC and reverted at 14:29 UTC. By 14:34 UTC, the reverted change was fully propagated. Services scaled to handle the increased throughput during recovery. All health checks were confirmed passing by 15:40 UTC.
Posted Aug 05, 2024 - 11:55 EDT
Monitoring
All health checks are now passing and throughput has returned to normal. Errors with station association in Americas (US East) have cleared and throughput is decreasing. Dialer tests are now passing in Americas (US East). Monitoring all services to ensure continued stability. Next update 15 minutes.
Posted Aug 05, 2024 - 11:33 EDT
Update
Regions are passing health checks. Americas (US East) is still seeing errors with phone service (affecting new station association) and errors with dialer campaigns and call back services. Next update 15 minutes.
Posted Aug 05, 2024 - 11:21 EDT
Update
Services have largely recovered. Teams are monitoring health checks and determining if any additional actions need to be taken to speed resolution. Next update 15 minutes.
Posted Aug 05, 2024 - 11:05 EDT
Identified
Increased maintenance mode errors have been identified globally. Root cause has been identified and mitigation steps applied. Affected services are in recovery. Next update top of the hour.
Posted Aug 05, 2024 - 10:48 EDT
This incident affected: Platform (Americas (Sao Paulo), Americas (US East), Americas (US West), Americas (Canada), Asia Pacific (Mumbai), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Seoul), EMEA (Dublin), EMEA (Frankfurt), EMEA (London), FedRAMP, EMEA (Zurich), EMEA (UAE), Asia Pacific (Osaka)).