Resolved -
The maintenance was completed successfully with the cluster replaced. Resolving incident. Root cause analysis will be available early next week upon customer request.
Apr 20, 13:33 EDT
Identified -
The development teams are beginning maintenance on the cluster at this time. Updates will be provided as information becomes available throughout the window.
Apr 20, 11:07 EDT
Monitoring -
Maintenance window to replace cluster will take place at midnight Tokyo time. Monitoring for stability until window.
Apr 20, 06:53 EDT
Update -
Customers who have rebooted their edges after development actions have noticed improvement. Any users impacted should attempt to reboot edges until the maintenance window to replace the cluster.
Apr 20, 05:53 EDT
Update -
Efforts have continued to relieve degradation until the cluster is replaced. Updates to follow.
Apr 20, 04:09 EDT
Update -
An emergency maintenance window has been set for midnight Tokyo time 4/21. The maintenance window will be used to replace the cluster of the impacted service. Users will be in a degraded state until the cluster is replaced.
Apr 20, 03:07 EDT
Update -
The development teams have continued efforts to repair the cluster while enabling services to be up although in a degraded state. There is no ETR at this time and the cluster will need to be replaced in an emergency maintenance window after hours. The teams are continuing various remediation steps to avoid full outage at this time. Updates will be provided minimum hourly or sooner as information becomes available.
Apr 20, 02:41 EDT
Update -
Development teams are continuing actions to improve degraded conditions. No ETR at this time.
Apr 20, 01:48 EDT
Update -
Error counts are beginning to improve and throughput rising. Users will be operational although in a degraded state until fix actions are completed.
Apr 20, 01:03 EDT
Update -
The development teams are removing a node to route traffic to verified healthy nodes. Users will experience a brief period of interruption.
Apr 20, 00:45 EDT
Update -
Although services are currently operational, some remain in a degraded state. The development teams are continuing to take remediation steps. Updates to follow.
Apr 20, 00:26 EDT
Update -
The instance is back online, error rates are beginning to improve and throughput is increasing. Updates to follow.
Apr 19, 23:54 EDT
Update -
Call controls, phone selection, and IVR's are impacted by this event.
Apr 19, 23:27 EDT
Update -
Additional development teams have been engaged to provide assistance. Issues were identified on the new instance. Updates to follow.
Apr 19, 23:23 EDT
Update -
Development teams have replaced an unhealthy instance however some services are still experiencing errors. Updates to follow.
Apr 19, 23:09 EDT
Identified -
Some customers in the Japan region have reported issues with call wrap up failing. The development teams are currently engaged for review.
Apr 19, 23:00 EDT