The maintenance was completed successfully with the cluster replaced. Resolving incident. Root cause analysis will be available early next week upon customer request.
Posted Apr 20, 2018 - 13:33 EDT
Identified
The development teams are beginning maintenance on the cluster at this time. Updates will be provided as information becomes available throughout the window.
Posted Apr 20, 2018 - 11:07 EDT
Monitoring
Maintenance window to replace cluster will take place at midnight Tokyo time. Monitoring for stability until window.
Posted Apr 20, 2018 - 06:53 EDT
Update
Customers who have rebooted their edges after development actions have noticed improvement. Any users impacted should attempt to reboot edges until the maintenance window to replace the cluster.
Posted Apr 20, 2018 - 05:53 EDT
Update
Efforts have continued to relieve degradation until the cluster is replaced. Updates to follow.
Posted Apr 20, 2018 - 04:09 EDT
Update
An emergency maintenance window has been set for midnight Tokyo time 4/21. The maintenance window will be used to replace the cluster of the impacted service. Users will be in a degraded state until the cluster is replaced.
Posted Apr 20, 2018 - 03:07 EDT
Update
The development teams have continued efforts to repair the cluster while enabling services to be up although in a degraded state. There is no ETR at this time and the cluster will need to be replaced in an emergency maintenance window after hours. The teams are continuing various remediation steps to avoid full outage at this time. Updates will be provided minimum hourly or sooner as information becomes available.
Posted Apr 20, 2018 - 02:41 EDT
Update
Development teams are continuing actions to improve degraded conditions. No ETR at this time.
Posted Apr 20, 2018 - 01:48 EDT
Update
Error counts are beginning to improve and throughput rising. Users will be operational although in a degraded state until fix actions are completed.
Posted Apr 20, 2018 - 01:03 EDT
Update
The development teams are removing a node to route traffic to verified healthy nodes. Users will experience a brief period of interruption.
Posted Apr 20, 2018 - 00:45 EDT
Update
Although services are currently operational, some remain in a degraded state. The development teams are continuing to take remediation steps. Updates to follow.
Posted Apr 20, 2018 - 00:26 EDT
Update
The instance is back online, error rates are beginning to improve and throughput is increasing. Updates to follow.
Posted Apr 19, 2018 - 23:54 EDT
Update
Call controls, phone selection, and IVR's are impacted by this event.
Posted Apr 19, 2018 - 23:27 EDT
Update
Additional development teams have been engaged to provide assistance. Issues were identified on the new instance. Updates to follow.
Posted Apr 19, 2018 - 23:23 EDT
Update
Development teams have replaced an unhealthy instance however some services are still experiencing errors. Updates to follow.
Posted Apr 19, 2018 - 23:09 EDT
Identified
Some customers in the Japan region have reported issues with call wrap up failing. The development teams are currently engaged for review.
Posted Apr 19, 2018 - 23:00 EDT
This incident affected: Inbound Calls (Asia Pacific (Tokyo)) and Outbound Calls (Asia Pacific (Tokyo)).