We are marking this issue as resolved since incoming data shows services are recovered, error rates have dropped, and loading times are stabilized.
Our team will continue to monitor services closely to ensure they remain stable moving forward, and actions have already been taken to ensure this stability is maintained.
Posted Oct 19, 2023 - 21:01 UTC
Monitoring
Services have been restored, and several sub-services were restarted in order to further support the recovery of all instances.
We are seeing load times gradually return to normal with only a minor performance impact reported.
We will continue to monitor the recovery closely to ensure our services remain stable.
Posted Oct 19, 2023 - 20:14 UTC
Identified
We have begun to restore services and have reduced the load on the impacted databases. Databases instances are being restarted to help mitigate the issue.
This is showing improvement on load times, however users may still experience long delays or see an "Error: Server Error" 502 message with degraded performance of all Chili Piper pages, links, Instant Booker, and Concierge.
We should start seeing more services gradually become healthy in time, and the loading speed will continue to increase, however due to the increased load times we are still considering this a partial outage.
Posted Oct 19, 2023 - 19:59 UTC
Investigating
We are currently investigating an issue impacting our database which is preventing Chili Piper assets from loading.
Instant Booker as well as Chili Piper's dashboard are both impacted. Some booking links appear to load, but performance is extremely degraded. Pages may load after a long delay.
Our Site Reliability infrastructure team is actively investigating this with priority, and we will provide an update once this issue is resolved.
Posted Oct 19, 2023 - 19:29 UTC
This incident affected: Booking - expand for components (Dashboard, Instant Booker, Booking Links, Chrome Extension) and Concierge Routing.