Our team received reports of degraded performance defined as, and ranging from, slowness loading Concierge routes; pages generally loading slowly; or cases where purchased licenses weren't being immediately updated.
We identified the most significant root cause as an issue with a third-party library used in our backoffice infrastructure which was causing our services to become "poisoned". This means that when the service faced a critical state, it did not terminate cleanly and instead kept running while sick -- resulting in slowness.
Our fix has been two-fold to begin with: 1. We have adjusted the library to a state where the issue does not exist. 2. We will monitor services for similar poisoned status, and if experienced, these services will be restarted so that they can recover properly.
We have already deployed fixes for this and have had confirmation that there is a marked improvement in load times. We will continue to monitor this moving forward, making adjustments as needed.