Performance with SPOT Application Services
Incident Report for SPOT Business Systems
Postmortem

On Tuesday, December 3rd, 2019, we received multiple reports of performance issues for hosted SPOT users. The issue was identified as a performance problem with one of the database servers hosting multiple deployment groups (listed below). Once the issue was identified, we were able to take measures using our high availability infrastructure to bring these deployment groups online on another database server that was not affected.

We have continued to monitor the performance of these database servers in order to make sure everything has fully stabilized and is performing as expected. We have also identified the root cause of the issue and will be making some configuration changes in order to prevent this issue from occurring again. These changes will be performed during an online maintenance window where all SPOT services will remain fully operational.

Deployment groups affected by this incident:

  • MDCR01
  • MDCR02
  • MDCR09
  • MDCR10
Posted Dec 04, 2019 - 17:41 MST

Resolved
This incident has been resolved.
Posted Dec 03, 2019 - 16:04 MST
Monitoring
We found the performance problems to be affecting customers on MDCR01, MDCR02, MDCR04, MDCR09 and MDCR10. Performance should have stabilized as of now and SPOT is fully operational. We are continuing to monitor for any further problems.
Posted Dec 03, 2019 - 13:47 MST
Update
We are continuing to investigate this issue.
Posted Dec 03, 2019 - 13:01 MST
Investigating
We are currently investigating reports of issues with SPOT performance.
Posted Dec 03, 2019 - 13:00 MST
This incident affected: SPOT Hosting Services (SPOT Database Servers).