Earlier today, beginning at 12:15pm MDT, our monitoring alerted us about connectivity issues for hosted SPOT services. The duration of the issue was approximately 38 minutes. There was a related brief connectivity issue this afternoon that was resolved within 3 minutes. The issue was identified as an upstream connectivity issue between our primary firewall and our provider. We worked together with our provider to address the issue and ensure that it does not happen again. We plan on performing needed, related maintenance tonight. SPOT services will remain fully operational during this maintenance window.
Posted Jun 01, 2020 - 15:57 MDT
Resolved
This incident was resolved as noted earlier. Additional details will be provided in a postmortem update shortly.
Posted Jun 01, 2020 - 14:12 MDT
Monitoring
With the issue resolved we are continuing to monitor as we implement new measures to prevent this from occurring again.
Posted Jun 01, 2020 - 13:14 MDT
Identified
A problem has been identified with our top level networking infrastructure.
Posted Jun 01, 2020 - 13:12 MDT
Investigating
We are investigating connectivity issues to SPOT
Posted Jun 01, 2020 - 12:23 MDT
This incident affected: SPOT Hosting Services (SPOT Application Servers, Web Service, SPOT Database Servers, Automated Conveyor Service, Kiosk Service, CustomerConnect - Web Service, CustomerConnect - Mobile Service, Dashboard Service, ProductionTrac Service, DeliveryConnect - Mobile Service, Print Service, Network Infrastructure).