Infraly, LLC - Infraly - Chicago – Partial Network Outage – Incident details

All impacted services have been operational for 4+ hours now. We will be providing a detailed post-mortem for this incident later this week, following internal discussions with management and our network team.

This is the most widespread/impactful outage we've ever experienced as a company, so transparency and ensuring customers are fully aware of what happened, why it happened, what we did to fix, and further prevent it from happening again are a top priority.

We sincerely apologize for any inconvenience this incident has caused.

The majority of impacted services are operational at this time. We do still have a limited number of machines (less than 40) that are experiencing issues still. We are still working on this and will provide updates as soon as possible.

We are continuing to investigate route instability isolated to our Chicago location. The issue is stemming from communication inconsistencies between our edge routers at the spine layer. This is resulting in frequent route flapping and intermittent disruptions in traffic flow.

Our network engineering team is working to rebalance route propagation and mitigate further impact. We'll continue to provide additional updates as soon as possible.

Our network team is still working on this issue. We will share information when we are able to do so.