Infraly, LLC - Infraly - Chicago – Partial Network Outage – Incident details

We will be posting a detailed post-mortem regarding this incident later this week. We have internal meetings with our network engineers as well as Zayo this week to discuss this incident and the exact failures on Zayo's end.

Following this unprecedented event, all carrier connections have come online without issues and are traversing traffic with healthy throughput checks. We will continue to monitor over the next few hours. Please stay tuned for the post-mortem, as that will be coming with additional information on this incident. Thank you again for your patience and understanding.

We have been monitoring our connections and see that all failed connections are online and operational now. We will begin turning up these connections now.

Zayo has given us an estimated restoration time of 6:00 am CT. Upon restoration of our fiber ring (connection to 350 E Cermak and another connection to 427 La Salle), our additional carrier connections will restore themselves and alleviate congestion on existing on-net carrier connections from Comcast and Lumen. We will provide updates ASAP.

For transparency, only Comcast and Lumen on-net connections are traversing traffic at this moment, with heavy congestion in the Chicago area.

We have complete confirmation that our Zayo fiber ring is completely broken now. Runs from our West Chicago facility to 350 E Cermak and West Chicago to 427 La Salle St are offline. Only connections available are our on-net connections, but these are heavily congested due to the limited capacity that carriers have themselves, due to the fiber ring issues with Zayo.

We're seeing heavy congestion currently on available uplinks. This suggests more fiber paths have been impacted on Zayo's end. We can confirm we have a complete outage on one of our fiber ring connections to downtown. We will provide an update ASAP.

We're seeing more instability on more dark fiber paths, which is causing a deeper impact on services. We are currently evaluating the situation as several Zayo fiber paths have failed now.

Route changes were made for the lingering packet loss issues impacting some connections. This should clear up most connections. We are still awaiting updates from Zayo.

As our investigation suggested, Zayo confirmed they are encountering fiber issues on paths that carry some of our Tier 1 carrier connections. We will provide updates as we receive them from Zayo. Our network team is working on any lingering issues at the current moment.

Our network team has been working on this to restore any prefixes with special routing requests. We believe this to be a dark fiber cut on Zayo's network that we haul in some connections on that are impacting upstream ports.