Infraly, LLC - Infraly - Chicago – Partial Network Outage – Incident details

Infraly - Chicago – Partial Network Outage

Resolved
Partial outage
Started 2 months agoLasted 6 days

Affected

Chicago, Illinois Data Center

Partial outage from 3:21 AM to 3:41 AM, Degraded performance from 3:41 AM to 4:59 AM, Operational from 4:59 AM to 5:52 AM, Degraded performance from 5:52 AM to 10:25 AM, Operational from 10:25 AM to 12:00 AM

Core Network Infrastructure

Partial outage from 3:21 AM to 3:41 AM, Degraded performance from 3:41 AM to 4:59 AM, Operational from 4:59 AM to 5:52 AM, Degraded performance from 5:52 AM to 10:25 AM, Operational from 10:25 AM to 12:00 AM

GTT Communications (Internet Carrier)

Partial outage from 3:21 AM to 3:41 AM, Degraded performance from 3:41 AM to 4:59 AM, Operational from 4:59 AM to 5:52 AM, Degraded performance from 5:52 AM to 10:25 AM, Operational from 10:25 AM to 12:00 AM

Cogent Communications (Internet Carrier)

Partial outage from 3:21 AM to 3:41 AM, Degraded performance from 3:41 AM to 4:59 AM, Operational from 4:59 AM to 5:52 AM, Degraded performance from 5:52 AM to 10:25 AM, Operational from 10:25 AM to 12:00 AM

Chicago Internet Exchange (Internet Carrier)

Partial outage from 3:21 AM to 3:41 AM, Degraded performance from 3:41 AM to 4:59 AM, Operational from 4:59 AM to 5:52 AM, Degraded performance from 5:52 AM to 10:25 AM, Operational from 10:25 AM to 12:00 AM

Updates
  • Resolved
    Resolved

    We will be posting a detailed post-mortem regarding this incident later this week. We have internal meetings with our network engineers as well as Zayo this week to discuss this incident and the exact failures on Zayo's end.

  • Monitoring
    Monitoring

    Following this unprecedented event, all carrier connections have come online without issues and are traversing traffic with healthy throughput checks. We will continue to monitor over the next few hours. Please stay tuned for the post-mortem, as that will be coming with additional information on this incident. Thank you again for your patience and understanding.

  • Update
    Update

    We have been monitoring our connections and see that all failed connections are online and operational now. We will begin turning up these connections now.

  • Update
    Update

    Zayo has given us an estimated restoration time of 6:00 am CT. Upon restoration of our fiber ring (connection to 350 E Cermak and another connection to 427 La Salle), our additional carrier connections will restore themselves and alleviate congestion on existing on-net carrier connections from Comcast and Lumen. We will provide updates ASAP.

  • Update
    Update

    For transparency, only Comcast and Lumen on-net connections are traversing traffic at this moment, with heavy congestion in the Chicago area.

  • Update
    Update

    We have complete confirmation that our Zayo fiber ring is completely broken now. Runs from our West Chicago facility to 350 E Cermak and West Chicago to 427 La Salle St are offline. Only connections available are our on-net connections, but these are heavily congested due to the limited capacity that carriers have themselves, due to the fiber ring issues with Zayo.

  • Update
    Update

    We're seeing heavy congestion currently on available uplinks. This suggests more fiber paths have been impacted on Zayo's end. We can confirm we have a complete outage on one of our fiber ring connections to downtown. We will provide an update ASAP.

  • Identified
    Identified

    We're seeing more instability on more dark fiber paths, which is causing a deeper impact on services. We are currently evaluating the situation as several Zayo fiber paths have failed now.

  • Monitoring
    Monitoring

    Route changes were made for the lingering packet loss issues impacting some connections. This should clear up most connections. We are still awaiting updates from Zayo.

  • Update
    Update

    As our investigation suggested, Zayo confirmed they are encountering fiber issues on paths that carry some of our Tier 1 carrier connections. We will provide updates as we receive them from Zayo. Our network team is working on any lingering issues at the current moment.

  • Identified
    Identified

    Our network team has been working on this to restore any prefixes with special routing requests. We believe this to be a dark fiber cut on Zayo's network that we haul in some connections on that are impacting upstream ports.

  • Investigating
    Investigating
    We are currently investigating this incident.