Infraly, LLC - Notice history

Core Network Infrastructure experiencing degraded performance

Notice history

Jul 2025

WISP - Front-end Outage
  • Resolved
    Resolved
    At 23:08 UTC, the WISP front-end services came back online and operational again. The root cause was traced to a failure on our main Chicago node, coupled with a routing issue affecting our London node. This resulted in 2 out of 3 sites falling out of sync and becoming inaccessible due to a loss of quorum. Our team quickly enacted recovery procedures in London, restoring it to production. We will continue to monitor the platform closely over the next 24 hours as we work to bring additional nodes back into global quorum. Thank you for your patience and understanding.
  • Investigating
    Investigating
    At 22:57 UTC, our monitoring reported WISP front-end services became inaccessible. Our team is currently working on recovering and bringing online.
Infraly - Global - Upstream Maintenance - July 23rd, 2025
  • Completed
    July 23, 2025 at 3:00 PM
    Completed
    July 23, 2025 at 3:00 PM
    Maintenance has completed successfully
  • In progress
    July 23, 2025 at 1:00 PM
    In progress
    July 23, 2025 at 1:00 PM
    Maintenance is now in progress
  • Planned
    July 23, 2025 at 1:00 PM
    Planned
    July 23, 2025 at 1:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Global - Upstream Maintenance - July 23rd, 2025
    Locations Affected: Chicago, Illinois & London, UK Path-Routed Prefixes
    Start Date: July 23rd, 2025 @ 13:00 UTC
    Estimated End Date: July 23rd, 2025 @ 15:00 UTC

    Summary: Path, one of our DDoS protection providers, is performing updates to critical BGP routing sessions in select PoPs (points-of-presence). We apologize for the short notice and inconvenience. We were notified ~4 hours before the maintenance is scheduled to take place.

    Impact on Customer: Disruptions are expected to be minimal and short-lived. As a result, we will be moving Chicago Path-routed traffic before the maintenance window to our on-site filtering appliances. We believe this will minimize the impact of disruptions. For London, our on-site filtering appliances are not fully active, so we are unable to move Path-routed prefixes without causing greater traffic disruptions. Our team will be monitoring throughout the maintenance. We apologize for the inconvenience.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - General Maintenance – July 15th, 2025
  • Completed
    July 15, 2025 at 8:00 PM
    Completed
    July 15, 2025 at 8:00 PM
    Maintenance has completed successfully
  • In progress
    July 15, 2025 at 7:00 PM
    In progress
    July 15, 2025 at 7:00 PM
    Maintenance is now in progress
  • Planned
    July 15, 2025 at 7:00 PM
    Planned
    July 15, 2025 at 7:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    WISP - General Maintenance – July 15th, 2025

    Services Affected: WISP Front-end Services
    Start Date: July 15th, 2025 @ 19:00 UTC
    Estimated End Date: July 15th, 2025 @ 20:00 UTC

    Summary: Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability.

    Impact on Customer: As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - General Maintenance – July 14th, 2025
  • Completed
    July 14, 2025 at 11:00 AM
    Completed
    July 14, 2025 at 11:00 AM
    Maintenance has completed successfully
  • In progress
    July 14, 2025 at 10:00 AM
    In progress
    July 14, 2025 at 10:00 AM
    Maintenance is now in progress
  • Planned
    July 14, 2025 at 10:00 AM
    Planned
    July 14, 2025 at 10:00 AM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    WISP - General Maintenance – July 14th, 2025

    Services Affected: WISP Front-end Services
    Start Date: July 14th, 2025 @ 10:00 UTC
    Estimated End Date: July 14th, 2025 @ 11:00 UTC

    Summary: Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability.

    Impact on Customer: As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Chicago - Network Maintenance - July 9th, 2025
  • Completed
    July 09, 2025 at 7:00 PM
    Completed
    July 09, 2025 at 7:00 PM
    Maintenance has completed successfully
  • In progress
    July 09, 2025 at 3:00 PM
    In progress
    July 09, 2025 at 3:00 PM
    Maintenance is now in progress
  • Planned
    July 09, 2025 at 3:00 PM
    Planned
    July 09, 2025 at 3:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Chicago - Network Maintenance – July 9th, 2025

    Services Affected: Chicago, Illinois Data Center
    Start Date: July 9th, 2025 @ 15:00 UTC
    Estimated End Date: July 9th, 2025 @ 19:00 UTC

    Summary: We will be performing maintenance on our core network infrastructure to further enhance performance, reliability, and capability.

    Impact on Customer: During this window, our network team will make changes to better optimize traffic routing. While we take every possible step to minimize and isolate any impact, there is always an inherent risk of disruption as changes are applied. We appreciate your patience and understanding.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly

Jun 2025

Infraly - Chicago - Carrier Maintenance - June 25th, 2025
  • Completed
    June 25, 2025 at 10:00 AM
    Completed
    June 25, 2025 at 10:00 AM
    Maintenance has completed successfully
  • In progress
    June 25, 2025 at 7:00 AM
    In progress
    June 25, 2025 at 7:00 AM
    Maintenance is now in progress
  • Planned
    June 25, 2025 at 7:00 AM
    Planned
    June 25, 2025 at 7:00 AM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Chicago - Carrier Maintenance - June 25th, 2025
    Location Affected: Chicago, Illinois Path-Routed Prefixes
    Start Date: June 25th, 2025 @ 7:00 UTC
    Estimated End Date: June 25th, 2025 @ 10:00 UTC

    Summary: GTT Communications, one of Path's DDoS-protection upstream providers in Chicago, is performing maintenance in their Chicago place of presence (PoP).

    Impact on Customer: We will move all prefixes routed to Path remote protection to our Infraly filtering appliances before this maintenance starts to limit disruptions. Connection states are synced between the different locations, but there is always a possibility for disruptions to TCP and UDP traffic when the route change occurs.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Global - Filter Update - June 18th, 2025
  • Completed
    June 18, 2025 at 6:16 PM
    Completed
    June 18, 2025 at 6:16 PM
    Maintenance has completed successfully.
  • In progress
    June 18, 2025 at 6:00 PM
    In progress
    June 18, 2025 at 6:00 PM
    Maintenance is now in progress
  • Planned
    June 18, 2025 at 6:00 PM
    Planned
    June 18, 2025 at 6:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Global - Filter Update - June 18th, 2025
    Locations Affected: Chicago, Illinois & London, UK Path-Routed Prefixes
    Start Date: June 18th, 2025 @ 18:00 UTC
    Estimated End Date: June 18th, 2025 @ 19:00 UTC

    Summary: Path, one of our DDoS protection providers, is performing updates to its filtering stack to further improve performance, reliability, and capability.

    Impact on Customer: Services using the TCP service (symmetric) filter may see a brief interruption as the filter update is rolled out. They expect the overall window of impact to be less than 15 minutes.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

[Rescheduled] Infraly - London - Carrier Maintenance - June 17th, 2025
  • Completed
    June 18, 2025 at 2:00 AM
    Completed
    June 18, 2025 at 2:00 AM
    Maintenance has completed successfully
  • In progress
    June 17, 2025 at 9:00 PM
    In progress
    June 17, 2025 at 9:00 PM
    Maintenance is now in progress
  • Planned
    June 17, 2025 at 9:00 PM
    Planned
    June 17, 2025 at 9:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - London - Carrier Maintenance - June 17th, 2025
    Location Affected: London, UK Path-Routed Prefixes
    Start Date: June 17th, 2025 @ 21:00 UTC
    Estimated End Date: June 18th, 2025 @ 2:00 UTC

    Summary: One of our DDoS-protection upstreams is performing maintenance in their London place of presence (PoP) to further improve performance, reliability, and capability.

    Impact on Customer: Customers will experience a route change at the beginning and end of this maintenance as traffic is routed between London and Amsterdam. We will actively monitor throughout the maintenance period and make adjustments to upstream-routed prefixes if required.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - London - Carrier Maintenance - June 16th - 18th, 2025
  • Completed
    June 18, 2025 at 7:00 AM
    Completed
    June 18, 2025 at 7:00 AM
    Maintenance has completed successfully.
  • In progress
    June 16, 2025 at 4:00 AM
    In progress
    June 16, 2025 at 4:00 AM
    Maintenance is now in progress
  • Planned
    June 16, 2025 at 4:00 AM
    Planned
    June 16, 2025 at 4:00 AM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - London - Carrier Maintenance - June 16th - 18th, 2025
    Location Affected: London, UK Data Center
    Start Date: June 16th, 2025 @ 4:00 UTC
    Estimated End Date: June 18th, 2025 @ 7:00 UTC

    Summary: One of our on-net carriers is performing firmware updates on their equipment to further improve performance, reliability, and capability over a period of 3 days.

    Impact on Customer: Work will be carried out in stages over the 3-day window, with activity scheduled each day between 4:00 and 7:00 UTC. While the majority of this work is expected to be non-disruptive, there may be brief interruptions lasting approximately 15–20 minutes during device reboots required for the firmware updates.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - London - LON-P-VPS-1 Upgrade – June 13th, 2025
  • Completed
    June 15, 2025 at 7:03 PM
    Completed
    June 15, 2025 at 7:03 PM

    The affected machine is now back online following the arrival and installation of a replacement motherboard earlier today. Our team completed the swap and validation successfully, and all VMs on LON-P-VPS-1 should now be fully operational.

    To clarify, we initially attempted to restore service using our standby hardware. However, that system was also found to have a failed motherboard and would not post, which required us to order a replacement. While delivery was originally expected on Saturday based on tracking information, the part did not arrive until today, Sunday, which unfortunately extended the outage.

    We sincerely apologize for the disruption and inconvenience this has caused. We’re providing a full month of service credit to all affected customers, going beyond what’s covered under our SLA. No ticket is necessary; this credit will be automatically applied to the impacted services.

    Thank you again for your patience and understanding throughout this incident.

  • In progress
    June 14, 2025 at 12:00 AM
    In progress
    June 14, 2025 at 12:00 AM

    This service remains offline due to a critical component failure that occurred following a recent upgrade. Unfortunately, the standby hardware we had on-site was also found to be non-functional, which further delayed recovery efforts. An emergency replacement has been ordered and is being delivered today, Saturday. Once received, our team will immediately proceed with installation and testing to bring the machine back online and fully operational. We sincerely apologize for the ongoing disruption. Per our Service Level Agreement (SLA), account credits will be issued to all affected customers. Further updates will be provided as progress is made.

  • Planned
    June 13, 2025 at 5:00 PM
    Planned
    June 13, 2025 at 5:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - London - LON-P-VPS-1 Upgrade – June 13th, 2025

    Services Affected: LON-P-VPS-1
    Start Date: June 13th, 2025 @ 17:00 UTC
    Estimated End Date: June 14th, 2025 @ 00:00 UTC

    Summary: We will be performing maintenance on LON-P-VPS-1 to further enhance performance, reliability, and capability.

    Impact on Customer: During this window, one of our datacenter technicians will be performing a hardware upgrade that will require powering down LON-P-VPS-1. Once the upgrade is complete, the machine will be brought back online. We have scheduled this window to account for other pending tasks that may affect the exact timing of this work. We appreciate your patience and understanding.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

  • In progress
    June 13, 2025 at 5:00 PM
    In progress
    June 13, 2025 at 5:00 PM
    Maintenance is now in progress.

May 2025

Upstream-Routed Services - London Capacity Upgrades
  • Completed
    May 31, 2025 at 1:52 AM
    Completed
    May 31, 2025 at 1:52 AM
    Maintenance has completed successfully
  • In progress
    May 31, 2025 at 12:52 AM
    In progress
    May 31, 2025 at 12:52 AM
    Maintenance is now in progress
  • Planned
    May 31, 2025 at 12:52 AM
    Planned
    May 31, 2025 at 12:52 AM

    Upstream has informed us they are performing earlier than expected upgrades on capacity in London. Traffic may experience interruption during this period.

Infraly - Chicago - Emergency Network Maintenance - May 20th, 2025
  • Completed
    May 20, 2025 at 6:08 PM
    Completed
    May 20, 2025 at 6:08 PM

    Cleanup work has been completed, and we are exiting this emergency maintenance period. We will continue to monitor over the next few hours, and if anything comes up, we will make adjustments.

    Thank you again for your understanding, and we apologize sincerely for the short notice. Our goal is always to provide stable and reliable services and this maintenance was necessary to ensure that happens.

  • Update
    May 20, 2025 at 5:53 PM
    Update
    May 20, 2025 at 5:53 PM

    We have achieved what we needed and wanted to get done, and have been in the final cleanup stage before marking this as complete. Will update when all clear shortly.

  • Update
    May 20, 2025 at 5:15 PM
    Update
    May 20, 2025 at 5:15 PM

    Our network team is almost done with this maintenance. We will continue to provide updates.

  • Update
    May 20, 2025 at 4:55 PM
    Update
    May 20, 2025 at 4:55 PM

    Our network team is continuing to make progress. We are prepping changes that may have an impact in the coming minutes on certain ISP ports on our spines.

  • Update
    May 20, 2025 at 3:55 PM
    Update
    May 20, 2025 at 3:55 PM

    VRRP changeover occurred. We are just pending lingering traffic traversing over edge1 currently to move to edge0. We will continue to provide updates.

  • Update
    May 20, 2025 at 3:31 PM
    Update
    May 20, 2025 at 3:31 PM

    For insight, all Tier 1 and Neoprotect-routed prefixes moved traffic seamlessly as expected with metric changes. Path-routed prefixes saw a disruption in moving traffic as the metric changes applied were not being accepted, and thus resulted in a brief disruption.

    We are now altering our Virtual Router Redundancy Protocol (VRRP) priorities for IP subnets on our edge routers to alleviate impacts that we do on non-primary devices throughout the maintenance.

    We will continue to provide updates.

  • In progress
    May 20, 2025 at 2:00 PM
    In progress
    May 20, 2025 at 2:00 PM
    Maintenance is now in progress
  • Planned
    May 20, 2025 at 2:00 PM
    Planned
    May 20, 2025 at 2:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Chicago - Emergency Network Maintenance – May 20th, 2025

    Services Affected: Chicago, Illinois Data Center
    Start Date: May 20th, 2025 @ 14:00 UTC
    Estimated End Date: May 20th, 2025 @ 17:30 UTC

    Summary: Following the network outage that occurred during the late evening of May 18th, we are performing emergency maintenance to implement preventative measures and improve infrastructure reliability. We understand the short notice and sincerely apologize, but this work is necessary to ensure a more stable environment for all users.

    Impact on Customer: During this window, our network team will make core-level changes. While we aim to minimize disruption, brief connectivity blips may occur. We appreciate your patience and understanding.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - General Maintenance – May 20th, 2025
  • Completed
    May 20, 2025 at 7:00 PM
    Completed
    May 20, 2025 at 7:00 PM
    Maintenance has completed successfully.
  • Update
    May 20, 2025 at 2:00 PM
    Update
    May 20, 2025 at 2:00 PM

    We’re still turning up some instances, but will message when all clear and they are online and operational. Please note the new location capacity will be turned up throughout the day.

  • In progress
    May 20, 2025 at 12:00 PM
    In progress
    May 20, 2025 at 12:00 PM
    Maintenance is now in progress
  • Planned
    May 20, 2025 at 12:00 PM
    Planned
    May 20, 2025 at 12:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    WISP - General Maintenance – May 20th, 2025

    Services Affected: WISP Front-end Services
    Start Date: May 20th, 2025 @ 12:00 UTC
    Estimated End Date: May 20th, 2025 @ 14:00 UTC

    Summary: Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability by activating our new London and Singapore locations.

    Impact on Customer: As new locations go live, front-end services will begin using geo-routing to connect users to the closest available location. During this transition, you may notice temporary connection inconsistencies. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Chicago – Partial Network Outage
  • Resolved
    Resolved

    All impacted services have been operational for 4+ hours now. We will be providing a detailed post-mortem for this incident later this week, following internal discussions with management and our network team.

    This is the most widespread/impactful outage we've ever experienced as a company, so transparency and ensuring customers are fully aware of what happened, why it happened, what we did to fix, and further prevent it from happening again are a top priority.

    We sincerely apologize for any inconvenience this incident has caused.

  • Monitoring
    Monitoring

    The majority of impacted services are operational at this time. We do still have a limited number of machines (less than 40) that are experiencing issues still. We are still working on this and will provide updates as soon as possible.

  • Update
    Update

    We are continuing to investigate route instability isolated to our Chicago location. The issue is stemming from communication inconsistencies between our edge routers at the spine layer. This is resulting in frequent route flapping and intermittent disruptions in traffic flow.

    Our network engineering team is working to rebalance route propagation and mitigate further impact. We'll continue to provide additional updates as soon as possible.

  • Update
    Update

    Our network team is still working on this issue. We will share information when we are able to do so.

  • Investigating
    Investigating

    We are currently investigating this incident. This is only affecting certain traffic landing on edge0.

Infraly - Chicago – Partial Network Outage
  • Resolved
    Resolved

    We will be posting a detailed post-mortem regarding this incident later this week. We have internal meetings with our network engineers as well as Zayo this week to discuss this incident and the exact failures on Zayo's end.

  • Monitoring
    Monitoring

    Following this unprecedented event, all carrier connections have come online without issues and are traversing traffic with healthy throughput checks. We will continue to monitor over the next few hours. Please stay tuned for the post-mortem, as that will be coming with additional information on this incident. Thank you again for your patience and understanding.

  • Update
    Update

    We have been monitoring our connections and see that all failed connections are online and operational now. We will begin turning up these connections now.

  • Update
    Update

    Zayo has given us an estimated restoration time of 6:00 am CT. Upon restoration of our fiber ring (connection to 350 E Cermak and another connection to 427 La Salle), our additional carrier connections will restore themselves and alleviate congestion on existing on-net carrier connections from Comcast and Lumen. We will provide updates ASAP.

  • Update
    Update

    For transparency, only Comcast and Lumen on-net connections are traversing traffic at this moment, with heavy congestion in the Chicago area.

  • Update
    Update

    We have complete confirmation that our Zayo fiber ring is completely broken now. Runs from our West Chicago facility to 350 E Cermak and West Chicago to 427 La Salle St are offline. Only connections available are our on-net connections, but these are heavily congested due to the limited capacity that carriers have themselves, due to the fiber ring issues with Zayo.

  • Update
    Update

    We're seeing heavy congestion currently on available uplinks. This suggests more fiber paths have been impacted on Zayo's end. We can confirm we have a complete outage on one of our fiber ring connections to downtown. We will provide an update ASAP.

  • Identified
    Identified

    We're seeing more instability on more dark fiber paths, which is causing a deeper impact on services. We are currently evaluating the situation as several Zayo fiber paths have failed now.

  • Monitoring
    Monitoring

    Route changes were made for the lingering packet loss issues impacting some connections. This should clear up most connections. We are still awaiting updates from Zayo.

  • Update
    Update

    As our investigation suggested, Zayo confirmed they are encountering fiber issues on paths that carry some of our Tier 1 carrier connections. We will provide updates as we receive them from Zayo. Our network team is working on any lingering issues at the current moment.

  • Identified
    Identified

    Our network team has been working on this to restore any prefixes with special routing requests. We believe this to be a dark fiber cut on Zayo's network that we haul in some connections on that are impacting upstream ports.

  • Investigating
    Investigating
    We are currently investigating this incident.

May 2025 to Jul 2025

Next