Infraly, LLC - Notice history

All impacted services have been operational for 4+ hours now. We will be providing a detailed post-mortem for this incident later this week, following internal discussions with management and our network team.

This is the most widespread/impactful outage we've ever experienced as a company, so transparency and ensuring customers are fully aware of what happened, why it happened, what we did to fix, and further prevent it from happening again are a top priority.

We sincerely apologize for any inconvenience this incident has caused.

The majority of impacted services are operational at this time. We do still have a limited number of machines (less than 40) that are experiencing issues still. We are still working on this and will provide updates as soon as possible.

We are continuing to investigate route instability isolated to our Chicago location. The issue is stemming from communication inconsistencies between our edge routers at the spine layer. This is resulting in frequent route flapping and intermittent disruptions in traffic flow.

Our network engineering team is working to rebalance route propagation and mitigate further impact. We'll continue to provide additional updates as soon as possible.

Our network team is still working on this issue. We will share information when we are able to do so.

We will be posting a detailed post-mortem regarding this incident later this week. We have internal meetings with our network engineers as well as Zayo this week to discuss this incident and the exact failures on Zayo's end.

Following this unprecedented event, all carrier connections have come online without issues and are traversing traffic with healthy throughput checks. We will continue to monitor over the next few hours. Please stay tuned for the post-mortem, as that will be coming with additional information on this incident. Thank you again for your patience and understanding.

We have been monitoring our connections and see that all failed connections are online and operational now. We will begin turning up these connections now.

Zayo has given us an estimated restoration time of 6:00 am CT. Upon restoration of our fiber ring (connection to 350 E Cermak and another connection to 427 La Salle), our additional carrier connections will restore themselves and alleviate congestion on existing on-net carrier connections from Comcast and Lumen. We will provide updates ASAP.

For transparency, only Comcast and Lumen on-net connections are traversing traffic at this moment, with heavy congestion in the Chicago area.

We have complete confirmation that our Zayo fiber ring is completely broken now. Runs from our West Chicago facility to 350 E Cermak and West Chicago to 427 La Salle St are offline. Only connections available are our on-net connections, but these are heavily congested due to the limited capacity that carriers have themselves, due to the fiber ring issues with Zayo.

We're seeing heavy congestion currently on available uplinks. This suggests more fiber paths have been impacted on Zayo's end. We can confirm we have a complete outage on one of our fiber ring connections to downtown. We will provide an update ASAP.

We're seeing more instability on more dark fiber paths, which is causing a deeper impact on services. We are currently evaluating the situation as several Zayo fiber paths have failed now.

Route changes were made for the lingering packet loss issues impacting some connections. This should clear up most connections. We are still awaiting updates from Zayo.

As our investigation suggested, Zayo confirmed they are encountering fiber issues on paths that carry some of our Tier 1 carrier connections. We will provide updates as we receive them from Zayo. Our network team is working on any lingering issues at the current moment.

Our network team has been working on this to restore any prefixes with special routing requests. We believe this to be a dark fiber cut on Zayo's network that we haul in some connections on that are impacting upstream ports.

Traffic has been traversing the network over the past 24 hours without issues. As a result, we are marking this as resolved

The cables have been replaced, and the network is traversing traffic as expected. We will continue to monitor logs over the next 24 hours for interface errors. As stated previously, we sincerely apologize for any inconvenience this incident has caused.

We currently have someone on-site who is prepping these changes. Please note that this is expected to occur in the next 15 minutes.

As promised in our previous message, we will be performing replacements of these DACs with backups from our storage at an off-peak time. This is scheduled to occur at 10:00 UTC on Wednesday, April 30th. We will provide an update when this is completed.

Our network team has concluded the investigation, and we have determined the root cause to be failing direct attached copper (DAC) cables between our spines and edge routers. Due to the severity of failing cables, the impact was felt more drastically throughout the network. As a result, we plan to replace the failing cables in the early morning of Wednesday, April 30th, at off-peak hours to minimize disruptions. We will provide a time for this in the coming hours. We apologize for any inconvenience this incident has caused.

Our network team is investigating deeper, but we can confirm that one of our core spine switches flapped all LACP interfaces. Physical interfaces remained online.

The flapping of the LACP interfaces caused some connections between our top-of-rack switches and carriers to flap that were connected to said spine switch. More info will be posted soon.

CHI-P-VPS-1 is now fully operational following the successful installation of a new motherboard. All virtual machines are online and functioning as expected. This incident is now considered resolved. As this event qualifies for SLA credit, we have proactively credited affected accounts with three months of additional service free of charge. This exceeds our SLA requirements, and we’ve done so as a gesture of goodwill and to reaffirm our commitment to providing reliable service and going above and beyond for our customers. We appreciate your patience and understanding.

A technician has replaced the suspected failed hardware. Unfortunately, the replacement is showing the same signs of failure. Due to this being legacy hardware, we have scaled down our backups as we no longer use these systems in any other production environment. Our systems team has ordered the replacement part and is expected to be delivered tomorrow, May 1st, 2025. Upon delivery, we will perform another replacement.

Our team has concluded that this is due to hardware failure. We are currently awaiting a technician to replace the motherboard and get this back in production. We will provide updates as soon as possible.

At 12:23 UTC, all daemon communication issues were fully resolved. After a thorough investigation, we determined that the disruption was caused by a botnet operated by a customer against our legacy infrastructure services, which will remain in use until May 1st.

The legacy infrastructure's upstream provider, upon detecting this abnormal traffic, believed it to be malicious, and their automated systems imposed a block on all daemon communication ports. Unfortunately, this block also prevented legitimate traffic from reaching client services.

Once we identified the cause, we immediately worked with the provider to remove the block, and services returned to normal operation. We sincerely apologize for the disruption and any inconvenience this may have caused. We have since implemented additional safeguards to better detect and mitigate similar incidents moving forward.

We want to assure you that as of May 1st, all services will be transitioned to our new, more resilient infrastructure, which is specifically designed to prevent these types of issues. Please make sure to prepare for such changes as we have previously announced.

Traffic has been stable for several days now. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused.

Services have been stable and online after the facility completed changes on our fiber connections. We will continue to monitor.

We're still working with datacenter staff on-site. We will provide an update ASAP.

We are currently investigating an issue in our London location this morning. DC engineers are already working on this.

Traffic has been stable for 12+ hours. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused.

Following an investigation with the datacenter manager on-site, we have concluded this to be a bad patch panel box that contains all of our fiber connections from carriers.

In light of this finding, the datacenter has replaced the patch panel box and all optics connected to our core devices. We hope this will resolve any outages we have experienced today. We will continue to monitor over the next 24 hours. If this does recur, the datacenter has reassured us that there will be technical staff on-site who will be able to take actions to bring connections back up.

We have detected another drop in our London facility. The datacenter manager is continuing to work on this and doing tests now that this second drop has occurred unexpectedly. We will provide an update shortly.

Service was restored just a few minutes ago after the datacenter completed work on their end. We are currently awaiting information from them and are investigating with our own systems.

Our inquiry with the facility is being handled directly with the datacenter site manager. We will provide more information as soon as possible.

We are actively communicating with the facility in London, and this incident is marked in their systems as a high-priority 1 issue. We will provide more information as soon as possible.

Traffic volumes have been stable for some time now. As a result, we are marking this as resolved.

CHI-P-VPS-12 has been stable for 12+ hours now. We are marking this resolved, but have disabled new deployments on this hypervisor until we reach a week of stability. Please note that this incident does qualify for SLA credit. If you were impacted, please open a ticket to receive your credit.

The BIOS and kernel were updated. The machine is currently rebooting into the new kernel. We will continue to review logs on the machine for additional findings and implement changes as needed with notification.

Our team is currently updating the motherboard's BIOS for this machine. We will continue to look at logs in the OS upon being finished.

At 19:48 UTC on March 16th, 2025, we received the all-clear from Path that GTT had completed their maintenance. Since then, there have been no issues with connections. We will continue to monitor as always.

Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 15:45 UTC today and is expected to end at 19:45 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance.

Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 6:00 UTC today and is expected to end at 7:00 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance.