Infraly, LLC - Notice history

Traffic has been traversing the network over the past 24 hours without issues. As a result, we are marking this as resolved

The cables have been replaced, and the network is traversing traffic as expected. We will continue to monitor logs over the next 24 hours for interface errors. As stated previously, we sincerely apologize for any inconvenience this incident has caused.

We currently have someone on-site who is prepping these changes. Please note that this is expected to occur in the next 15 minutes.

As promised in our previous message, we will be performing replacements of these DACs with backups from our storage at an off-peak time. This is scheduled to occur at 10:00 UTC on Wednesday, April 30th. We will provide an update when this is completed.

Our network team has concluded the investigation, and we have determined the root cause to be failing direct attached copper (DAC) cables between our spines and edge routers. Due to the severity of failing cables, the impact was felt more drastically throughout the network. As a result, we plan to replace the failing cables in the early morning of Wednesday, April 30th, at off-peak hours to minimize disruptions. We will provide a time for this in the coming hours. We apologize for any inconvenience this incident has caused.

Our network team is investigating deeper, but we can confirm that one of our core spine switches flapped all LACP interfaces. Physical interfaces remained online.

The flapping of the LACP interfaces caused some connections between our top-of-rack switches and carriers to flap that were connected to said spine switch. More info will be posted soon.

CHI-P-VPS-1 is now fully operational following the successful installation of a new motherboard. All virtual machines are online and functioning as expected. This incident is now considered resolved. As this event qualifies for SLA credit, we have proactively credited affected accounts with three months of additional service free of charge. This exceeds our SLA requirements, and we’ve done so as a gesture of goodwill and to reaffirm our commitment to providing reliable service and going above and beyond for our customers. We appreciate your patience and understanding.

A technician has replaced the suspected failed hardware. Unfortunately, the replacement is showing the same signs of failure. Due to this being legacy hardware, we have scaled down our backups as we no longer use these systems in any other production environment. Our systems team has ordered the replacement part and is expected to be delivered tomorrow, May 1st, 2025. Upon delivery, we will perform another replacement.

Our team has concluded that this is due to hardware failure. We are currently awaiting a technician to replace the motherboard and get this back in production. We will provide updates as soon as possible.

At 12:23 UTC, all daemon communication issues were fully resolved. After a thorough investigation, we determined that the disruption was caused by a botnet operated by a customer against our legacy infrastructure services, which will remain in use until May 1st.

The legacy infrastructure's upstream provider, upon detecting this abnormal traffic, believed it to be malicious, and their automated systems imposed a block on all daemon communication ports. Unfortunately, this block also prevented legitimate traffic from reaching client services.

Once we identified the cause, we immediately worked with the provider to remove the block, and services returned to normal operation. We sincerely apologize for the disruption and any inconvenience this may have caused. We have since implemented additional safeguards to better detect and mitigate similar incidents moving forward.

We want to assure you that as of May 1st, all services will be transitioned to our new, more resilient infrastructure, which is specifically designed to prevent these types of issues. Please make sure to prepare for such changes as we have previously announced.

Traffic has been stable for several days now. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused.

Services have been stable and online after the facility completed changes on our fiber connections. We will continue to monitor.

We're still working with datacenter staff on-site. We will provide an update ASAP.

We are currently investigating an issue in our London location this morning. DC engineers are already working on this.

Traffic has been stable for 12+ hours. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused.

Following an investigation with the datacenter manager on-site, we have concluded this to be a bad patch panel box that contains all of our fiber connections from carriers.

In light of this finding, the datacenter has replaced the patch panel box and all optics connected to our core devices. We hope this will resolve any outages we have experienced today. We will continue to monitor over the next 24 hours. If this does recur, the datacenter has reassured us that there will be technical staff on-site who will be able to take actions to bring connections back up.

We have detected another drop in our London facility. The datacenter manager is continuing to work on this and doing tests now that this second drop has occurred unexpectedly. We will provide an update shortly.

Service was restored just a few minutes ago after the datacenter completed work on their end. We are currently awaiting information from them and are investigating with our own systems.

Our inquiry with the facility is being handled directly with the datacenter site manager. We will provide more information as soon as possible.

We are actively communicating with the facility in London, and this incident is marked in their systems as a high-priority 1 issue. We will provide more information as soon as possible.

Traffic volumes have been stable for some time now. As a result, we are marking this as resolved.

CHI-P-VPS-12 has been stable for 12+ hours now. We are marking this resolved, but have disabled new deployments on this hypervisor until we reach a week of stability. Please note that this incident does qualify for SLA credit. If you were impacted, please open a ticket to receive your credit.

The BIOS and kernel were updated. The machine is currently rebooting into the new kernel. We will continue to review logs on the machine for additional findings and implement changes as needed with notification.

Our team is currently updating the motherboard's BIOS for this machine. We will continue to look at logs in the OS upon being finished.

At 19:48 UTC on March 16th, 2025, we received the all-clear from Path that GTT had completed their maintenance. Since then, there have been no issues with connections. We will continue to monitor as always.

Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 15:45 UTC today and is expected to end at 19:45 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance.

Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 6:00 UTC today and is expected to end at 7:00 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance.

This incident has been resolved. We should expect larger pipes to be installed in more locations this week to completely mitigate against these types of issues.