Infraly, LLC - Notice history

All systems have been reporting no issues for the past few hours. All panels are now operating as expected. We are now going to mark this resolved. If you encounter any issues, do not hesitate to message support. Once again, we apologize for the inconvenience this may have caused.

The infrastructure machine is in a recovery state and has been operating as expected for a few hours. We will continue to monitor. Please note that you may still continue to experience increased load times/timeouts when attempting to visit such pages. We apologize for the inconvenience.

The additional load balancer has been running smoothly for the past few hours. All packet loss and additional latency issues have been resolved. We will now be marking this incident as resolved.

Following up regarding information from Path: they attempted to add another load balancer pursuant to the packet loss issue. While usually seamless, this addition ultimately resulted in the immediate removal of it, which caused the flap in one of our traffic sessions. They are currently working on resolving and bringing back into rotation.

One of our load-balanced BGP sessions with Path flapped, which resulted in roughly 50% of traffic experiencing a drop in connectivity while it reconvened on our other BGP session that remained online and operational. More information to follow.

The hypervisor has remained stable for approximately three days now. Our team will continue monitoring as usual, but we will mark this outage resolved. If you have any questions or concerns, do not hesitate to reach out to our client relations team via live chat, ticket, or phone.

The hypervisor is back online and functioning. VMs should be booting currently. Our team is continuing to investigate the root cause. We will provide updates as permitted.

Operations at Equinix CH1 have begun to stabilize thanks to the swift response by their team. As a result, we are beginning to rotate more of our traffic back to our carrier endpoints downtown.

Update from Equinix: 10 out of 15 portable chillers have been activated. Temperatures have continued to decrease to an average 88F.

Update from Equinix: 3 more portable chillers have been installed. Twelve more are expected to come online shortly. Temperatures have continued to decrease to an average of 95F.

Update from Equinix: 4 chillers are currently active, and temperatures are starting to stabilize. More equipment is being brought on-site to assist with cooling.

Update from Equinix: 5 out of 15 portable chillers have arrived on-site. More are expected to arrive shortly.

Update from Equinix: 2 out of 8 chillers are now fully operational. A third chiller is operating at a reduced capacity. Temperatures in CH1 are beginning to decrease as more chillers are returning online. Portable chillers are expected to be on-site shortly.

Update from Equinix: One chiller is fully operational now, with another one operating at reduced capacity. Technicians are still working to restore the necessary 4-out-of-8 chillers to return the temperatures back to normal conditions. Portable chillers should have been delivered around 9 PM CT and would be setup by technicians as soon as possible.

Update from Equinix: Technicians continue to troubleshoot the chiller plant failure. IBX Critical Facility Engineers have deployed all available floor fans, opened all available doors and have engaged local rental company to source additional fans and portable spot coolers.

Our automation system has rolled over traffic to our on-net carriers due to losing complete connectivity to our endpoint in Equinix CH1. As traffic was moved, you may have experienced a brief disconnect or increased load times as routes readjusted on these alternative carriers. We will continue to monitor and provide updates.

The machine has turned back online after the hardware swap at 3:04 PM ET. We will continue to monitor for any continued issues.

All Path-routed services have been fully restored back with symmetric routing. Monitoring continues to show traffic traversing the network without issues. We appreciate your understanding!

We plan on transitioning Path-routed services back to their full capacity at 3 AM ET. Please note that this may cause a minuscule drop in packets and/or disconnects while the routes are picked up. We will provide an update when this has been completed. We apologize for the inconvenience this may cause.