Infraly, LLC - Notice history

We have received an update from the data center regarding the outage we experienced earlier today. According to SBA Edge's network operations center, there was an issue related to the redundant configuration between their new and old network equipment, which prevented a clean route change until it was resolved. They have assured us that this has been corrected and are actively working on their core network maintenance (which was not supposed to affect us).

As of now, we will continue to monitor traffic traversing our network and provide updates if needed. Once again, we sincerely apologize for the inconvenience this has caused.

Our monitoring systems have detected that the systems have come back online. We are still awaiting more information from the data center about why we were affected. We will continue to post updates as we get them.

From our investigation, the system froze completely and did not respond to authoritative commands to restart services. As a result, we performed a power reset command, and the machine is now back online and communicating with our monitoring systems. We will continue to monitor and provide updates.

Our monitoring systems detected our Path-routed services offline at 11:47 AM ET, August 19th, 2023. As a result, our network team swiftly, within seconds, routed our Path-routed services to our alternative carrier blend. However, due to the large number of prefixes we announce (greater than 10), the Border Gateway Protocol (BGP) sessions with our facility’s meet-me-room were flapping due to a configuration error on their end that prevented us from announcing more than 10 prefixes simultaneously through that blend. Our network team quickly contacted the data center network operations center (NOC) to adjust the advertised prefix amount to be greater than 10 to prevent any such further incident from happening again. At this time, services are routed as expected via our blend and Path.net.

Path.net did confirm the instability of their DDoS mitigation network for remote protection clients today. They are still investigating at the current moment and will be providing more information. We will continue to monitor the stability of our Path-routed services and route traffic as needed if there are any future issues.

In overview, non-Path-routed and Path-routed services both experienced a drop in traffic today. Total downtime today was just over 10 minutes for Path-routed services, and around 9 minutes for non-Path-routed services.

Our monitoring systems reported that CHI-P-VPS-2, a premium VPS hosting machine, experienced a kernel panic at 2:19 PM ET, on July 30th, 2023. As a result, we have taken the necessary precautions to prevent this in the future, and the machine returned back online 4 minutes later at 2:23 PM ET. We will continue to monitor over the next 24 hours.

Our monitoring systems reported that CHI-P-VPS-4, a premium VPS hosting machine, experienced a kernel panic at 2:13 PM ET, on July 30th, 2023. As a result, we have taken the necessary precautions to prevent this in the future, and the machine returned back online 5 minutes later at 2:18 PM ET. CHI-P-VPS-4 has experienced these kernel panics before, and we have taken the precaution of winding down deployments on this machine. We anticipate on moving existing clients off this machine and onto another premium VPS server over the next week or so due to frequent issues. We will be in touch with clients on this machine within the next 36 hours.

All Path-routed services were returned back to Path connectivity roughly near 5 PM ET after our team concluded their network was operating as expected. Our team has been monitoring since the route change and has experienced no issues. As a result, we will be marking this incident as resolved. Connectivity issues actually seen by clients totaled less than 1 minute today thanks to alternative carriers.

Path has confirmed they are having issues in the Chicago region. We see that their endpoint has returned online after being routed to their NY PoP. We will stay on alternative carriers until we see traceroutes return to CHI PoP to prevent additional latency increases.