Infraly, LLC - Infraly - Chicago - Addressing Frequent Blips – Maintenance details

Experiencing partially degraded performance

Infraly - Chicago - Addressing Frequent Blips

Completed
Scheduled for February 27, 2025 at 3:00 PM – February 28, 2025 at 11:00 PM

Affects

Chicago, Illinois

Under maintenance from 3:00 PM to 11:00 PM

Backbone Network

Under maintenance from 3:00 PM to 11:00 PM

Updates
  • Completed
    February 28, 2025 at 11:00 PM
    Completed
    February 28, 2025 at 11:00 PM
    Maintenance has completed successfully
  • Update
    February 28, 2025 at 1:50 AM
    Update
    February 28, 2025 at 1:50 AM

    Maintenance for this has been completed today. We've found the root cause to these Layer 2 issues relating to the router appliance itself. Luckily we have built the network to be redundant, so we will be shifting traffic over to a working router while we work on getting this defective one replaced.

    Again this maintenance will continue to take place at 9:00 AM Central time to avoid dropping players in the peak hours of the day. In the mean time some changes have been made to lessen the effects of these spikes.

    To clarify, we would've had this done today, however the backup router was still in the process of being setup and was paused while on-site was being developed. We needed a few prerequisites completed from our upstream before we could shift any traffic onto it. Which is why we are continuing this tomorrow as it went too far into prime time today to justify dropping traffic.

  • Update
    February 27, 2025 at 8:33 PM
    Update
    February 27, 2025 at 8:33 PM

    We are pushing some changes, you may notice a slight interruption.

  • In progress
    February 27, 2025 at 3:00 PM
    In progress
    February 27, 2025 at 3:00 PM
    Maintenance is now in progress
  • Planned
    February 27, 2025 at 3:00 PM
    Planned
    February 27, 2025 at 3:00 PM

    Our network team has identified the root cause of the frequent short drops that have plagued the network over the past few days. We understand the severity of these drops and have identified the root cause. To permanently resolve these issues, we will be conducting emergency maintenance at 9:00 AM Central Time. During this maintenance, you will experience brief disruptions as mass changes will be done. We apologize in advance for any inconvenience this may pose to you, especially over these past few weeks where we have suffered instability from one of our upstream providers as well as the network maintenance we conducted to upgrade our core late last month. To be frank, the timing could not have been worse.

    For transparency, the issue stems from a Layer 2 issue inside our EVPN-VXLAN network, which is affecting our spine switches that process essentially all traffic on our network. As a result, this was felt widespread and not just related to one upstream or one cabinet. This issue appears to have sprung from the preparation of onboarding our on-prem protection suite into our core. We have been working at a fast pace to get that online and operational in hopes of shifting all traffic over in order to provide stability.

    We know the past few weeks have been difficult, and we recognize the importance of a reliable network for our customers and are tirelessly working to improve our offerings.