Infraly, LLC - London Outage – Incident details

North America → Phoenix, Arizona experiencing major outage

London Outage

Monitoring
Major outage
Started 3 days ago

Affected

London, United Kingdom Data Center

Major outage from 1:56 PM to 9:57 AM, Operational from 9:57 AM to 11:40 AM, Major outage from 11:40 AM to 1:59 PM, Operational from 1:59 PM to 12:00 AM

Core Network Infrastructure

Major outage from 1:56 PM to 9:57 AM, Operational from 9:57 AM to 11:40 AM, Major outage from 11:40 AM to 1:59 PM, Operational from 1:59 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    Services have been stable and online after the facility completed changes on our fiber connections. We will continue to monitor.

  • Update
    Update

    We're still working with datacenter staff on-site. We will provide an update ASAP.

  • Investigating
    Investigating

    We are currently investigating an issue in our London location this morning. DC engineers are already working on this.

  • Resolved
    Resolved

    Traffic has been stable for 12+ hours. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused.

  • Monitoring
    Monitoring

    Following an investigation with the datacenter manager on-site, we have concluded this to be a bad patch panel box that contains all of our fiber connections from carriers.

    In light of this finding, the datacenter has replaced the patch panel box and all optics connected to our core devices. We hope this will resolve any outages we have experienced today. We will continue to monitor over the next 24 hours. If this does recur, the datacenter has reassured us that there will be technical staff on-site who will be able to take actions to bring connections back up.

  • Update
    Update

    We have detected another drop in our London facility. The datacenter manager is continuing to work on this and doing tests now that this second drop has occurred unexpectedly. We will provide an update shortly.

  • Update
    Update

    Service was restored just a few minutes ago after the datacenter completed work on their end. We are currently awaiting information from them and are investigating with our own systems.

  • Update
    Update

    Our inquiry with the facility is being handled directly with the datacenter site manager. We will provide more information as soon as possible.

  • Update
    Update

    We are actively communicating with the facility in London, and this incident is marked in their systems as a high-priority 1 issue. We will provide more information as soon as possible.

  • Investigating
    Investigating

    We are currently investigating this incident. It appears to be an issue at the DC itself. Currently awaiting communications from them.