Infraly, LLC - Notice history

Experiencing partial outage

Notice history

Sep 2025

WISP - Front-end Outage
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating
    At 1:33 UTC, our monitoring reported WISP front-end services became inaccessible. Our team is currently working on recovering and bringing online.
Infraly - London - Route Change - September 19th, 2025
  • Completed
    September 19, 2025 at 6:01 AMUTC
    Completed
    September 19, 2025 at 6:01 AMUTC
    Maintenance has completed successfully
  • In progress
    September 19, 2025 at 6:00 AMUTC
    In progress
    September 19, 2025 at 6:00 AMUTC
    Maintenance is now in progress
  • Planned
    September 19, 2025 at 6:00 AMUTC
    Planned
    September 19, 2025 at 6:00 AMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - London - Route Change - September 19th, 2025
    Services Affected: 194.69.160.0/24 IPs
    Start Date: September 19th, 2025 @ 6:00 UTC
    Estimated End Date: September 19th, 2025 @ 6:01 UTC

    Summary: Following maintenance by one of our third-party DDoS-protection providers in London, we will shift traffic of our 194.69.160.0/24 prefix back to their network.

    Impact on Customer: During the window, there is a possibility that connections may drop and then re-establish as routes reappear in the global routing table under another upstream. Services with filters enabled on the firewall portal will notice greater effects due to all existing traffic before the route change being invalidated and requiring re-authentication.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Chicago - Connectivity Issues
  • Resolved
    UTC
    Resolved
    At 12:54 UTC, all affected services reported healthy operational status. Our network engineering team identified the root cause as a fault within one of our on-net Tier 1 carrier connections, specifically with Cogent. While BGP sessions to this carrier remained stable throughout the event, traffic traversing this link experienced degradation due to a carrier-side fault. During the investigation, we discovered that certain remote DDoS protection tunnels were affected by asymmetric routing caused by recent configuration changes implemented prior to this event. These changes introduced scenarios where inbound and outbound traffic were not consistently using the same carrier, resulting in portions of egress traffic being routed over the impaired Cogent link. This amplified the service impact and prevented our normal failover mechanisms from fully mitigating the issue. We are deploying updated configurations to enforce strict path symmetry and improve failover behavior, ensuring that future single-carrier faults are properly isolated. We have notified Cogent and are awaiting their root cause analysis. Additional updates will be provided as new information becomes available.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Infraly - London - Carrier Maintenance - September 17th, 2025
  • Completed
    September 18, 2025 at 12:00 AMUTC
    Completed
    September 18, 2025 at 12:00 AMUTC
    Maintenance has completed successfully
  • In progress
    September 17, 2025 at 8:00 PMUTC
    In progress
    September 17, 2025 at 8:00 PMUTC
    Maintenance is now in progress
  • Planned
    September 17, 2025 at 8:00 PMUTC
    Planned
    September 17, 2025 at 8:00 PMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - London - Carrier Maintenance - September 17th, 2025
    Location Affected: London, UK Path-Routed Prefixes
    Start Date: September 17th, 2025 @ 20:00 UTC
    Estimated End Date: September 18th, 2025 @ 0:00 UTC

    Summary: Path.net, one of our third-party DDoS-protection providers in London, is performing maintenance in its London place of presence (PoP).

    Impact on Customer: During the window, there is a possibility that connections may drop and then re-establish as routes fail over to our secondary DDoS protection upstream, Neoprotect. You may see a change in latency as routes will be different throughout the maintenance. Our team will monitor throughout the window.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - Front-end Degraded
  • Resolved
    UTC
    Resolved

    At 14:56:09 UTC, a desynchronization event occurred with one node location in our database cluster, triggered by congestion at the upstream provider caused by a large inbound DDoS attack targeting another customer of that provider. This attack was not directed at our platform, but it impacted the cluster's ability to communicate effectively.

    The desynchronization caused a cascading effect, a behavior we've seen previously with our current database platform, where multiple nodes began marking themselves as unhealthy, and some briefly returned to a healthy state before failing again. At 15:17:23 UTC, the Singapore node, the last remaining healthy node, also marked itself as unhealthy while our team was working to bring the other node locations back online. This caused WISP front-end services to become fully inaccessible.

    From 15:17:23 to 15:30:01 UTC, no healthy nodes were available to handle traffic. At 15:30:02 UTC, the Singapore node recovered. Following this, our team scaled up the other locations as the remaining nodes resynchronized, fully restoring services.

    Since taking over WISP, we've gone through three different node configurations to improve reliability and ensure true geographic redundancy. Under the previous ownership, all nodes were hosted with a single provider in one facility, which did not meet our standards for redundancy. This design left the platform vulnerable to geographic outages or unexpected catastrophic events, and it lacked the true geographic load balancing that WISP is known for.

    To fix these issues, we tried the following approaches:
    - Five-node setup (3 Chicago, 2 London): Provided low inter-site latency and some stability, but if one location had problems, the entire cluster was impacted.
    - Three-node setup (Chicago, London, Singapore): Improved geographic diversity but created a new issue where a single node failure could trigger a cascading cluster failure.
    - Current five-node setup (Chicago, London, Singapore, Los Angeles, Germany): Our most stable design yet, offering two-node failure tolerance and strong geographic diversity. However, limitations with the underlying database system still cause occasional random desynchronization and nodes marking themselves as unhealthy even without actual communication problems.

    Due to these recurring issues, one of our team members has been tasked exclusively with evaluating alternative database backends. For the past several weeks, well before this incident occurred, we have been rigorously testing other solutions to prevent random desync behavior and eliminate the self-failure conditions that can cascade through the system.

    While our current five-node cluster is much more stable than past setups, this incident highlights a known issue with the current database platform under extreme conditions where latency and congestion disrupt syncing between sites. Normally, the cluster self-heals, and we've experienced this behavior before without any service impact, making this event unusual. Since moving to the current setup, these issues have been far less frequent, but they remain a known limitation. We are currently evaluating other database platforms to replace the existing system and prevent these problems entirely.

    Additionally, before this event, a separate issue occurred with the scheduler, where the queue manager became stuck due to failed API requests caused by Cloudflare API problems. This temporarily prevented certain tasks from being processed. The issue has been resolved, and we are implementing additional safeguards to ensure that future scheduler problems do not prevent other tasks from running as expected.

    We are fully committed to improving WISP's reliability through ongoing testing, architecture improvements, and a planned migration to a more resilient database platform following validation. We sincerely apologize for the disruption and thank you for your trust as we work tirelessly to enhance the stability and resilience of WISP.

  • Identified
    UTC
    Identified
    We're aware of issues communicating to panel. Our team is working on it.

Aug 2025

Infraly - Chicago - Increased Latency on GTT
  • Resolved
    UTC
    Resolved
    Following issues with GTT, a tier 1 carrier, in the Chicago region, we are observing elevated latency across several monitors on Path-routed prefixes. This is partly due to some users being re-routed to other Path PoPs. Traffic is traversing as expected, but latency may be higher than normal as a result. We will continue to monitor. As issues with GTT subside, there is potential for route changes that may impact your connection if you are normally routed via the Chicago region and are experiencing increased latency.
Physgun - Machine Maintenance - chi-s-game-1
  • Completed
    August 08, 2025 at 9:10 AMUTC
    Completed
    August 08, 2025 at 9:10 AMUTC

    Firmware has been updated with little to no impact! Maintenance is now complete.

  • In progress
    August 08, 2025 at 9:00 AMUTC
    In progress
    August 08, 2025 at 9:00 AMUTC
    Maintenance is now in progress
  • Planned
    August 08, 2025 at 9:00 AMUTC
    Planned
    August 08, 2025 at 9:00 AMUTC

    We will be performing quick firmware updates on chi-s-game-1.physgun.com this will cause a brief disruption but players will be-able to reconnect almost immediately after the firmware has been updated.

Jul 2025

WISP - Front-end Outage
  • Resolved
    UTC
    Resolved
    At 23:08 UTC, the WISP front-end services came back online and operational again. The root cause was traced to a failure on our main Chicago node, coupled with a routing issue affecting our London node. This resulted in 2 out of 3 sites falling out of sync and becoming inaccessible due to a loss of quorum. Our team quickly enacted recovery procedures in London, restoring it to production. We will continue to monitor the platform closely over the next 24 hours as we work to bring additional nodes back into global quorum. Thank you for your patience and understanding.
  • Investigating
    UTC
    Investigating
    At 22:57 UTC, our monitoring reported WISP front-end services became inaccessible. Our team is currently working on recovering and bringing online.
Infraly - Global - Upstream Maintenance - July 23rd, 2025
  • Completed
    July 23, 2025 at 3:00 PMUTC
    Completed
    July 23, 2025 at 3:00 PMUTC
    Maintenance has completed successfully
  • In progress
    July 23, 2025 at 1:00 PMUTC
    In progress
    July 23, 2025 at 1:00 PMUTC
    Maintenance is now in progress
  • Planned
    July 23, 2025 at 1:00 PMUTC
    Planned
    July 23, 2025 at 1:00 PMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Global - Upstream Maintenance - July 23rd, 2025
    Locations Affected: Chicago, Illinois & London, UK Path-Routed Prefixes
    Start Date: July 23rd, 2025 @ 13:00 UTC
    Estimated End Date: July 23rd, 2025 @ 15:00 UTC

    Summary: Path, one of our DDoS protection providers, is performing updates to critical BGP routing sessions in select PoPs (points-of-presence). We apologize for the short notice and inconvenience. We were notified ~4 hours before the maintenance is scheduled to take place.

    Impact on Customer: Disruptions are expected to be minimal and short-lived. As a result, we will be moving Chicago Path-routed traffic before the maintenance window to our on-site filtering appliances. We believe this will minimize the impact of disruptions. For London, our on-site filtering appliances are not fully active, so we are unable to move Path-routed prefixes without causing greater traffic disruptions. Our team will be monitoring throughout the maintenance. We apologize for the inconvenience.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - General Maintenance – July 15th, 2025
  • Completed
    July 15, 2025 at 8:00 PMUTC
    Completed
    July 15, 2025 at 8:00 PMUTC
    Maintenance has completed successfully
  • In progress
    July 15, 2025 at 7:00 PMUTC
    In progress
    July 15, 2025 at 7:00 PMUTC
    Maintenance is now in progress
  • Planned
    July 15, 2025 at 7:00 PMUTC
    Planned
    July 15, 2025 at 7:00 PMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    WISP - General Maintenance – July 15th, 2025

    Services Affected: WISP Front-end Services
    Start Date: July 15th, 2025 @ 19:00 UTC
    Estimated End Date: July 15th, 2025 @ 20:00 UTC

    Summary: Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability.

    Impact on Customer: As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

WISP - General Maintenance – July 14th, 2025
  • Completed
    July 14, 2025 at 11:00 AMUTC
    Completed
    July 14, 2025 at 11:00 AMUTC
    Maintenance has completed successfully
  • In progress
    July 14, 2025 at 10:00 AMUTC
    In progress
    July 14, 2025 at 10:00 AMUTC
    Maintenance is now in progress
  • Planned
    July 14, 2025 at 10:00 AMUTC
    Planned
    July 14, 2025 at 10:00 AMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    WISP - General Maintenance – July 14th, 2025

    Services Affected: WISP Front-end Services
    Start Date: July 14th, 2025 @ 10:00 UTC
    Estimated End Date: July 14th, 2025 @ 11:00 UTC

    Summary: Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability.

    Impact on Customer: As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Chicago - Network Maintenance - July 9th, 2025
  • Completed
    July 09, 2025 at 7:00 PMUTC
    Completed
    July 09, 2025 at 7:00 PMUTC
    Maintenance has completed successfully
  • In progress
    July 09, 2025 at 3:00 PMUTC
    In progress
    July 09, 2025 at 3:00 PMUTC
    Maintenance is now in progress
  • Planned
    July 09, 2025 at 3:00 PMUTC
    Planned
    July 09, 2025 at 3:00 PMUTC

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly - Chicago - Network Maintenance – July 9th, 2025

    Services Affected: Chicago, Illinois Data Center
    Start Date: July 9th, 2025 @ 15:00 UTC
    Estimated End Date: July 9th, 2025 @ 19:00 UTC

    Summary: We will be performing maintenance on our core network infrastructure to further enhance performance, reliability, and capability.

    Impact on Customer: During this window, our network team will make changes to better optimize traffic routing. While we take every possible step to minimize and isolate any impact, there is always an inherent risk of disruption as changes are applied. We appreciate your patience and understanding.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly

Jul 2025 to Sep 2025

Next