Infraly, LLC Status - Incident history

Infraly - Chicago - CHI-S-GAME-15 Maintenance – May 17th, 2026

Mon, 18 May 2026 04:59:00 +0000

Type: Maintenance

Duration: 2 hours and 16 minutes

Affected Components:

May 18, 04:59:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - CHI-S-GAME-15 Upgrade – May 17th, 2026** **Services Affected:** CHI-S-GAME-15 **Summary**: We will be performing maintenance on CHI-S-GAME-15 to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, our team will be performing routine changes that will require downtime of CHI-S-GAME-15\. The machine will be brought back online once the upgrade is complete. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, Pterodactyl & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [pterodactyl.io](http://pterodactyl.io) / [buildurly.com](https://buildurly.com).

May 18, 04:59:01 GMT+0
Identified - Maintenance is now in progress.

May 18, 07:15:10 GMT+0
Completed - Maintenance has completed successfully..

Infraly - Chicago - 45.62.160.0/24 Routing Issues

Sun, 17 May 2026 01:21:11 +0000

Type: Incident

Affected Components: , , , ,

May 17, 01:21:11 GMT+0
Identified - 45.62.160.0/24 is experiencing routing issues. This prefix is one of our last remaining leased prefixes, as we have moved solely to deploying new systems on fully-owned prefixes. From our initial investigation, this routing issue is caused by a bug in the IP lease systems, where their automation removed the ROA and RPKI objects for the IP subnet. These two objects are how internet providers know that a provider, such as us in this case, is permitted to route traffic to. We are working with the IP leaser to resolve this problem immediately. If you would like to change IP addresses to not encounter this issue, please open a ticket, and we will assist with doing so. We will provide updates as soon as possible..

May 18, 07:11:42 GMT+0
Monitoring - The IP holder has confirmed this was an issue on their end and has re-created the ROA and RPKI objects for 45.62.160.0/24\. As these changes propagate, routing to the prefix will be restored. Most networks should see connectivity return within the next few hours, though some providers may take longer to refresh their RPKI validators..

Infraly - London - LON-S-GAME-5 Maintenance – May 7th, 2026

Thu, 7 May 2026 00:30:00 +0000

Type: Maintenance

Duration: 2 hours and 30 minutes

Affected Components:

May 7, 00:30:01 GMT+0
Identified - Maintenance is now in progress.

May 7, 00:30:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - LON-S-GAME-5 Reboot – May 7th, 2026** **Services Affected:** LON-S-GAME-5 **Start Date:** May 7th, 2026 @ 00:30 UTC **Estimated End Date:** May 7th, 2026 @ 03:00 UTC **Summary**: We will be performing maintenance on LON-S-GAME-5 to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, our team will be performing routine changes that will require a reboot of LON-S-GAME-5\. The machine will be brought back online once the reboot is complete. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, Pterodactyl & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [pterodactyl.io](http://pterodactyl.io) / [buildurly.com](https://buildurly.com).

May 7, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Infraly - Global - Emergency Upstream Maintenance - February 26th, 2026

Thu, 26 Feb 2026 19:00:00 +0000

Type: Maintenance

Duration: 3 hours

Affected Components: ,

Feb 26, 19:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Global - Emergency Upstream Maintenance – February 26th, 2026** **Services Affected:** Chicago & London Data Centers **Start:** February 26th, 2026 @ 19:00 UTC **Estimated End:** February 26th, 2026 @ 22:00 UTC **Summary:** Gcore, one of our DDoS mitigation providers, is performing emergency network maintenance to their DDoS platform. **Impact on Customer:** During this window, Gcore’s network team will be making changes that may impact traffic. Our team will be monitoring and make changes to minimize impact if needed. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Feb 26, 19:00:01 GMT+0
Identified - Maintenance is now in progress..

Feb 26, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Infraly - Chicago - Upstream Maintenance – January 29th, 2026

Fri, 30 Jan 2026 09:00:00 +0000

Type: Maintenance

Duration: 5 hours

Affected Components:

Jan 30, 09:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Upstream Maintenance – January 29th, 2026** **Services Affected:** GCore Network Infrastructure **Start:** January 29th, 2026 @ 9:00AM UTC **Estimated End:** January 29th, 2026 @ 10:00AM UTC **Summary**: GCore, one of our DDoS mitigation providers will be rerouting traffic off a US South PoP very early in the morning. This is being done to ensure traffic is not impacted while the PoP goes under maintenance for upgrades. During this window, some clients who are routing through this PoP may experience a brief disruption as they are rerouted to the nearest PoP. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jan 30, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 30, 10:01:30 GMT+0
Identified - Maintenance is still in progress until upstream sends confirmation for completion..

Jan 30, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Infraly - Chicago - Carrier Network Maintenance – January 28th, 2026

Wed, 28 Jan 2026 05:00:00 +0000

Type: Maintenance

Duration: 6 hours

Affected Components:

Jan 28, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 28, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jan 28, 05:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Carrier Network Maintenance – January 28th, 2026** **Services Affected:** Chicago, Illinois Data Center **Start:** January 28th, 2026 @ 5:00 UTC **Estimated End:** January 28th, 2026 @ 11:00 UTC **Summary**: Cogent Communications, one of our upstream providers, will be performing network maintenance in the Chicago region to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, Cogent’s network team will be making changes that will impact traffic. Prior to the maintenance, our network team will isolate as much traffic as possible away from Cogent to limit any potential disruption. Traffic impact, if any, should be limited to routes that have no alternative path outside of Cogent. While we will take every possible step to minimize and isolate impact, there is always an inherent risk of disruption while changes are applied. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - Chicago - Emergency Network Maintenance – January 19th, 2026

Mon, 19 Jan 2026 12:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components:

Jan 19, 12:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Emergency Network Maintenance – January 19th, 2026** **Services Affected:** Chicago, Illinois Data Center **Start:** January 19th, 2026 @ 12:00 UTC **Estimated End:** January 19th, 2026 @ 13:00 UTC **Summary**: Gcore, one of our DDoS mitigation providers, will be performing network maintenance on our Chicago traffic to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, Gcore’s network team will be making changes to optimize traffic routing. You will see a brief interruption while these changes are applied. We apologize for the short notice – we only received notice from their team within the past few hours. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jan 19, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 19, 13:00:00 GMT+0
Completed - Maintenance has completed successfully.

Infraly - London - LON-P-VPS-1 Memory Replacement – January 18th, 2026

Sun, 18 Jan 2026 14:30:00 +0000

Type: Maintenance

Duration: 17 days and 11 hours

Affected Components: ,

Jan 18, 14:30:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - LON-P-VPS-1 Memory Replacement – January 18th, 2026** **Services Affected:** LON-P-VPS-1 **Start Date:** January 18th, 2026 @ 14:30 UTC **Estimated End Date:** January 18th, 2026 @ 18:30 UTC **Summary**: We will be performing maintenance on LON-P-VPS-1 to replace a failing memory stick. **Impact on Customer:** During this window, one of our datacenter technicians will be performing a hardware replacement that will require powering down LON-P-VPS-1\. Once the replacement is complete, the machine will be brought back online. We have scheduled this window to account for other pending tasks that may affect the exact timing of this work. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jan 18, 14:30:01 GMT+0
Identified - Maintenance is now in progress.

Jan 18, 21:29:00 GMT+0
Completed - Maintenance has completed successfully..

Infraly - Chicago - Network Maintenance – January 8th, 2026

Thu, 8 Jan 2026 12:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components:

Jan 8, 12:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Network Maintenance – January 8th, 2026** **Services Affected:** Chicago, Illinois Data Center **Start:** January 8th, 2026 @ 12:00 UTC **Estimated End:** January 8th, 2026 @ 13:00 UTC **Summary**: We will be performing maintenance on our core network infrastructure to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, our network team will make changes to better optimize traffic routing. While we take every possible step to minimize and isolate any impact, there is always an inherent risk of disruption as changes are applied. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jan 8, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 8, 13:00:00 GMT+0
Completed - Maintenance has completed successfully.

Physgun - Machine Maintenance - lon-s-game-3

Thu, 1 Jan 2026 03:40:00 +0000

Type: Maintenance

Duration: 35 minutes

Affected Components:

Jan 1, 03:40:00 GMT+0
Identified - We will be performing updates to system software on [lon-s-game-3.physgun.com](http://chi-s-game-1.physgun.com). This should only last about 30 minutes. .

Jan 1, 03:40:01 GMT+0
Identified - Maintenance is now in progress.

Jan 1, 04:14:50 GMT+0
Completed - Maintenance has completed successfully..

Infraly - London - LON-P-VPS-1 Drive Replacement – December 30th, 2025

Tue, 30 Dec 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: ,

Dec 30, 13:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - LON-P-VPS-1 Drive Replacement – December 30th, 2025** **Services Affected:** LON-P-VPS-1 **Start Date:** December 30th, 2025 @ 13:00 UTC **Estimated End Date:** December 30th, 2025 @ 17:00 UTC **Summary**: We will be performing maintenance on LON-P-VPS-1 to replace a failed redundant drive in a RAID array. **Impact on Customer:** During this window, one of our datacenter technicians will be performing a hardware replacement that will require powering down LON-P-VPS-1\. Once the replacement is complete, the machine will be brought back online. We have scheduled this window to account for other pending tasks that may affect the exact timing of this work. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Dec 30, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 30, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Physgun - Panel Maintenance - gamecp.physgun.com

Fri, 5 Dec 2025 05:00:00 +0000

Type: Maintenance

Duration: 4 hours and 28 minutes

Affected Components: ,

Dec 5, 05:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Physgun - Panel Maintenance -** [**gamecp.physgun.com**](http://gamecp.physgun.com) **\- December 5th, 2025** **Location Affected:** All shared Physgun customers **Start Date:** December 5th, 2025 @ 05:00 UTC **Estimated End Date:** December 5th, 2025 @ 09:00 UTC **Summary:** We will be performing scheduled maintenance to migrate to our new game panel. The existing panel will be unavailable for an estimated **up to 4 hours** (though the window may be extended if needed). This maintenance only affects the control panel, not live game traffic. **Impact on Customer:** During the maintenance window, the **game panel will be inaccessible**, and you will not be able to manage, restart, or configure your servers from the panel. **Your servers will remain online during the migration**, but **once the upgrade is completed, all servers will be rebooted**, causing a brief interruption in service per server. As this is a major upgrade, bugs or issues may occur after the migration; if you notice anything unusual, please **open a ticket** so our team can investigate and resolve it. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Dec 5, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 5, 09:27:56 GMT+0
Completed - Maintenance has been completed successfully. You may need to start your server back up. If you have any issues, please open a ticket with details on [billing.physgun.com](http://billing.physgun.com). .

Infraly - Global - Gcore-routed Services

Mon, 24 Nov 2025 14:02:00 +0000

Type: Incident

Duration: 38 minutes

Affected Components: ,

Nov 24, 14:02:00 GMT+0
Investigating - At 14:02 UTC, our monitoring reported connectivity issues across Gcore-routed services on our network in Chicago and London. Our team is currently reviewing and will provide updates as needed..

Nov 24, 14:40:00 GMT+0
Resolved - On November 24th, 2025, at 14:02 UTC, we experienced an issue affecting certain traffic to our Chicago and London locations that was routed through our DDoS mitigation provider, Gcore. Due to the recent and ongoing DDoS activity over the past two months from the unprecedented Aisuru botnet, we had proactively routed a majority of our traffic through Gcore for protection. During this incident, our core network remained healthy, and traffic via other upstream providers continued operating normally. BGP sessions with Gcore also remained up, and they continued to accept and propagate our announcements globally; however, an automation error on Gcore’s side caused their filtering appliances to stop forwarding ingress traffic to our endpoints in Chicago and London. Our monitoring systems alerted us at 14:02 UTC, and our team acknowledged the alert immediately. After investigation, we identified Gcore as the source of the problem and began isolating affected prefixes. At 14:33 UTC, all Chicago Gcore-routed prefixes were shifted to alternative upstream providers, and at 14:40 UTC, we did the same for London Gcore-routed prefixes. Shortly after these changes, traffic levels and service availability for both locations returned to normal. Following our outreach, Gcore confirmed that the disruption was caused by an unintended automation change on their side and acknowledged that this should not have occurred. They quickly corrected the issue, after which we carefully shifted traffic back to Gcore on a per-/24 basis when it was appropriate to do so, based on traffic levels and mitigation needs. We are also reviewing our own routing, automation, and failover procedures to further reduce the impact of any similar provider-side issues in the future. We understand the disruption and inconvenience this incident may have caused and sincerely apologize for the impact on your services. Thank you for your patience while we mitigated the issue and worked with Gcore to restore normal operation. Please note that this incident does qualify for SLA credit. If you were impacted, please open a ticket to receive your credit. Please make sure to include service(s) affected, outage time, total affected outage period, and any other necessary information to assist with administering SLA credit..

WISP - Front-end Outage

Wed, 29 Oct 2025 13:10:00 +0000

Type: Incident

Duration: 15 hours and 14 minutes

Affected Components: ,

Oct 29, 13:10:00 GMT+0
Investigating - At 13:10 UTC, our monitoring reported WISP front-end services became inaccessible due to an issue with Cloudflare. Our team is currently working on recovering and bringing online..

Oct 29, 13:37:00 GMT+0
Resolved - At 13:37 UTC, front-end services have returned back online. We're in touch with our account manager and support staff at Cloudflare. If any users are experiencing issues, please follow the proper support channels. .

Infraly - Global - Network Instability

Thu, 2 Oct 2025 19:16:00 +0000

Type: Incident

Duration: 2 days, 7 hours and 59 minutes

Affected Components: , ,

Oct 4, 19:10:23 GMT+0
Monitoring - Our network team is in the process of activating several additional ports with our Tier 1 providers. During this activation period, there may be brief instances of connectivity instability as new paths are brought online and traffic is rebalanced. Please know that our team is doing everything possible to ensure a smooth transition while strengthening our overall capacity and redundancy. These upgrades are part of our ongoing efforts to stay ahead of the evolving challenges within today’s complex network landscape. We appreciate your patience and understanding as we continue improving reliability and performance across our global network..

Oct 4, 20:11:58 GMT+0
Resolved - Additional network capacity has been successfully added without any issues. Our team continues to monitor performance closely to ensure optimal stability and reliability. In the coming days, several more ports are scheduled for activation with our Tier 1 providers, further strengthening our network capacity and resilience. We want to sincerely thank everyone who has supported us throughout this unprecedented event. Your patience and trust mean a great deal to us, and we’re proud to continue delivering solid, stable experiences for all of our customers..

Oct 2, 19:16:00 GMT+0
Monitoring - We want to provide clarity regarding the network disruptions some customers have experienced. At present, the Internet is facing major instability caused by a record-breaking DDoS botnet known as Aisuru. This botnet has launched attacks in the multi-terabit per second (Tbps) range, so large that they not only affect targeted networks but also cause congestion across Internet service providers and upstream carriers worldwide. The impact is being seen far beyond any single provider, with effects rippling across a wide range of services globally. To minimize the impact, we have been actively shifting traffic between all available DDoS protection partners at our disposal. Our team is closely monitoring attacks and implementing real-time routing adjustments to reroute traffic away from congested paths, while also coordinating with upstream carriers to ensure the fastest possible recovery when instability occurs. It is important to understand that no provider is fully immune to this botnet due to its unprecedented scale. Industry-wide, providers such as OVH, GSL, Cosmic Guard, Path, NeoProtect, and many others have all faced the same challenges. If attackers target a protection provider’s own network, temporary instability can still occur until routes are rebalanced, and even with multiple providers in place, there is no guarantee that these issues can be fully resolved when attacks of this magnitude occur. Online games are particularly sensitive to these conditions because of the way their protocols handle connections and timeouts. Even if routing is corrected quickly, the initial attack may already have caused packet loss, lag spikes, or disconnects before adjustments can take effect. Please be assured that it is in our best interest to keep your services stable and resilient. Our team is monitoring this situation around the clock and taking every possible step to reduce impact. We will continue adjusting traffic flows and working closely with our DDoS protection partners as the situation evolves. This level of attack is unprecedented, and while it is affecting providers worldwide, we remain committed to doing everything possible to shield your services and maintain stability..

Infraly - London - Carrier Maintenance - October 1st, 2025

Wed, 1 Oct 2025 20:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , ,

Oct 1, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 2, 00:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 1, 20:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Carrier Maintenance - October 1st, 2025** **Location Affected:** London, UK Path-Routed Prefixes **Start Date:** October 1st, 2025 @ 20:00 UTC **Estimated End Date:** October 2nd, 2025 @ 0:00 UTC **Summary:** Path.net, one of our third-party DDoS-protection providers in London, is performing maintenance in its London place of presence (PoP). **Impact on Customer:** During the window, there is a possibility that connections may drop and then re-establish. If deemed necessary, we will shift the prefixes to our secondary DDoS protection upstream, Neoprotect. You may see a change in latency as routes will be different throughout the maintenance. Our team will monitor throughout the window. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

WISP - Front-end Outage

Sun, 28 Sep 2025 01:33:00 +0000

Type: Incident

Duration: 37 minutes

Affected Components: ,

Sep 28, 01:33:00 GMT+0
Investigating - At 1:33 UTC, our monitoring reported WISP front-end services became inaccessible. Our team is currently working on recovering and bringing online..

Sep 28, 02:09:53 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 28, 02:10:00 GMT+0
Resolved - This incident has been resolved..

Infraly - London - Route Change - September 19th, 2025

Fri, 19 Sep 2025 06:00:00 +0000

Type: Maintenance

Duration: 1 minute

Affected Components: , ,

Sep 19, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 19, 06:01:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 19, 06:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Route Change - September 19th, 2025** **Services Affected:** 194.69.160.0/24 IPs **Start Date:** September 19th, 2025 @ 6:00 UTC **Estimated End Date:** September 19th, 2025 @ 6:01 UTC **Summary:** Following maintenance by one of our third-party DDoS-protection providers in London, we will shift traffic of our 194.69.160.0/24 prefix back to their network. **Impact on Customer:** During the window, there is a possibility that connections may drop and then re-establish as routes reappear in the global routing table under another upstream. Services with filters enabled on the firewall portal will notice greater effects due to all existing traffic before the route change being invalidated and requiring re-authentication. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - Chicago - Connectivity Issues

Thu, 18 Sep 2025 12:47:00 +0000

Type: Incident

Duration: 7 hours

Affected Components:

Sep 18, 12:47:00 GMT+0
Investigating - We are currently investigating this incident..

Sep 18, 12:54:00 GMT+0
Resolved - At 12:54 UTC, all affected services reported healthy operational status. Our network engineering team identified the root cause as a fault within one of our on-net Tier 1 carrier connections, specifically with Cogent. While BGP sessions to this carrier remained stable throughout the event, traffic traversing this link experienced degradation due to a carrier-side fault. During the investigation, we discovered that certain remote DDoS protection tunnels were affected by asymmetric routing caused by recent configuration changes implemented prior to this event. These changes introduced scenarios where inbound and outbound traffic were not consistently using the same carrier, resulting in portions of egress traffic being routed over the impaired Cogent link. This amplified the service impact and prevented our normal failover mechanisms from fully mitigating the issue. We are deploying updated configurations to enforce strict path symmetry and improve failover behavior, ensuring that future single-carrier faults are properly isolated. We have notified Cogent and are awaiting their root cause analysis. Additional updates will be provided as new information becomes available..

Infraly - London - Carrier Maintenance - September 17th, 2025

Wed, 17 Sep 2025 20:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Sep 17, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 18, 00:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 17, 20:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Carrier Maintenance - September 17th, 2025** **Location Affected:** London, UK Path-Routed Prefixes **Start Date:** September 17th, 2025 @ 20:00 UTC **Estimated End Date:** September 18th, 2025 @ 0:00 UTC **Summary:** Path.net, one of our third-party DDoS-protection providers in London, is performing maintenance in its London place of presence (PoP). **Impact on Customer:** During the window, there is a possibility that connections may drop and then re-establish as routes fail over to our secondary DDoS protection upstream, Neoprotect. You may see a change in latency as routes will be different throughout the maintenance. Our team will monitor throughout the window. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

WISP - Front-end Degraded

Sat, 13 Sep 2025 15:17:00 +0000

Type: Incident

Duration: 13 minutes

Affected Components: ,

Sep 13, 15:30:00 GMT+0
Resolved - At 14:56:09 UTC, a desynchronization event occurred with one node location in our database cluster, triggered by congestion at the upstream provider caused by a large inbound DDoS attack targeting another customer of that provider. This attack was not directed at our platform, but it impacted the cluster's ability to communicate effectively. The desynchronization caused a cascading effect, a behavior we've seen previously with our current database platform, where multiple nodes began marking themselves as unhealthy, and some briefly returned to a healthy state before failing again. At 15:17:23 UTC, the Singapore node, the last remaining healthy node, also marked itself as unhealthy while our team was working to bring the other node locations back online. This caused WISP front-end services to become fully inaccessible. From 15:17:23 to 15:30:01 UTC, no healthy nodes were available to handle traffic. At 15:30:02 UTC, the Singapore node recovered. Following this, our team scaled up the other locations as the remaining nodes resynchronized, fully restoring services. Since taking over WISP, we've gone through three different node configurations to improve reliability and ensure true geographic redundancy. Under the previous ownership, all nodes were hosted with a single provider in one facility, which did not meet our standards for redundancy. This design left the platform vulnerable to geographic outages or unexpected catastrophic events, and it lacked the true geographic load balancing that WISP is known for. To fix these issues, we tried the following approaches: \- Five-node setup (3 Chicago, 2 London): Provided low inter-site latency and some stability, but if one location had problems, the entire cluster was impacted. \- Three-node setup (Chicago, London, Singapore): Improved geographic diversity but created a new issue where a single node failure could trigger a cascading cluster failure. \- Current five-node setup (Chicago, London, Singapore, Los Angeles, Germany): Our most stable design yet, offering two-node failure tolerance and strong geographic diversity. However, limitations with the underlying database system still cause occasional random desynchronization and nodes marking themselves as unhealthy even without actual communication problems. Due to these recurring issues, one of our team members has been tasked exclusively with evaluating alternative database backends. For the past several weeks, well before this incident occurred, we have been rigorously testing other solutions to prevent random desync behavior and eliminate the self-failure conditions that can cascade through the system. While our current five-node cluster is much more stable than past setups, this incident highlights a known issue with the current database platform under extreme conditions where latency and congestion disrupt syncing between sites. Normally, the cluster self-heals, and we've experienced this behavior before without any service impact, making this event unusual. Since moving to the current setup, these issues have been far less frequent, but they remain a known limitation. We are currently evaluating other database platforms to replace the existing system and prevent these problems entirely. Additionally, before this event, a separate issue occurred with the scheduler, where the queue manager became stuck due to failed API requests caused by Cloudflare API problems. This temporarily prevented certain tasks from being processed. The issue has been resolved, and we are implementing additional safeguards to ensure that future scheduler problems do not prevent other tasks from running as expected. We are fully committed to improving WISP's reliability through ongoing testing, architecture improvements, and a planned migration to a more resilient database platform following validation. We sincerely apologize for the disruption and thank you for your trust as we work tirelessly to enhance the stability and resilience of WISP..

Sep 13, 15:17:00 GMT+0
Identified - We're aware of issues communicating to panel. Our team is working on it..

Infraly - London - Carrier Maintenance - September 8th, 2025

Mon, 8 Sep 2025 01:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Sep 8, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 8, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 8, 01:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Carrier Maintenance - September 8th, 2025** **Location Affected:** London, UK Path-Routed Prefixes **Start Date:** September 8th, 2025 @ 1:00 UTC **Estimated End Date:** September 8th, 2025 @ 5:00 UTC **Summary:** GTT Communications, one of Path's DDoS-protection upstream providers in London, is performing maintenance in its London place of presence (PoP). **Impact on Customer:** During the window, there is a possibility that connections may drop and then re-establish as routes fail over to Path's secondary upstream. You may see a change in latency as routes will be different throughout the maintenance. Our team will monitor throughout the window. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - Chicago - Increased Latency on GTT

Fri, 15 Aug 2025 14:26:58 +0000

Type: Incident

Affected Components:

Aug 15, 14:26:58 GMT+0
Resolved - Following issues with GTT, a tier 1 carrier, in the Chicago region, we are observing elevated latency across several monitors on Path-routed prefixes. This is partly due to some users being re-routed to other Path PoPs. Traffic is traversing as expected, but latency may be higher than normal as a result. We will continue to monitor. As issues with GTT subside, there is potential for route changes that may impact your connection if you are normally routed via the Chicago region and are experiencing increased latency..

Physgun - Machine Maintenance - chi-s-game-1

Fri, 8 Aug 2025 09:00:00 +0000

Type: Maintenance

Duration: 11 minutes

Affected Components:

Aug 8, 09:10:31 GMT+0
Completed - Firmware has been updated with little to no impact! Maintenance is now complete..

Aug 8, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 8, 09:00:00 GMT+0
Identified - We will be performing quick firmware updates on [chi-s-game-1.physgun.com](http://chi-s-game-1.physgun.com) this will cause a brief disruption but players will be-able to reconnect almost immediately after the firmware has been updated..

WISP - Front-end Outage

Mon, 28 Jul 2025 22:57:00 +0000

Type: Incident

Duration: 11 hours

Affected Components: ,

Jul 28, 22:57:00 GMT+0
Investigating - At 22:57 UTC, our monitoring reported WISP front-end services became inaccessible. Our team is currently working on recovering and bringing online..

Jul 28, 23:08:00 GMT+0
Resolved - At 23:08 UTC, the WISP front-end services came back online and operational again. The root cause was traced to a failure on our main Chicago node, coupled with a routing issue affecting our London node. This resulted in 2 out of 3 sites falling out of sync and becoming inaccessible due to a loss of quorum. Our team quickly enacted recovery procedures in London, restoring it to production. We will continue to monitor the platform closely over the next 24 hours as we work to bring additional nodes back into global quorum. Thank you for your patience and understanding..

Infraly - Global - Upstream Maintenance - July 23rd, 2025

Wed, 23 Jul 2025 13:00:00 +0000

Type: Maintenance

Duration: 2 hours

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , ,

Jul 23, 13:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Global - Upstream Maintenance - July 23rd, 2025** **Locations Affected:** Chicago, Illinois & London, UK Path-Routed Prefixes **Start Date:** July 23rd, 2025 @ 13:00 UTC **Estimated End Date:** July 23rd, 2025 @ 15:00 UTC **Summary:** Path, one of our DDoS protection providers, is performing updates to critical BGP routing sessions in select PoPs (points-of-presence). We apologize for the short notice and inconvenience. We were notified \~4 hours before the maintenance is scheduled to take place. **Impact on Customer:** Disruptions are expected to be minimal and short-lived. As a result, we will be moving Chicago Path-routed traffic before the maintenance window to our on-site filtering appliances. We believe this will minimize the impact of disruptions. For London, our on-site filtering appliances are not fully active, so we are unable to move Path-routed prefixes without causing greater traffic disruptions. Our team will be monitoring throughout the maintenance. We apologize for the inconvenience. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jul 23, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 23, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

WISP - General Maintenance – July 15th, 2025

Tue, 15 Jul 2025 19:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components: ,

Jul 15, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 15, 19:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **WISP - General Maintenance – July 15th, 2025** **Services Affected:** WISP Front-end Services **Start Date:** July 15th, 2025 @ 19:00 UTC **Estimated End Date:** July 15th, 2025 @ 20:00 UTC **Summary:** Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability. **Impact on Customer:** As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jul 15, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

WISP - General Maintenance – July 14th, 2025

Mon, 14 Jul 2025 10:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components: ,

Jul 14, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 14, 10:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **WISP - General Maintenance – July 14th, 2025** **Services Affected:** WISP Front-end Services **Start Date:** July 14th, 2025 @ 10:00 UTC **Estimated End Date:** July 14th, 2025 @ 11:00 UTC **Summary:** Our team will perform changes to the infrastructure of WISP to further improve performance, reliability, and capability. **Impact on Customer:** As changes are applied, front-end services have the potential for disruptions. This change does not affect game servers - WISP daemons will continue to operate as usual without interruption. We have scheduled the hour window as a precaution, but do not anticipate using the entire window if the changes apply successfully. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jul 14, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Infraly - Chicago - Network Maintenance - July 9th, 2025

Wed, 9 Jul 2025 15:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components:

Jul 9, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 9, 15:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Network Maintenance – July 9th, 2025** **Services Affected:** Chicago, Illinois Data Center **Start Date:** July 9th, 2025 @ 15:00 UTC **Estimated End Date:** July 9th, 2025 @ 19:00 UTC **Summary**: We will be performing maintenance on our core network infrastructure to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, our network team will make changes to better optimize traffic routing. While we take every possible step to minimize and isolate any impact, there is always an inherent risk of disruption as changes are applied. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly.

Jul 9, 19:00:00 GMT+0
Completed - Maintenance has completed successfully.

Infraly - Chicago - Carrier Maintenance - June 25th, 2025

Wed, 25 Jun 2025 07:00:00 +0000

Type: Maintenance

Duration: 3 hours

Affected Components: , ,

Jun 25, 10:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 25, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 25, 07:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Chicago - Carrier Maintenance - June 25th, 2025** **Location Affected:** Chicago, Illinois Path-Routed Prefixes **Start Date:** June 25th, 2025 @ 7:00 UTC **Estimated End Date:** June 25th, 2025 @ 10:00 UTC **Summary:** GTT Communications, one of Path's DDoS-protection upstream providers in Chicago, is performing maintenance in their Chicago place of presence (PoP). **Impact on Customer:** We will move all prefixes routed to Path remote protection to our Infraly filtering appliances before this maintenance starts to limit disruptions. Connection states are synced between the different locations, but there is always a possibility for disruptions to TCP and UDP traffic when the route change occurs. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - Global - Filter Update - June 18th, 2025

Wed, 18 Jun 2025 18:00:00 +0000

Type: Maintenance Duration: 16 minutes Affected Components: , Backbone Network, Europe → Warsaw, Poland, API, , Backbone Network, Colocation, Dedicated Servers, North America → Chicago, Illinois, WAF, Budget VPS, Europe → Madrid, Spain, Web Hosting, dedi.hosturly.com, North America → Miami, Florida, Europe → Frankfurt, Germany, North America → New York, New York, dedicp.physgun.com, Game Hosting, Premium VPS, North America → Silicon Valley, California, North America → Salt Lake City, Utah, Europe → London, United Kingdom, Colocation, Dedicated Servers, Web Hosting, Premium VPS, , , Europe → Sofia, Bulgaria, , North America → Dallas, Texas, Budget VPS, North America → Phoenix, Arizona, Europe → Amsterdam, Netherlands, Asia Pacific → Tokyo, Japan, Asia Pacific → Singapore, Asia Pacific → Sydney, Australia, vps.hosturly.com, gamecp.physgun.com, North America → Kansas City, Missouri, North America → Los Angeles, California, cloud.physgun.com, Remote DDoS Protection, Websites & Panels → Path.net → Hosturly → Physgun → Jun 18, 18:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 18, 18:16:04 GMT+0 - Completed - Maintenance has completed successfully. Jun 18, 18:00:00 GMT+0 - Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - Global - Filter Update - June 18th, 2025** **Locations Affected:** Chicago, Illinois & London, UK Path-Routed Prefixes **Start Date:** June 18th, 2025 @ 18:00 UTC **Estimated End Date:** June 18th, 2025 @ 19:00 UTC **Summary:** Path, one of our DDoS protection providers, is performing updates to its filtering stack to further improve performance, reliability, and capability. **Impact on Customer:** Services using the TCP service (symmetric) filter may see a brief interruption as the filter update is rolled out. They expect the overall window of impact to be less than 15 minutes. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe to alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com)

[Rescheduled] Infraly - London - Carrier Maintenance - June 17th, 2025

Tue, 17 Jun 2025 21:00:00 +0000

Type: Maintenance

Duration: 5 hours

Affected Components: ,

Jun 18, 02:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 17, 21:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 17, 21:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Carrier Maintenance - June 17th, 2025** **Location Affected:** London, UK Path-Routed Prefixes **Start Date:** June 17th, 2025 @ 21:00 UTC **Estimated End Date:** June 18th, 2025 @ 2:00 UTC **Summary:** One of our DDoS-protection upstreams is performing maintenance in their London place of presence (PoP) to further improve performance, reliability, and capability. **Impact on Customer:** Customers will experience a route change at the beginning and end of this maintenance as traffic is routed between London and Amsterdam. We will actively monitor throughout the maintenance period and make adjustments to upstream-routed prefixes if required. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - London - Carrier Maintenance - June 16th - 18th, 2025

Mon, 16 Jun 2025 04:00:00 +0000

Type: Maintenance

Duration: 2 days and 3 hours

Affected Components: , , , , , , , , , ,

Jun 18, 07:00:00 GMT+0
Completed - Maintenance has completed successfully..

Jun 16, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 16, 04:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - Carrier Maintenance - June 16th - 18th, 2025** **Location Affected:** London, UK Data Center **Start Date:** June 16th, 2025 @ 4:00 UTC **Estimated End Date:** June 18th, 2025 @ 7:00 UTC **Summary:** One of our on-net carriers is performing firmware updates on their equipment to further improve performance, reliability, and capability over a period of 3 days. **Impact on Customer:** Work will be carried out in stages over the 3-day window, with activity scheduled each day between 4:00 and 7:00 UTC. While the majority of this work is expected to be non-disruptive, there may be brief interruptions lasting approximately 15–20 minutes during device reboots required for the firmware updates. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Infraly - London - LON-P-VPS-1 Upgrade – June 13th, 2025

Fri, 13 Jun 2025 17:00:00 +0000

Type: Maintenance

Duration: 2 days, 2 hours and 3 minutes

Affected Components: ,

Jun 14, 00:00:00 GMT+0
Identified - This service remains offline due to a critical component failure that occurred following a recent upgrade. Unfortunately, the standby hardware we had on-site was also found to be non-functional, which further delayed recovery efforts. An emergency replacement has been ordered and is being delivered today, Saturday. Once received, our team will immediately proceed with installation and testing to bring the machine back online and fully operational. We sincerely apologize for the ongoing disruption. Per our Service Level Agreement (SLA), account credits will be issued to all affected customers. Further updates will be provided as progress is made..

Jun 15, 19:03:00 GMT+0
Completed - The affected machine is now back online following the arrival and installation of a replacement motherboard earlier today. Our team completed the swap and validation successfully, and all VMs on LON-P-VPS-1 should now be fully operational. To clarify, we initially attempted to restore service using our standby hardware. However, that system was also found to have a failed motherboard and would not post, which required us to order a replacement. While delivery was originally expected on Saturday based on tracking information, the part did not arrive until today, Sunday, which unfortunately extended the outage. We sincerely apologize for the disruption and inconvenience this has caused. We’re providing a full month of service credit to all affected customers, going beyond what’s covered under our SLA. No ticket is necessary; this credit will be automatically applied to the impacted services. Thank you again for your patience and understanding throughout this incident..

Jun 13, 17:00:00 GMT+0
Identified - Dear user, Please refer to a very important maintenance announcement that may have an impact on hosting services: **Infraly - London - LON-P-VPS-1 Upgrade – June 13th, 2025** **Services Affected:** LON-P-VPS-1 **Start Date:** June 13th, 2025 @ 17:00 UTC **Estimated End Date:** June 14th, 2025 @ 00:00 UTC **Summary**: We will be performing maintenance on LON-P-VPS-1 to further enhance performance, reliability, and capability. **Impact on Customer:** During this window, one of our datacenter technicians will be performing a hardware upgrade that will require powering down LON-P-VPS-1\. Once the upgrade is complete, the machine will be brought back online. We have scheduled this window to account for other pending tasks that may affect the exact timing of this work. We appreciate your patience and understanding. **Updates:** We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (). If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support! Sincerely, Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA e: [hello@infraly.co](mailto:hello@infraly.co) p: +1 (833) INF-RALY w: [infraly.co](http://infraly.co) / [hosturly.com](http://hosturly.com) / [physgun.com](http://physgun.com) / [wisp.gg](https://wisp.gg) / [buildurly.com](https://buildurly.com).

Jun 13, 17:00:00 GMT+0
Identified - Maintenance is now in progress..

WISP - Front-end Degraded

Fri, 6 Jun 2025 16:04:00 +0000

Type: Incident

Duration: 46 minutes

Affected Components: ,

Jun 6, 16:04:00 GMT+0
Identified - We have identified an issue with the front-end not showing up for recent viewers of the billing and game panels. This is caused by Bunny.net, one of our third-party caching services. This issue is affecting all Bunny customers, making it widespread at the moment. We will provide updates as we have them from Bunny..

Jun 6, 16:50:00 GMT+0
Resolved - This incident has been resolved. Bunny has implemented a permanent fix. As such, we have removed our temporary workaround. Bunny has not provided an official status post for this issue, but did acknowledge it was an issue on their end. They claim it is "the impact was low as we caught it early and was already fixing it." We have pressed them on why their standard is such, as it did impact both US and EU streaming locations. We do not anticipate them doing such action in the future as many other Bunny customers were impacted by this..

Jun 6, 16:46:27 GMT+0
Monitoring - We've added a temporary edge rule to resolve the issue caused by Bunny. We will continue to monitor..

WISP - Front-end Outage

Thu, 5 Jun 2025 18:13:00 +0000

Type: Incident

Duration: 13 hours

Affected Components:

Jun 5, 18:13:00 GMT+0
Identified - We are currently investigating this incident. It is related to syncing between all sites..

Jun 5, 18:26:00 GMT+0
Resolved - A fix has been implemented and we are monitoring..

Infraly - Chicago – Partial Network Outage

Mon, 19 May 2025 01:46:06 +0000

Type: Incident

Duration: 18 days, 22 hours and 46 minutes

Affected Components:

May 19, 01:46:06 GMT+0
Investigating - We are currently investigating this incident. This is only affecting certain traffic landing on edge0\. .

May 19, 02:38:00 GMT+0
Investigating - Our network team is still working on this issue. We will share information when we are able to do so..

May 19, 05:09:00 GMT+0
Monitoring - The majority of impacted services are operational at this time. We do still have a limited number of machines (less than 40) that are experiencing issues still. We are still working on this and will provide updates as soon as possible. .

May 19, 03:45:52 GMT+0
Investigating - We are continuing to investigate route instability isolated to our Chicago location. The issue is stemming from communication inconsistencies between our edge routers at the spine layer. This is resulting in frequent route flapping and intermittent disruptions in traffic flow. Our network engineering team is working to rebalance route propagation and mitigate further impact. We'll continue to provide additional updates as soon as possible..

May 19, 09:20:52 GMT+0
Resolved - All impacted services have been operational for 4+ hours now. We will be providing a detailed post-mortem for this incident later this week, following internal discussions with management and our network team. This is the most widespread/impactful outage we've ever experienced as a company, so transparency and ensuring customers are fully aware of what happened, why it happened, what we did to fix, and further prevent it from happening again are a top priority. We sincerely apologize for any inconvenience this incident has caused..

Infraly - Chicago – Partial Network Outage

Tue, 13 May 2025 03:21:44 +0000

Type: Incident

Duration: 6 days, 5 hours and 49 minutes

Affected Components:

May 13, 03:21:44 GMT+0
Investigating - We are currently investigating this incident..

May 13, 04:05:45 GMT+0
Identified - As our investigation suggested, Zayo confirmed they are encountering fiber issues on paths that carry some of our Tier 1 carrier connections. We will provide updates as we receive them from Zayo. Our network team is working on any lingering issues at the current moment..

May 13, 04:59:45 GMT+0
Monitoring - Route changes were made for the lingering packet loss issues impacting some connections. This should clear up most connections. We are still awaiting updates from Zayo. .

May 13, 05:52:27 GMT+0
Identified - We're seeing more instability on more dark fiber paths, which is causing a deeper impact on services. We are currently evaluating the situation as several Zayo fiber paths have failed now..

May 13, 05:58:32 GMT+0
Identified - We're seeing heavy congestion currently on available uplinks. This suggests more fiber paths have been impacted on Zayo's end. We can confirm we have a complete outage on one of our fiber ring connections to downtown. We will provide an update ASAP. .

May 13, 06:18:46 GMT+0
Identified - For transparency, only Comcast and Lumen on-net connections are traversing traffic at this moment, with heavy congestion in the Chicago area. .

May 13, 03:41:29 GMT+0
Identified - Our network team has been working on this to restore any prefixes with special routing requests. We believe this to be a dark fiber cut on Zayo's network that we haul in some connections on that are impacting upstream ports..

May 13, 10:35:20 GMT+0
Monitoring - Following this unprecedented event, all carrier connections have come online without issues and are traversing traffic with healthy throughput checks. We will continue to monitor over the next few hours. Please stay tuned for the post-mortem, as that will be coming with additional information on this incident. Thank you again for your patience and understanding. .

May 13, 10:25:22 GMT+0
Identified - We have been monitoring our connections and see that all failed connections are online and operational now. We will begin turning up these connections now..

May 13, 06:04:30 GMT+0
Identified - We have complete confirmation that our Zayo fiber ring is completely broken now. Runs from our West Chicago facility to 350 E Cermak and West Chicago to 427 La Salle St are offline. Only connections available are our on-net connections, but these are heavily congested due to the limited capacity that carriers have themselves, due to the fiber ring issues with Zayo..

May 13, 07:14:36 GMT+0
Identified - Zayo has given us an estimated restoration time of 6:00 am CT. Upon restoration of our fiber ring (connection to 350 E Cermak and another connection to 427 La Salle), our additional carrier connections will restore themselves and alleviate congestion on existing on-net carrier connections from Comcast and Lumen. We will provide updates ASAP..

May 19, 09:10:46 GMT+0
Resolved - We will be posting a detailed post-mortem regarding this incident later this week. We have internal meetings with our network engineers as well as Zayo this week to discuss this incident and the exact failures on Zayo's end..

Infraly - Chicago Partial Blip

Tue, 29 Apr 2025 20:36:21 +0000

Type: Incident

Duration: 1 day, 17 hours and 30 minutes

Affected Components:

Apr 29, 20:36:21 GMT+0
Investigating - We are currently investigating this incident and will report back once we have more details..

Apr 29, 20:54:07 GMT+0
Identified - Our network team is investigating deeper, but we can confirm that one of our core spine switches flapped all LACP interfaces. Physical interfaces remained online. The flapping of the LACP interfaces caused some connections between our top-of-rack switches and carriers to flap that were connected to said spine switch. More info will be posted soon. .

Apr 29, 22:39:51 GMT+0
Identified - Our network team has concluded the investigation, and we have determined the root cause to be failing direct attached copper (DAC) cables between our spines and edge routers. Due to the severity of failing cables, the impact was felt more drastically throughout the network. As a result, we plan to replace the failing cables in the early morning of Wednesday, April 30th, at off-peak hours to minimize disruptions. We will provide a time for this in the coming hours. We apologize for any inconvenience this incident has caused. .

Apr 30, 00:36:22 GMT+0
Identified - As promised in our previous message, we will be performing replacements of these DACs with backups from our storage at an off-peak time. This is scheduled to occur at 10:00 UTC on Wednesday, April 30th. We will provide an update when this is completed..

Apr 30, 10:00:14 GMT+0
Identified - We currently have someone on-site who is prepping these changes. Please note that this is expected to occur in the next 15 minutes. .

Apr 30, 10:14:28 GMT+0
Monitoring - The cables have been replaced, and the network is traversing traffic as expected. We will continue to monitor logs over the next 24 hours for interface errors. As stated previously, we sincerely apologize for any inconvenience this incident has caused. .

May 1, 14:06:26 GMT+0
Resolved - Traffic has been traversing the network over the past 24 hours without issues. As a result, we are marking this as resolved .

Hosturly - Legacy CHI-P-VPS-1 Hardware Failure

Tue, 29 Apr 2025 19:14:00 +0000

Type: Incident

Duration: 2 days, 11 hours and 28 minutes

Affected Components:

Apr 29, 19:14:00 GMT+0
Investigating - At 19:14 UTC, our monitoring systems have detected an outage on one of our legacy CHI-P-VPS-1 machines. Our team is currently investigating. .

Apr 29, 20:12:00 GMT+0
Identified - Our team has concluded that this is due to hardware failure. We are currently awaiting a technician to replace the motherboard and get this back in production. We will provide updates as soon as possible. .

Apr 30, 01:55:00 GMT+0
Identified - A technician has replaced the suspected failed hardware. Unfortunately, the replacement is showing the same signs of failure. Due to this being legacy hardware, we have scaled down our backups as we no longer use these systems in any other production environment. Our systems team has ordered the replacement part and is expected to be delivered tomorrow, May 1st, 2025\. Upon delivery, we will perform another replacement. .

May 2, 06:41:55 GMT+0
Resolved - CHI-P-VPS-1 is now fully operational following the successful installation of a new motherboard. All virtual machines are online and functioning as expected. This incident is now considered resolved. As this event qualifies for SLA credit, we have proactively credited affected accounts with three months of additional service free of charge. This exceeds our SLA requirements, and we’ve done so as a gesture of goodwill and to reaffirm our commitment to providing reliable service and going above and beyond for our customers. We appreciate your patience and understanding..

WISP - Daemon Communication Problems

Mon, 28 Apr 2025 11:39:00 +0000

Type: Incident

Duration: 1 hour and 18 minutes

Affected Components:

Apr 28, 11:39:00 GMT+0
Investigating - Our team is aware of an issue with the daemon communicating with the panel for certain actions..

Apr 28, 12:57:27 GMT+0
Resolved - At 12:23 UTC, all daemon communication issues were fully resolved. After a thorough investigation, we determined that the disruption was caused by a botnet operated by a customer against our legacy infrastructure services, which will remain in use until May 1st. The legacy infrastructure's upstream provider, upon detecting this abnormal traffic, believed it to be malicious, and their automated systems imposed a block on all daemon communication ports. Unfortunately, this block also prevented legitimate traffic from reaching client services. Once we identified the cause, we immediately worked with the provider to remove the block, and services returned to normal operation. We sincerely apologize for the disruption and any inconvenience this may have caused. We have since implemented additional safeguards to better detect and mitigate similar incidents moving forward. We want to assure you that as of May 1st, all services will be transitioned to our new, more resilient infrastructure, which is specifically designed to prevent these types of issues. Please make sure to prepare for such changes as we have previously announced..

Infraly - London - Outage

Wed, 23 Apr 2025 13:56:01 +0000

Type: Incident

Duration: 4 days, 22 hours and 45 minutes

Affected Components:

Apr 24, 12:08:44 GMT+0
Investigating - We're still working with datacenter staff on-site. We will provide an update ASAP..

Apr 23, 13:56:01 GMT+0
Investigating - We are currently investigating this incident. It appears to be an issue at the DC itself. Currently awaiting communications from them..

Apr 23, 14:21:10 GMT+0
Investigating - We are actively communicating with the facility in London, and this incident is marked in their systems as a high-priority 1 issue. We will provide more information as soon as possible. .

Apr 23, 14:54:02 GMT+0
Investigating - Our inquiry with the facility is being handled directly with the datacenter site manager. We will provide more information as soon as possible. .

Apr 23, 15:01:29 GMT+0
Investigating - Service was restored just a few minutes ago after the datacenter completed work on their end. We are currently awaiting information from them and are investigating with our own systems. .

Apr 23, 16:07:47 GMT+0
Investigating - We have detected another drop in our London facility. The datacenter manager is continuing to work on this and doing tests now that this second drop has occurred unexpectedly. We will provide an update shortly. .

Apr 23, 17:00:52 GMT+0
Monitoring - Following an investigation with the datacenter manager on-site, we have concluded this to be a bad patch panel box that contains all of our fiber connections from carriers. In light of this finding, the datacenter has replaced the patch panel box and all optics connected to our core devices. We hope this will resolve any outages we have experienced today. We will continue to monitor over the next 24 hours. If this does recur, the datacenter has reassured us that there will be technical staff on-site who will be able to take actions to bring connections back up. .

Apr 24, 09:57:25 GMT+0
Resolved - Traffic has been stable for 12+ hours. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused. .

Apr 24, 11:40:47 GMT+0
Investigating - We are currently investigating an issue in our London location this morning. DC engineers are already working on this. .

Apr 24, 13:59:55 GMT+0
Monitoring - Services have been stable and online after the facility completed changes on our fiber connections. We will continue to monitor. .

Apr 28, 12:40:47 GMT+0
Resolved - Traffic has been stable for several days now. As a result, we will be marking this as resolved. We very much appreciate your understanding and apologize for any inconvenience this incident may have caused..

Infraly - Infrequent NYC PoP Packet Loss

Sun, 13 Apr 2025 16:23:14 +0000

Type: Incident

Duration: 1 day

Affected Components:

Apr 13, 16:23:14 GMT+0
Identified - Traffic routing through the New York City Point of Presence (PoP) may have experienced intermittent packet loss. Our upstream provider has confirmed that the issue originates within their internet carrier and is currently working on a resolution..

Apr 14, 16:23:00 GMT+0
Resolved - Traffic volumes have been stable for some time now. As a result, we are marking this as resolved. .

Hosturly & Physgun - Chicago - CHI-P-VPS-12 Outage

Sun, 16 Mar 2025 23:44:00 +0000

Type: Incident

Duration: 17 hours and 58 minutes

Affected Components:

Mar 17, 00:24:27 GMT+0
Investigating - Our team is currently updating the motherboard's BIOS for this machine. We will continue to look at logs in the OS upon being finished. .

Mar 16, 23:44:00 GMT+0
Investigating - Our monitoring systems reported that CHI-P-VPS-12, a premium VPS hosting machine, experienced several outages between 23:44 UTC on March 16th, 2025 and 00:09 UTC on March 17th, 2025\. We are currently investigating and will provide updates as soon as possible..

Mar 17, 00:43:25 GMT+0
Monitoring - The BIOS and kernel were updated. The machine is currently rebooting into the new kernel. We will continue to review logs on the machine for additional findings and implement changes as needed with notification. .

Mar 18, 00:43:00 GMT+0
Resolved - CHI-P-VPS-12 has been stable for 12+ hours now. We are marking this resolved, but have disabled new deployments on this hypervisor until we reach a week of stability. Please note that this incident does qualify for SLA credit. If you were impacted, please open a ticket to receive your credit..

Infraly - Global - Path-Routed Services - Packet Loss

Sun, 16 Mar 2025 01:06:05 +0000

Type: Incident

Duration: 22 hours and 42 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , ,

Mar 16, 01:06:05 GMT+0
Identified - We are aware of issues currently with Path-routed services worldwide. We are currently in communication with them as this is outside of our control. Please note that no protection offering is perfect when not controlled by us: Path and Neoprotect both encountered similar issues today..

Mar 16, 05:52:49 GMT+0
Identified - Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 6:00 UTC today and is expected to end at 7:00 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance. .

Mar 16, 15:42:29 GMT+0
Identified - Path has notified us that one of their upstreams, GTT, is performing emergency maintenance to hopefully correct the issues previously mentioned in this incident post. This maintenance will begin at 15:45 UTC today and is expected to end at 19:45 UTC. Please note that there may be a blip in connectivity for Path-routed services that traverse the PoPs undergoing maintenance..

Mar 16, 23:48:00 GMT+0
Resolved - At 19:48 UTC on March 16th, 2025, we received the all-clear from Path that GTT had completed their maintenance. Since then, there have been no issues with connections. We will continue to monitor as always. .

Hosturly - Chicago - CHI-P-VPS-4 Outage

Wed, 12 Mar 2025 13:50:00 +0000

Type: Incident

Duration: 9 hours and 32 minutes

Mar 12, 13:50:00 GMT+0
Investigating - Our monitoring systems reported that CHI-P-VPS-4, a legacy premium VPS hosting machine, experienced an outage at 13:50 UTC on March 12th, 2025\. We are currently investigating and will provide updates as soon as possible..

Mar 12, 15:15:18 GMT+0
Identified - We have identified that the motherboard has died for this machine, and we are arranging for a technician to move the drives to a standby machine we have in place. We will continue to keep you updated..

Mar 12, 23:22:19 GMT+0
Resolved - CHI-P-VPS-4 is back online and operational upon the drives being moved to a new machine. All VMs are online and working. As a result, we are marking this incident as complete. Please note that this incident does qualify for SLA credit. If you were impacted, please open a ticket to receive your credit..

Infraly - Global - Path-Routed Services - Packet Loss

Tue, 11 Mar 2025 19:31:56 +0000

Type: Incident

Duration: 10 hours and 47 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , ,

Mar 11, 19:31:56 GMT+0
Investigating - We're noticing packet loss from our upstream and have contacted them awaiting a resolution..

Mar 11, 19:42:57 GMT+0
Monitoring - Looks like it was a short spike, we have not witnessed an interruption since - still awaiting a response..

Mar 11, 21:11:59 GMT+0
Resolved - We've received an update from upstream, this was caused by a very massive and complex attack. They have sent us documentation of changes being implemented and filtering to prevent further interruptions from this attack..

Mar 12, 00:57:29 GMT+0
Monitoring - Unfortunately this issue has continued to reappear, we will be keeping this incident opened until we are fully confident the issue is resolved..

Mar 12, 06:19:21 GMT+0
Resolved - Monitoring has shown Path-routed services traversing the network as expected for several hours now. We have continued progressing on the on-prem solution to resolve these issues. We will release a maintenance announcement shortly regarding further implementation..

Infraly - Chicago - Path-Routed Services - Packet Loss

Thu, 6 Mar 2025 17:54:00 +0000

Type: Incident

Affected Components:

Mar 6, 17:54:00 GMT+0
Resolved - At 17:54 UTC, our monitoring systems detected a disruption in Path-routed traffic in Chicago. At 17:58 UTC, monitoring showed traffic was traversing as expected. Our investigation has concluded that the outage was caused by a JunOS bug triggered when additional monitoring filters were applied to the equipment in Chicago. While the same configuration was successfully implemented in London without issues, the equipment in Chicago, running an older firmware version, experienced disruptions. After reviewing the logs and comparing firmware versions, we identified the JunOS bug as the root cause. This issue has been documented and reported to Juniper for further analysis. We understand the importance of keeping our customers informed about potential disruptions. We aim to minimize unnecessary notifications, especially for routine changes to the routers that are expected to have no impact on customers. Given the frequency of these changes, it is not practical to announce every update. However, we issue alerts for planned changes that are anticipated to cause potential disruptions. Since this configuration had been successfully applied in London without issues, we did not anticipate any problems in Chicago and, therefore, did not send out a notification. We apologize for any inconvenience caused. If you have any questions or concerns, please contact our client relations team via ticket, and we will be happy to assist you..

Infraly - Chicago - Path-Routed Services Packet Loss

Tue, 25 Feb 2025 00:05:54 +0000

Type: Incident

Duration: 6 hours and 28 minutes

Affected Components:

Feb 25, 00:05:54 GMT+0
Identified - We are currently investigating the packet loss in Chicago for Path-routed prefixes. All other traffic is stable and not encountering issues. We will message as soon as we have updates from Path..

Feb 25, 06:34:20 GMT+0
Resolved - This incident has been resolved. We should expect larger pipes to be installed in more locations this week to completely mitigate against these types of issues..

Infraly - Chicago - Path-Routed Services - Packet Loss

Wed, 19 Feb 2025 22:11:22 +0000

Type: Incident

Duration: 6 hours and 7 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

Feb 19, 22:11:22 GMT+0
Identified - We have confirmed that our upstream provider is experiencing heavy packet loss and they are investigating..

Feb 19, 22:49:58 GMT+0
Identified - We will be re-routing traffic to a more stable upstream very soon. Within the next 10 to 20 minutes you may notice a drop as we migrate live traffic onto a more reliable provider. Apologies once again for the issues, we are working swiftly to resolve all of this..

Feb 19, 23:12:16 GMT+0
Identified - Path is about to bring up their Miami PoP, they are waiting for their ISP. This will add extra capacity which is very much needed. We are also going to be load balancing our traffic between both providers shortly..

Feb 19, 23:29:22 GMT+0
Identified - We are currently moving over a Prefix in our London location to another provider, you may notice a slight drop..

Feb 20, 03:19:40 GMT+0
Monitoring - Path will be pushing Frankfurt back online soon. Traffic will rebalance once this PoP is placed into production. .

Feb 20, 04:18:45 GMT+0
Resolved - Path's Frankfurt is now fully online..