Infraly, LLC - Notice history

Experiencing partially degraded performance

Notice history

Feb 2025

Path-Routed Services - Chicago PoP Migration
  • Completed
    March 01, 2025 at 7:47 AM
    Completed
    March 01, 2025 at 7:47 AM

    The Chicago PoP has been fully restored! You should notice latency normalizing once again.

  • In progress
    March 01, 2025 at 6:51 AM
    In progress
    March 01, 2025 at 6:51 AM

    Path is now in the process of migrating back onto the Chicago PoP.

  • Planned
    February 28, 2025 at 11:30 PM
    Planned
    February 28, 2025 at 11:30 PM

    One of our upstream providers, Path, is actively restoring their Chicago PoP, and we expect minimal disruptions throughout the process. The only potential brief interruption will occur during the final activation phase when traffic is routed back through Chicago.

    We are expecting updates as work continues, this will be the last PoP that needs to be migrated in the US until Phoenix, however Phoenix is TBD and is not expected to be migrated for a while.

    Once live, we will transition away from Kansas City (KC), and latency should begin to normalize.

  • In progress
    February 28, 2025 at 11:19 PM
    In progress
    February 28, 2025 at 11:19 PM

    We have been informed that Path Miami has been brought fully online today. Chicago is also expected to come online very soon and we have requested for it to come back online closer to off-peak hours to avoid serious interruptions.

    You will notice a full outage while Path brings up Chicago and we are routed back onto Chicago. However after Chicago is brought up latency will start normalizing again as we will no longer be routed to different PoPs!

Infraly - Chicago - Addressing Frequent Blips
  • Completed
    February 28, 2025 at 11:00 PM
    Completed
    February 28, 2025 at 11:00 PM
    Maintenance has completed successfully
  • Update
    February 28, 2025 at 1:50 AM
    In progress
    February 28, 2025 at 1:50 AM

    Maintenance for this has been completed today. We've found the root cause to these Layer 2 issues relating to the router appliance itself. Luckily we have built the network to be redundant, so we will be shifting traffic over to a working router while we work on getting this defective one replaced.

    Again this maintenance will continue to take place at 9:00 AM Central time to avoid dropping players in the peak hours of the day. In the mean time some changes have been made to lessen the effects of these spikes.

    To clarify, we would've had this done today, however the backup router was still in the process of being setup and was paused while on-site was being developed. We needed a few prerequisites completed from our upstream before we could shift any traffic onto it. Which is why we are continuing this tomorrow as it went too far into prime time today to justify dropping traffic.

  • Update
    February 27, 2025 at 8:33 PM
    In progress
    February 27, 2025 at 8:33 PM

    We are pushing some changes, you may notice a slight interruption.

  • In progress
    February 27, 2025 at 3:00 PM
    In progress
    February 27, 2025 at 3:00 PM
    Maintenance is now in progress
  • Planned
    February 27, 2025 at 3:00 PM
    Planned
    February 27, 2025 at 3:00 PM

    Our network team has identified the root cause of the frequent short drops that have plagued the network over the past few days. We understand the severity of these drops and have identified the root cause. To permanently resolve these issues, we will be conducting emergency maintenance at 9:00 AM Central Time. During this maintenance, you will experience brief disruptions as mass changes will be done. We apologize in advance for any inconvenience this may pose to you, especially over these past few weeks where we have suffered instability from one of our upstream providers as well as the network maintenance we conducted to upgrade our core late last month. To be frank, the timing could not have been worse.

    For transparency, the issue stems from a Layer 2 issue inside our EVPN-VXLAN network, which is affecting our spine switches that process essentially all traffic on our network. As a result, this was felt widespread and not just related to one upstream or one cabinet. This issue appears to have sprung from the preparation of onboarding our on-prem protection suite into our core. We have been working at a fast pace to get that online and operational in hopes of shifting all traffic over in order to provide stability.

    We know the past few weeks have been difficult, and we recognize the importance of a reliable network for our customers and are tirelessly working to improve our offerings.

Infraly - Chicago - More Progress With On-Site Protection
  • Completed
    March 01, 2025 at 3:00 PM
    Completed
    March 01, 2025 at 3:00 PM
    Maintenance has completed successfully
  • Update
    February 25, 2025 at 10:44 PM
    In progress
    February 25, 2025 at 10:44 PM

    We have completed maintenance for today, this extension saw no external loss to production switches. We have made a lot of real, meaningful progress today. We are around the home stretch now, simply a few things to tidy up before we push more live traffic to onsite!

    We will be continuing our work again tomorrow at 9:00 AM Central to avoid any potential interruptions during peak hours.

  • Update
    February 25, 2025 at 7:36 PM
    In progress
    February 25, 2025 at 7:36 PM

    Hello everyone, apologies for the short notice but we will be extending the previous maintenance for a few more hours.

    Why? Because today we’ve made some crazy good progress, and we think onsite can realistically be done very soon. We everything is online we even have some live traffic being pushed to it.

    To iterate again, this will be the solution for us to attempt to seamlessly move traffic from the global upstream to our localized mitigation stack when they have issues. Effectively keeping uptime with as little to no drops as possible.Thank you for bearing with us, we will send out some cool pictures soon

  • In progress
    February 25, 2025 at 3:00 PM
    In progress
    February 25, 2025 at 3:00 PM
    Maintenance is now in progress
  • Planned
    February 25, 2025 at 3:00 PM
    Planned
    February 25, 2025 at 3:00 PM

    At specifically 9AM Central Time we will be continuing development with our onsite network. We apologize for any interruptions this may cause, we are doing this early in the AM to ensure we impact as little traffic as possible.

    We understand the frequent issues from upstream isn't making this an easy pill to swallow, but we must continue work on this to ensure we are independent from future network issues once Onsite goes live.

    To give context on what onsite will bring for those who may have missed it:

    Infraly Onsite DDoS Protection will allow us to have a localized Path environment that will be isolated from the global network. This will allow us to attempt to seamlessly, without dropping traffic move between the global network and local network when needed.

    Take for example the issues that we have been facing now, these are all from the global network with Onsite we would be-able to push traffic onto our local mitigation stack, effectively isolating us from global network issues, ensuring full uptime. This also works in reverse, if the local network was to go offline for some reason, we would route back onto the global network, while again attempting to not drop any traffic.

    This system is extremely complex, and has been under development for a while now, we are speeding up progress but also doing so at the least impactful times, we understand no downtime is ideal, however this must be done in order to ensure we have a fully redundant network. I am sure everyone is tired of these upstream issues, we are too, and we truly believe onsite is the solution we are all looking for.

Infraly - Chicago - Path-Routed Services Packet Loss
  • Resolved
    Resolved

    This incident has been resolved. We should expect larger pipes to be installed in more locations this week to completely mitigate against these types of issues.

  • Identified
    Identified

    We are currently investigating the packet loss in Chicago for Path-routed prefixes. All other traffic is stable and not encountering issues. We will message as soon as we have updates from Path.

Physgun – Emergency Maintenance on LON-S-GAME-1 – February 23rd, 2025
  • Completed
    February 23, 2025 at 9:35 PM
    Completed
    February 23, 2025 at 9:35 PM

    This has been completed successfully. All servers on LON-GAME-1 should be back up and running. We sincerely apologize for the inconvenience this may have caused.

    As mentioned previously, we will be providing SLA credit to all clients affected. Please open a ticket on the billing site if your game server was affected by this. We will verify and then credit accordingly.

  • In progress
    February 23, 2025 at 6:50 PM
    In progress
    February 23, 2025 at 6:50 PM
    Maintenance is now in progress
  • Planned
    February 23, 2025 at 6:50 PM
    Planned
    February 23, 2025 at 6:50 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Emergency Maintenance on LON-S-GAME-1 – February 23rd, 2025

    Services Affected: Physgun's LON-S-GAME-1
    Start Date: February 23rd, 2025 @ 18:50 UTC
    Estimated End Date: February 23rd, 2025 @ 22:50 UTC

    Summary: We have identified a critical issue on LON-S-GAME-1 and need to perform updates and migrations to ensure stability for the machine.

    Impact on Customer: During the maintenance, users on LON-S-GAME-1 will experience a service disruption. Due to the short notice of this maintenance, we are offering SLA credit for this. More details will come for this later.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Jan 2025

Physgun - eu-phs-ger - Server Startup Issues
  • Resolved
    Resolved

    We've been monitoring this issue for a few days, and we are confident it has been fully resolved since the fix was implemented!

  • Monitoring
    Monitoring

    The issue seems to have been resolved, we believe this was due to an outage from our Upstream provider. We are still awaiting further details from them on the incident while we continue monitor EU game servers.

  • Investigating
    Investigating

    We're aware of issues effecting people not being able to startup their server. It's being investigated by our team right now and we're working fast to get this issue resolved.

WISP – Emergency General Maintenance – January 29th, 2025
  • Completed
    January 29, 2025 at 8:51 PM
    Completed
    January 29, 2025 at 8:51 PM

    The maintenance has been completed with under 5 minutes of downtime during the maintenance period.

  • In progress
    January 29, 2025 at 8:00 PM
    In progress
    January 29, 2025 at 8:00 PM
    Maintenance is now in progress
  • Planned
    January 29, 2025 at 8:00 PM
    Planned
    January 29, 2025 at 8:00 PM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Emergency General Maintenance – January 29th, 2025

    Services Affected: WISP Front-end Services
    Start Date: January 29th, 2025 @ 3:00 PM ET
    Estimated End Date: January 29th, 2025 @ 4:00 PM ET

    Summary: Our team will perform emergency maintenance to prepare to implement the new US infrastructure later this week.

    Impact on Customer: Front-end services will become inaccessible for a short period in this maintenance. We have scheduled an hour window for caution, but do not anticipate a loss of accessibility for the entire window. Please note that game servers are not affected - WISP daemons will continue to operate.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Physgun - eu-phs-ger12.physgun.com - Hardware fault
  • Resolved
    Resolved

    Remote hands has replaced the defective CPU and we have thoroughly stress tested the new one and passed a couple other checks to ensure it's good to go!

    Your servers should now be accessible once again if you were on the affected node.

  • Identified
    Identified

    We've identified a hardware fault with the CPU for eu-phs-ger12, we've put in a ticket for remote hands to fix the issue ASAP. This is the cause of the recent unusual connection loss events, after the remote hands finishes this issue will be fixed!

[EXTENDED] Infraly – Emergency Network Maintenance – January 22nd - 29th, 2025
  • Completed
    January 30, 2025 at 5:44 AM
    Completed
    January 30, 2025 at 5:44 AM

    We are now exiting our maintenance period, and we sincerely apologize for the extended duration and any disruptions this may have caused. We understand that any downtime can cause frustration, and we deeply regret the impact on your operations.

    This overhaul of our core infrastructure was necessary to enhance performance, reliability, and scalability, ensuring a stronger network moving forward. We truly appreciate your patience and support.

    Further details on the enhancements will be shared soon.

  • Update
    January 29, 2025 at 6:57 PM
    In progress
    January 29, 2025 at 6:57 PM

    Over the next 10 minutes, a blip will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 29, 2025 at 6:05 PM
    In progress
    January 29, 2025 at 6:05 PM

    Over the next 20 minutes, there may be a few blips that will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 29, 2025 at 4:41 PM
    In progress
    January 29, 2025 at 4:41 PM

    Over the next 60 minutes, there may be a few blips that will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 29, 2025 at 1:45 AM
    In progress
    January 29, 2025 at 1:45 AM

    Over the next 20 minutes, there may be a few blips that will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 29, 2025 at 1:14 AM
    In progress
    January 29, 2025 at 1:14 AM

    Over the next 20 minutes, there may be a few blips that will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 28, 2025 at 11:24 PM
    In progress
    January 28, 2025 at 11:24 PM

    Over the next 60 minutes, there may be a few blips that will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 28, 2025 at 2:22 AM
    In progress
    January 28, 2025 at 2:22 AM

    Our network engineering team is finished for the day and will regroup tomorrow. Our goal is to finish this maintenance period tomorrow before the Wednesday deadline.

    Tomorrow, Tuesday, January 28th, 2025, we anticipate possible disruptions as we move traffic. We will provide updates whenever any disruption of service is expected. We strongly encourage you to subscribe to this maintenance post to receive real-time notifications.

    We sincerely apologize for the inconvenience caused by this maintenance period. Once this maintenance is completed, we will be releasing an announcement regarding the upgrades completed and how they will lead to a more reliable service offering.

  • Update
    January 27, 2025 at 5:43 AM
    In progress
    January 27, 2025 at 5:43 AM

    Please note that this maintenance period has been extended to Wednesday, January 29th, 2025 11:59 pm CT. This extension is necessary to ensure the successful implementation of critical upgrades.

    Our network engineering team is finished for the night and will regroup tomorrow morning to begin additional work.

    We will provide updates whenever any disruption of service is expected. We strongly encourage you to subscribe to this maintenance post to receive real-time notifications.

    We sincerely apologize for the inconvenience caused by this maintenance period. Once this maintenance is completed, we will be releasing an announcement regarding the upgrades completed and how they will lead to a more reliable service offering.

  • Update
    January 27, 2025 at 3:27 AM
    In progress
    January 27, 2025 at 3:27 AM

    Over the next 20 minutes, a blip will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 27, 2025 at 1:42 AM
    In progress
    January 27, 2025 at 1:42 AM

    Within the next 20 minutes, a blip will disrupt some traffic routed over a core network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • Update
    January 26, 2025 at 9:57 PM
    In progress
    January 26, 2025 at 9:57 PM

    At 4:05 pm CT, roughly 10 minutes from now, a blip will disrupt some traffic routed over an edge network device. We apologize for the inconvenience. Our maintenance is one step closer to being completed after this. :)

  • In progress
    January 23, 2025 at 1:00 AM
    In progress
    January 23, 2025 at 1:00 AM
    Maintenance is now in progress
  • Planned
    January 23, 2025 at 1:00 AM
    Planned
    January 23, 2025 at 1:00 AM

    Dear user,

    Please refer to a very important maintenance announcement that may have an impact on hosting services:

    Infraly – Emergency Network Maintenance – January 22nd - 26th, 2025
    Location Affected: Chicago, Illinois
    Start Date: January 22nd, 2025
    Estimated End Date: January 26th, 2025

    Summary: We will be performing maintenance on our core network infrastructure to further improve performance, reliability, and capability.

    Impact on Customer: We will be limiting the impact on customers by isolating equipment that is being replaced while we upgrade our core. During this time, redundant equipment will remain online. Rack colocation clients with a single connection will be subject to at least one brief disruption as those cross-connects are moved to the upgraded equipment. While we have extensively planned this maintenance behind the scenes to limit any impact, please note that there is always the possibility for a brief disruption of service throughout maintenance periods. All possible measures are being taken to minimize disruptions.

    Updates: We will provide updates via our status page throughout the maintenance. You can subscribe for alerts on our status page (https://status.infraly.co).

    If you have any questions, don't hesitate to contact our client relations team via support ticket. Thank you again for your cooperation and support!

    Sincerely,

    Infraly, LLC DBA Hosturly, Physgun, WISP, & Buildurly
    a: 1636 N Cedar Crest Blvd, #122, Allentown, PA 18104, USA
    e: hello@infraly.co
    p: +1 (833) INF-RALY
    w: infraly.co / hosturly.com / physgun.com / wisp.gg / buildurly.com

Infraly - Path-Routed Services Packet Loss
  • Resolved
    Resolved

    Path-routed services have been stable for 12+ hours. As a result, we are marking this as resolved. We will provide an update for the root cause upon Path's investigation concluding.

  • Monitoring
    Monitoring

    We are continuing to monitor. Our tools show that services have been stable for the past 5-10 minutes. We are awaiting a reply from Path regarding the root cause of the disruption.

  • Investigating
    Investigating

    Our monitoring systems have detected packet loss on Path-routed services throughout the early morning. We have been in contact with them regarding the ongoing issues. Our team is continuing to monitor and will provide updates as needed. We apologize for the inconvenience.

Dec 2024

No notices reported this month

Dec 2024 to Feb 2025

Next