Infraly, LLC - WISP - Daemon Communication Problems – Incident details

WISP - Daemon Communication Problems

Resolved
Degraded performance
Started 3 months agoLasted about 1 hour

Affected

WISP

Operational from 11:39 AM to 12:57 PM

Panel

Operational from 11:39 AM to 12:57 PM

Updates
  • Resolved
    Resolved

    At 12:23 UTC, all daemon communication issues were fully resolved. After a thorough investigation, we determined that the disruption was caused by a botnet operated by a customer against our legacy infrastructure services, which will remain in use until May 1st.

    The legacy infrastructure's upstream provider, upon detecting this abnormal traffic, believed it to be malicious, and their automated systems imposed a block on all daemon communication ports. Unfortunately, this block also prevented legitimate traffic from reaching client services.

    Once we identified the cause, we immediately worked with the provider to remove the block, and services returned to normal operation. We sincerely apologize for the disruption and any inconvenience this may have caused. We have since implemented additional safeguards to better detect and mitigate similar incidents moving forward.

    We want to assure you that as of May 1st, all services will be transitioned to our new, more resilient infrastructure, which is specifically designed to prevent these types of issues. Please make sure to prepare for such changes as we have previously announced.

  • Investigating
    Investigating

    Our team is aware of an issue with the daemon communicating with the panel for certain actions.