At 12:54 UTC, all affected services reported healthy operational status.
Our network engineering team identified the root cause as a fault within one of our on-net Tier 1 carrier connections, specifically with Cogent. While BGP sessions to this carrier remained stable throughout the event, traffic traversing this link experienced degradation due to a carrier-side fault.
During the investigation, we discovered that certain remote DDoS protection tunnels were affected by asymmetric routing caused by recent configuration changes implemented prior to this event. These changes introduced scenarios where inbound and outbound traffic were not consistently using the same carrier, resulting in portions of egress traffic being routed over the impaired Cogent link. This amplified the service impact and prevented our normal failover mechanisms from fully mitigating the issue.
We are deploying updated configurations to enforce strict path symmetry and improve failover behavior, ensuring that future single-carrier faults are properly isolated.
We have notified Cogent and are awaiting their root cause analysis. Additional updates will be provided as new information becomes available.
