The issue with Pyth Pro API and WebSocket streaming endpoints have been resolved. All systems are now operating normally.
The incident was caused by a major outage at TeraSwitch, a data center provider hosting part of our infrastructure. At the time, two of four NATS nodes were hosted in TeraSwitch facilities. When those nodes became unreachable, NATS lost quorum and dependent Router services became unavailable.
We had already been working to improve infrastructure redundancy and reduce provider concentration. This work is now being accelerated, including onboarding an additional infrastructure provider and distributing critical services across independent failure domains.
The goal is to ensure that a future outage affecting any single infrastructure provider does not result in a broader service disruption.
Resolved
The issue with Pyth Pro API and WebSocket streaming endpoints have been resolved. All systems are now operating normally.
The incident was caused by a major outage at TeraSwitch, a data center provider hosting part of our infrastructure. At the time, two of four NATS nodes were hosted in TeraSwitch facilities. When those nodes became unreachable, NATS lost quorum and dependent Router services became unavailable.
We had already been working to improve infrastructure redundancy and reduce provider concentration. This work is now being accelerated, including onboarding an additional infrastructure provider and distributing critical services across independent failure domains.
The goal is to ensure that a future outage affecting any single infrastructure provider does not result in a broader service disruption.
Investigating
We’re currently investigating an incident affecting Pyth Pro API and WebSocket streaming endpoints.
Current impact: Data is not currently available.
We’ll share more updates as soon as we have them.
