Monitoring
The API is back to full operational capacity, we’ve added nodes to cover the ones that were unavailable.
Identified
We are serving capacity again now, and are rolling out more GPUs across DCs that have availability.
Identified
Multiple DCs that a third party relies on have needed to undergo emergency unplanned maintenance, we are working on bringing new capacity online as soon as possible, we expect to have some capacity available within around an hour.
We will start returning requests again once capacity becomes available.
Identified
We have started load shedding all requests to models whilst the infrastructure provider root causes the underlying problem.
As soon as they are back online, we will start receiving requests again normally.
Identified
Our primary inference infrastructure provider is down across all data centers. This is causing all requests to fail, we are working on load shedding them with 429s in the meantime whilst working with the provider to resolve the issue downstream.
We will provide an update shortly.