A model validation related process triggered repeated calls to an inefficient library function that exhausted memory in a subset of our US East infrastructure. A fix has been deployed that removes calls to the inefficient library function and we have validated it resolves the memory exhaustion.
Posted Apr 29, 2025 - 15:10 PDT
Update
We have identified the root cause of the issue and are implementing a fix.
Posted Apr 29, 2025 - 12:57 PDT
Monitoring
We have mitigated the issue and are monitoring the system.
Posted Apr 29, 2025 - 10:35 PDT
Investigating
Users are experiencing slow responses in the US region, our team is investigating.