Unbabel - Degraded performance - Machine translation step – Incident details

This incident has been resolved. We no longer expect translation requests failing systematically as to compromise any operation.

We addressed the root causes and are currently monitoring the result. We'll provide an update once no more translations are detected as failing.

Although we have all the necessary resources focused on identifying the root cause and fixing this issue, we still observe degraded performance. We have automated mechanisms in place that are acting as a mitigation procedure allowing any failed request to be retried and ultimately delivered. Nonetheless, we understand this is not an optimal workaround so please accept again our sincere apologies.
We will be providing updates as soon as we hear again from our engineering team.

The previous mitigation actions we took allowed for temporary recovery but we've started to observe degraded performance again, unfortunately. Rest assured this is deserving our utmost attention and we will keep you posted on any updates. Thank you for bearing with us.

Our engineering team implemented a series of mitigation actions which have proved to stabilise performance and allow our MT flows to resume working as expected. We'll continue to monitor and investigate the root cause of the issue.

As soon as we have more insights, we will update this incident report. We apologise again for any inconvenience caused.