Delay in channels
Incident Report for Aivo
Postmortem

On 12/02, at 5:00 pm and until 7:15 pm (GMT -3), we experienced a degradation in the quality of our service which caused delays in the delivery of messages in asynchronous channels. This was generated, among other causes, due to a drop in the performance of Generative AI APIs, which caused a queue of waiting messages to be created, slowing down message delivery. This Generative AI incident directly and significantly affected the performance of our APIs. To mitigate the dependency on Generative AI APIs a hotfix will be deployed so that our system will fall back to the internal engine to deliver messages on time.

Posted Dec 03, 2024 - 15:18 GMT-03:00

Resolved
This incident has been resolved.
Posted Dec 02, 2024 - 20:15 GMT-03:00
Monitoring
The incident has been stabilized. We are currently monitoring all our services. However, we are still experiencing some delays in message delivery due to the reprocessing of all data.
Posted Dec 02, 2024 - 19:11 GMT-03:00
Investigating
We are experiencing a delay in channel response since 17:00 HRS (GMT -3).
The technical team is working on implementing the necessary measures for its solution.
Posted Dec 02, 2024 - 17:58 GMT-03:00
This incident affected: AgentBot (Chat Window) and Channels (Whatsapp, Facebook, Instagram).