KnovvuCACallAnalysisCannotKeepUp

Meaning

This alert is triggered when new call conversations continue to arrive, but they are not being analyzed at the same rate. This suggests that the system is falling behind in processing the incoming data. If this backlog continues to grow over a 10-minute period, the alert fires at a warning level.

Impact

If analysis fails to keep up with the rate of incoming calls:

• Reporting and dashboards may reflect incomplete or stale data.
• Real-time alerts and insights may be delayed or unavailable.
• Data pipelines depending on analyzed conversations may experience cascading issues.
• A growing backlog could exhaust system capacity or lead to dropped data.

Diagnosis

• Review dashboards comparing the rate of received and analyzed call conversations.
• Check the health and performance of the analysis pipeline (ca-analysis), including processing throughput and job queues.
• Focus on the core-sr service, which performs speech-to-text transcription and is typically the heaviest load component. Look for CPU, memory, and GPU (if applicable) utilization, as well as slow or failing transcription jobs.
• Inspect logs for signs of degraded performance, bottlenecks, or failed processing attempts.
• Validate that the ingestion service (ca-external-api) is delivering data consistently and without interruption.
• Review recent deployment or infrastructure changes that may have impacted the system's processing speed or concurrency.

Mitigation

• Scale out the core-sr service if it is identified as the processing bottleneck. Ensure it has sufficient compute resources to meet current conversation volume.
• Restart or reconfigure ca-analysis if it appears to be stalling or failing to dispatch jobs properly.
• Monitor the load queue and introduce throttling or prioritization if the system is overwhelmed.
• Optimize the configuration of speech-to-text jobs for performance, especially under high concurrency.
• If the lag is expected due to a known spike in conversation volume (e.g., campaign day), continue to monitor the backlog and confirm it eventually drains.
• Escalate to engineering if the backlog grows persistently or the cause cannot be resolved operationally.