KnovvuCAChatIndexerCannotKeepUp

Meaning

This alert is triggered when chat conversations continue to arrive, but they are not being indexed at the same rate. Unlike calls, chat conversations do not go through an analysis step and are expected to be indexed directly. If the backlog between incoming and indexed chats increases continuously for 10 minutes, the system raises a warning-level alert.

Impact

If chat conversations are not indexed promptly:

• Chats may not appear in dashboards, reports, or search results.
• Supervisors and QA teams may lack visibility into recent chat interactions.
• Data pipelines that rely on indexed conversations may work with incomplete data.
• An increasing backlog can lead to delays and potential data processing issues.

Diagnosis

• Review dashboards showing chat ingestion and indexing trends over the past few hours.
• Verify that the indexing manager component (ca-index-manager) is creating and dispatching jobs for incoming chat conversations.
• Investigate the performance of the ca-indexer service, which performs the actual indexing. Check CPU, memory, disk throughput, and indexing latency.
• Inspect logs for errors, slowdowns, queue buildups, or configuration issues.
• Confirm that incoming chat data is well-formed and not triggering errors during indexing.
• Look for recent deployments, scaling events, or configuration changes that may have introduced bottlenecks.

Mitigation

• Scale out the ca-indexer component to increase indexing throughput. This service is typically designed for horizontal scaling under load.
• Restart or adjust the configuration of ca-indexer if it is underperforming or misbehaving.
• Ensure ca-index-manager is operational and not stalled in dispatching indexing jobs.
• Address any malformed or unusually large chat entries that could be blocking queues.
• Monitor the backlog closely and confirm that it begins to decrease after corrective action.
• Escalate to engineering if the issue is persistent or root cause cannot be identified quickly.