Success Measurement

KPI measurement evaluates the performance of a Virtual Agent (VA) and is measured using real customer interactions after the project goes live.

Key Principles for KPI Measurement

1. Data Selection & Distribution

To ensure reliable KPI measurement and accurate data distribution:

• Customer interactions should be selected randomly from various time periods, such as weekdays, weekends, mornings, and evenings.
• No filters should be applied based on attributes like customer ID, emotional state, or channel, ensuring true data randomness.
• Both spoken and written customer requests should be evaluated, depending on the project scope.

2. Success Criteria for KPI Measurement

• An interaction is marked as "successful (1)" if the Virtual Agent provides the correct response to the customer’s request. If the response is incorrect, the interaction is marked as "unsuccessful (0)".
• If an interaction involves multiple requests, each request is evaluated separately.
  • Example: If a customer asks about their account balance and then inquires about the status of a card application, these are treated as two separate requests. Each request is evaluated independently for success.
• For voice interactions, exact word matching is not required; the interaction is considered successful if the customer receives the correct response.
• The voice input used in the SR process must be clear, intelligible, and free from background noise or interruptions. Also, lossless encoding and compression methods should be utilized.

3. KPI Measurement

The KPI accuracy is calculated as the ratio of successful responses to the total number of customer requests evaluated.

After reviewing the recommended number of interactions:
• Successful responses are labeled as 1, and unsuccessful responses as 0.
• KPI is calculated as the ratio of successful responses to the total number of customer requests reviewed.

Example:

• If 100 interactions are analyzed and a total of 150 customer requests are identified, and 135 of these requests are successful, the KPI is calculated as:

  (135 ÷ 150)*100 = 90%