Success Measurement

Recognition rate, speaker separation and automated evaluation accuracy are measurable values that demonstrate how effectively CA is achieving key business objectives.
1. Recognition rate: Recognition rate is measured and confirmed by Sestek Consultancy Team and customer. Recognition rate is measured by selecting 10 calls from different agents over 180 seconds on various days.
Recognition rate is measured by comparing the recognition model results and manual transcription outputs. The system interface can be used to do the comparison. The measurement method is as follows:

Example 1:
Uttered sentence: “I want to cancel my credit card”
Recognized sentence: “I want to pencil my credit card”
Result: the word “cancel” is recognized as “pencil”. 6 out of 7 words are recognized correctly, therefore the recognition rate is 86%.

Example 2:
Uttered sentence: “I want to cancel my credit card”
Recognized sentence: “I want to cancel”
Result: Although the words “my”, “credit” and “card” are clearly pronounced by the customer representative, they are not recognized. 4 out of 7 words are recognized correctly, therefore the recognition rate is 57%.

2. Speaker separation: A one minute mono call channel conversation is recorded. And a csv file including the information where the agent and the customer speaks is created by Sestek Consultancy Team and customer. The created csv file is imported to the /speaker/diarization/der endpoint of the diarization service which basicaly compares the result with the output given by the team and check for the confusion error.

3. Automated evaluations accuracy: The compatibility of scripts in automatic evaluation forms is created by Sestek Consultancy Team.

• Compatibility of greeting scenario (e.g., hello, good afternoon, good evening, good morning...)
• Compatibility of closing scenario (e.g., good afternoon, good evening, good night, goodbye...)
• Forbidden words (e.g., so, excuse me, pardon...)

To measure the accuracy of automated evaluations, results from different users and various days are considered. The scenario compatibility accuracy can be reviewed by filtering the relevant query on the page of the conversations.