Success Measurement
- 16 Jun 2023
- 3 Minutes to read
- Contributors
- Print
- PDF
Success Measurement
- Updated on 16 Jun 2023
- 3 Minutes to read
- Contributors
- Print
- PDF
Article summary
Did you find this summary helpful?
Thank you for your feedback
Accuracy of a Biometric System
To evaluate the accuracy and performance of a biometric system, three key metrics are commonly utilized: the False Acceptance Rate (FAR), the False Rejection Rate (FRR), and the Equal Error Rate (EER).
The False Acceptance Rate (FAR) measures the rate at which the biometric system incorrectly accepts impostors or unauthorized users. It quantifies the system's vulnerability to impostor attacks and represents the security level of the system. A lower FAR indicates a more secure system.
The False Rejection Rate (FRR) measures the rate at which the biometric system incorrectly rejects genuine users. It reflects the level of convenience for the end-user. A higher FRR implies more instances where genuine users are mistakenly denied access, potentially causing inconvenience and frustration.
The Equal Error Rate (EER) represents the point where the FAR and FRR intersect. It serves as a threshold that balances the system's ability to distinguish impostors from genuine users. At the EER, the system makes an equal number of false accepts and false rejects. It provides a single measure to compare the performance of the biometric system. A lower EER indicates a more accurate and convenient system.
Note:
The optimal values for FAR, FRR, and EER may vary depending on the specific use case and the desired trade-off between security and convenience.
Measure the Success
Step 1 - Random Selection: Randomly select a certain number of authentication/identification operations (N) from the customer environment at different time intervals.
Sample Size:
To ensure statistically objective and accurate analysis, it is recommended to examine a sample size of at least 1000 operations (N = 1000).
Step 2 - Recording Actual Results: Note the users and their actual authentication results obtained from the biometric system.
Step 3 - User Verification: Listen to the enrollment records of each user and meticulously compare them with their corresponding authentication records. Note the expected results of the biometric authentication operations for each user.
Notes:
- At least 3 different observers examine the records independently without knowledge of the system outputs or each other's assessments.
- The observers label the authentication results as the same person, different person, or indecisive.
- Only cases where the observers unanimously agree on whether the records belong to the same person or different persons are considered.
Step 4 - Comparison of Expected and Actual Results: Compare the expected results of the biometric authentication operations with the actual results obtained from the system. Based on this comparison, each operation is labeled as Successful, False Accept (FA), or False Reject (FR).
Exclusion of Ambiguous Cases and Evaluation of Error Rates
- Cases where speakers cannot be identified with human ear as the same or different person by Sestek’s examination will be excluded from the evaluation.
- Error rates will be evaluated based on faulty cases where Sestek and customer agree.
Step 5 - Calculation of FAR and FRR: Calculate the FAR and FRR over the N operations. These ratios provide insights into the system's accuracy and reliability.
Calculations:
- FAR = (Number of False Accept (FA) operations / N) * 100
- FRR = (Number of False Reject (FR) operations / N) * 100
The table below is provided for recording observations:
| User | Observers | Expected Results | Actual Results | Result |
|---|---|---|---|---|
| Test User-1 | Same person | Authentication accepted | Authentication accepted | Successful |
| Test User-2 | Different person | Authentication rejected | Authentication rejected | Successful |
| Test User-3 | Different person | Authentication rejected | Authentication accepted | False Accept |
| Test User-4 | Same person | Authentication accepted | Authentication rejected | False Reject |
Test User: The identifier of the user being tested.
Observers: The observers' assessment of the user's authentication records (same person, different person, indecisive).
Expected Results: The anticipated outcome of the authentication operation.
Actual Results: The observed outcome of the authentication operation.
Result: The final classification of the authentication operation (Successful, False Accept, or False Reject).
Exclusion of Non-Standard Audio Recordings:
In order to maintain the integrity of the assessment, cases involving non-standard audio recordings for enrollment or authentication are not included. Records to be considered non-standard:
- Recordings where the operator's voice is fully or partially heard in the customer channel.
- Conversations that involve speakers other than the main customer.
- Conversations with significant interruptions caused by network issues.
- Speech recordings with severely reduced quality due to environmental factors such as excessive noise.
- Music or tone signals detected as speech by the system, exceeding 10% of the recording duration.
Was this article helpful?