What is the Faithfulness Score in AI Responses?

The Faithfulness Score helps measure how accurately an AI-generated response aligns with the specific information provided by your business. It focuses on factual consistency, ensuring that the AI’s answers reflect the data you’ve supplied without adding or changing key details.

What Does the Faithfulness Score Mean?

The Faithfulness Score, ranging from 0 to 1, shows how well the AI’s response matches the facts from your data:

• Score of 1: A fully accurate answer where all statements in the response can be verified by your data.
• Score of 0: An unfaithful answer where none of the information aligns with the provided context.
• Scores between 0 and 1: Partial accuracy, where some statements are supported by the data but others may not be.

A high Faithfulness Score indicates that every claim made in the AI’s answer can be inferred directly from the data source, while a lower score means the response may contain inaccuracies or assumptions that aren’t supported by the data.

How is the Faithfulness Score Calculated?

To calculate the Faithfulness Score, the system follows these steps:

1. Identify Claims in the Answer: The AI breaks down the response into individual claims or statements.
2. Verify Each Claim: Each claim is then checked against the data provided to see if it is fully supported by the context.
3. Calculate the Score: Based on the percentage of claims that align with the provided context, a Faithfulness Score between 0 and 1 is assigned.

Example

• Question: "What are the hours for customer service, and is there a weekend helpline?"

  • Context: "Our customer service team is available Monday through Friday, 9 AM to 6 PM. Weekend support is not available."

  • High Faithfulness Answer: "Customer service is available Monday to Friday, 9 AM to 6 PM, with no weekend support."

  • Low Faithfulness Answer: "Customer service is available daily, including weekends, from 9 AM to 6 PM."

In this example, the high-faithfulness answer matches the provided context exactly, while the low-faithfulness answer introduces incorrect information about weekend support.

Why is the Faithfulness Score Important?

The Faithfulness Score ensures that the AI is providing factually consistent information based on your data, which is crucial in areas where accuracy matters, such as customer support, technical help, healthcare, or legal services. A high score means the AI is sticking closely to your facts, reinforcing customer trust and minimizing misinformation.

A lower score can indicate that the AI is including creative or inaccurate details, which may not be ideal when customers require precise information.

How Can You Use the Faithfulness Score?

Monitoring the Faithfulness Score helps ensure that your AI is providing the most accurate responses possible. If low scores appear frequently, it may suggest a need to adjust or expand your data to better cover the types of questions customers are asking.

The Faithfulness Score is a valuable tool for tracking the reliability of the AI’s answers, ensuring that your customers receive factually correct and relevant information every time they interact with your AI. This focus on accuracy helps build trust and improves the overall quality of your customer support.