Error occurred on basic LLMContextRecall usage, "reference Input should be a valid string"

[bode135]

input:

from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
from langchain_openai.chat_models import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas import evaluate
from datasets import Dataset
import pandas as pd

# Example dataset
data = {
    "question": ["What is the capital of France?"],
    "answer": ["The capital of France is Paris."],
    # "retrieved_contexts": [["Paris is the capital city of France."]],
    "retrieved_contexts": [["Paris is the capital city of France."]],
    "ground_truth": [["Paris is the capital city of France."]]
}
 
df = pd.DataFrame(data)
eval_dataset = Dataset.from_dict(df)


model_name = "gpt-4o"
# model_name = "deepseek-v2"
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model=model_name))
metrics = [LLMContextRecall(), FactualCorrectness(), Faithfulness()]
results = evaluate(dataset=eval_dataset, metrics=metrics, llm=evaluator_llm)

print(results)

output:

File "/opt/homebrew/anaconda3/envs/my_rag_checker/lib/python3.10/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for SingleTurnSample
reference
  Input should be a valid string [type=string_type, input_value=['Paris is the capital city of France.'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/string_type

version info

• system: mac air 2021 m1

>>>  python -V
Python 3.10.15

>>> pip list | grep ragas
ragas                     0.2.2

[bode135]

Actually , change "ground_truth": [["ground_truth --- Paris is the capital city of France."]] to "ground_truth": ["ground_truth --- Paris is the capital city of France."] will solve the problem!
over.