For that, you need to evaluate your Response and Retrieval using Ragas or DeepEval frameworks.

I will cover these evaluation frameworks in this RAG series soon.