📄️ Tracing
Tracing is the foundation of evaluating, debugging, and optimizing your RAG application. RAG Workbench provides a unique capability to trace both your ingestion and query pipelines, offering a comprehensive view of your system. The two fundamental components in tracing are spans and traces.
📄️ Logging
Logging allows you to annotate a Trace with important information for troubleshooting and debugging. Your log statements are tied to a Trace which makes it easy to collect and analyze important information quickly and systematically. You can use logging to understand user behavior, streamline debugging, and identify areas of improvement for your RAG system.
📄️ Evaluation Runs
Evaluation Runs help measure an LLM application's performance by taking the user query, LLM response, and an evaluator function. An evaluation run can also take in ground truth and other metadata; these required fields are usually determined by the specific evaluator used. Here are a couple key terms:
📄️ Test Sets
Test Sets are collections of example data used to evaluate the performance of your RAG system. They provide a standardized set of inputs, LLM-generated resposnes, and, if available, expected outputs to assess how well your RAG system generates responses. Each example within a Test Set is called a Test Case.
📄️ Evaluation Metrics
LastMile Evaluators help you assess the quality and performance of your LLM application. Suppose you have a RAG application where you ingest a document and ask an LLM to do the following:
📄️ Prompt Iteration
Prompt Iteration is a popular way to optimize your RAG System. Refining your prompts can significantly improve your system's performance at low cost and effort. RAG Workbench has a specific tool designed for refining the prompts. It allows you to experiment with your prompt using relevant metadata (ex. retrieved context) and try different models as well.
📄️ User Feedback
User Feedback is the feedback responses obtained from end users. These responses are typically related to the quality of the output of your application. RAG Workbench allows you to incorporate user feedback into your trace via the LastMile Tracing SDK, providing valuable insights into how users perceive the performance your RAG system.