Skip to main content

User Feedback

User Feedback is the feedback responses obtained from end users. These responses are typically related to the quality of the output of your application. RAG Workbench allows you to incorporate user feedback into your trace via the LastMile Tracing SDK, providing valuable insights into how users perceive the performance your RAG system.

User feedback can be captured in various forms:

  • Feedback Score: Users can rate a specific span, such as a LLM-generated output or a retrieved document (e.g., 👍 / 👎 or on a scale 1-10)
  • Expected Value: Users can provide the expected or correct output for a span, which is stored in the expected field alongside input and output. This is useful for capturing corrections.
  • User Comment: Users can share additional context or qualitative feedback using a free-form text field.

See the User Feedback Tutorial for a full walkthrough on how to log user feedback.

Logging User Feedback​

User feedback is 1:1 with a specific trace. First, you need to instantiate a LastMile Tracer.

tracer: LastMileTracer = get_lastmile_tracer(
    tracer_name="my-tracer",
    project_name="Quick-Start-LM"
)

To log user feedback to your trace, use the log_feedback() method by specifying the trace_id, span_id and the user feedback. Here is an example below:

tracer.log_feedback(span_id=span_id, trace_id=trace_id, feedback='This was amazing! 👍')

Now your user feedback has been logged to your Trace!

Viewing User Feedback​

You can view your logs in the RAG Workbench UI.

Run the following command in your terminal to launch the UI:

rag-debug launch

Navigate to the url provided by the RAG Workbench (opens up your web browser). This will look like http://localhost:8080/

  1. Click the Traces Tab.
  2. Select your Project.
  3. Click on a Trace within your Project.
  4. View User Feedback for that Trace.

NOTE: You can create Test Sets by selecting traces that have specific user feedback associated with them. This feature allows you to easily experiment with and evaluate your model using data that has been labeled by real users. Please read about Test Sets for more information.

Example User Feedback for a Trace:

Screenshot 2024-06-13 at 3 32 01 PM

Coming Soon​

  • Using Real-time Feedback for Fine-Tuning

More Resources​

Here are other helpful guides related to logging user feedback with your trace: