Getting Started LLM Feedback (Beta)

LLM Feedback (Beta)

The LLM Feedback feature enables users to provide direct feedback on AI model responses within Companion, with all feedback automatically recorded and accessible in Navigator. This powerful tool helps you understand how your deployed models are performing in real-world interactions, identify areas for improvement, and make data-driven decisions about model refinement and optimization. This feature is currently in Beta with more updates to come! 

Overview

LLM Feedback creates a seamless feedback loop between your users and your AI deployments. When users interact with your deployed models in Companion, they can provide feedback on individual responses, indicating whether the AI's output was helpful, accurate, or met their expectations. This feedback is automatically captured and stored in Navigator, giving you valuable insights into your model's performance across all user interactions.

How LLM Feedback Works

Providing Feedback in Companion

When users interact with a deployed model in Companion, they can provide feedback on any response generated by the AI. This feedback mechanism is integrated directly into the chat interface, making it easy and intuitive for users to share their thoughts on the quality and usefulness of each response.

  1. After receiving a response from the AI model, users will see feedback options available for that specific message
  2. Users can select from predefined feedback tags or provide additional context through optional text comments 
  3. Feedback is automatically submitted and recorded in Navigator without requiring any additional action from the user

Feedback is collected anonymously and associated with the specific Application and conversation thread, allowing you to track patterns across different user interactions while maintaining user privacy.

Recording Feedback in Navigator

Navigator automatically receives and stores all feedback submitted through Companion. This feedback is tied to each Application. The feedback data includes:

  • Query ID 
  • Timestamp of the feedback
  • Feedback provided
  • Prompt & Response 
  • Tags given during feedback 
  • Additional Comments 

This comprehensive data collection enables you to understand not just what feedback was given, but also the context in which it occurred, helping you identify patterns and areas for improvement.

Viewing Feedback in Navigator

Accessing the Feedback Dashboard

To view feedback collected from your deployments:

  1. Navigate to the Applications tab in Navigator
  2. Click on the Application you want to review feedback for
  3. Click the LLM Feedback tab


Feedback Types

The feedback system supports multiple types of feedback to capture different aspects of user satisfaction:

  • Good Response (Thumbs Up): Indicates the response was helpful, accurate, or met the user's needs
  • Bad Response (Thumbs Down): Highlights areas where the response fell short or was incorrect
  • Safety Issue (Flag): Flags any safety concerns regarding the responses there generated
     

Understanding the Feedback Display

The feedback dashboard provides several views and filters to help you analyze the collected data:

  • Feedback Score: Weighted score of all feedback your application has received to date 

Our feedback scores are calculated using a Bayesian binomial approach, which helps to stabilize scores against individual votes and ensures that new information is incorporated gradually. The score is initially based on a default value (alpha). As user feedback is received, the score is adjusted by adding the number of upvotes (numerator) to alpha, while the total number of votes (denominator) is also updated. 
As more feedback is received, the score adapts to reflect the changing ratio of upvotes to total votes, providing an accurate and nuanced representation of user opinion.
 

 

  • Filtered Views: Filter feedback by feedback type, date range, tags, or all conversations that have additional comments

  • Detailed View: Expand individual feedback entries to see the full conversation context and user comments by clicking on each response
     

Exporting and Using Feedback Data

Export Options

Navigator allows you to export feedback data for further analysis or integration with other tools:

  1. Navigate to the Feedback section of your application
  2. Use the filter options to select the feedback you want to export

  3. Click the Export button to download the feedback data

  4. Choose your preferred export format (CSV or JSONL)

Export Formats

Navigator supports two export formats to accommodate different workflows and analysis tools. Select the format that best fits your use case:

CSV Format

The CSV (Comma-Separated Values) format is ideal for spreadsheet applications and traditional data analysis tools:

  • Compatibility: Opens directly in Excel, Google Sheets, and other spreadsheet applications
  • Structure: Each row represents a single feedback entry with columns for prompt, response, feedback_type, and tags
  • Best For: Quick analysis, creating charts and visualizations, sharing with team members who prefer spreadsheet tools

JSONL Format

The JSONL (JSON Lines) format provides structured data that is well-suited for programmatic processing and machine learning workflows:

  • Compatibility: Works seamlessly with Python, data processing pipelines, and LLM fine-tuning tools
  • Structure: Each line contains a complete JSON object representing a single feedback entry, including question, answer, and sources (citations provided).
  • Best For: Automated processing, integration with training pipelines, creating fine-tuning datasets, and advanced data analysis scripts

Next Steps