RAG Inference Pipeline

RAG (Retrieval-Augmented Generation) is a technique that allows AI models to generate responses based on specific document collections. This guide walks you through utilizing the RAG Inference Pipeline to begin creating a complete RAG system.

To use the RAG Inference Pipeline, you must first complete a successful RAG Document Ingestion Pipeline run. For detailed steps, please refer to the Ingestion Pipeline article.

RAG Inference Pipeline

The Inference Pipeline allows you to chat with your documents using the Vector Database created in the Ingestion Pipeline. It processes user queries and returns relevant information from your documents.

Step 1: Configuring the Embedding Element

1. Click on the RAG Inference Pipeline in the Featured Template section.
2. Click on the Embedding element to view the element's settings.
3. Toggle off the Is Ingestion setting. This will switch the Embedding element to inference mode.
4. This will bring up additional settings. Click on the Update button in the Trained Artifact Path setting and select the Run_[#]_[Date] folder that we created in the Ingestion Pipeline.

Your settings for the Embedding element should look similar to the following:

This setting tells Navigator that this Inference Pipeline should use the Vector Database created from the documents provided (during the Ingestion Pipeline) and allow you to query those documents through an LLM within Navigator.

Step 2: Configuring the Large Language Model Chat Element (Optional)

By default, the Large Language Model Chat element will use the Gemma 1.1 2 Bit model. However, you can change the model to better fit your flow's needs. For more information on the models in the Base Model Architecture dropdown, visit our Supported LLM Base Models.

For more information on the Large Language Model Chat Element's settings, visit the Large Language Model Chat section in our Element Registry.

Step 3: Run Your Flow

Configuring the Embedding element is the only required step to successfully run your Inference Pipeline. If you'd like to adjust the other elements in this flow, continue reading to the Additional Information section.

1. Click on the Run button in the upper right hand corner of your canvas
2. Once the run is complete, a chat bot will open up inside of Navigator.
3. To test whether your Inference Pipeline is working correctly, try asking a very specific question about your documents that you used in the Ingestion step.

You've now set up your RAG system! Large Language Models (LLMs) do not have inherent knowledge of your internal documentation, they can only reference the information explicitly provided to them. The RAG Pipeline Process enables you to supply your own documents as a local knowledge source, allowing the model to retrieve and generate contextually accurate responses based on that data. This process runs entirely on your local machine, ensuring your information remains secure, private, and fully under your control.

Additional Information

You do not need to adjust any of the API Element's settings. However, if you'd like to integrate your RAG system with other applications, you would do so by using the API Element.

Element Settings:

• API Key (Optional): You can provide an API key to help prevent unwanted access to the network. A default key is already included, so this is optional for most users.
• Endpoint Timeout (Seconds 0-100): timeout for SSE endpoint, default is 0 (no timeout).
• Maximum Concurrent Requests: Limits how many requests can be processed at the same time. Use this to control system load and performance.
• Maximum Queued Requests: Sets the number of requests that can wait in line when the system is busy. If the queue is full, new requests will be rejected with a 429 Too Many Requests error.

Next Steps

After completing both pipelines, you can:

• Test your RAG system by asking questions about your documents
• Fine-tune response quality by adjusting chunk sizes or retrieval settings in your Ingestion Pipeline.