Quick Start Building Your First Custom LLM Chatbot

Building Your First Custom LLM Chatbot

This comprehensive step-by-step guide teaches developers and technical professionals how to build Retrieval-Augmented Generation (RAG) systems that create AI-powered chatbots capable of answering questions about specific documents like enterprise knowledge bases, customer support materials, legal documents, or technical documentation. 

The guide walks users through two main steps: setting up a document ingestion pipeline to process and vectorize documents, followed by creating an inference pipeline to chat with those documents using an LLM, making it ideal for implementing secure, private AI solutions for document-based question answering without sending data to the cloud.

Use Cases

  • Enterprise Knowledge Base → Company documentation and policy Q&A
  • Customer Support → Automated responses from support materials
  • Legal Document Analysis → Contract and compliance document search
  • Technical Documentation → API docs, manuals, and troubleshooting guides
  • Research Assistant → Academic paper and research document querying
  • Medical Information → Healthcare documentation and procedure guides
  • Educational Content → Textbook and curriculum material assistance
  • Compliance Systems → Regulatory and standards documentation
  • Code Documentation → Software documentation and codebase understanding
  • Training Materials → Employee handbook and training resource systems

Quick Start in Two Steps
 

Step 1: RAG Document Ingestion Pipeline

The Ingestion Pipeline processes your documents and transforms them into a format that AI models can understand, search, and interact with. The output is a Vector Database that you'll use when chatting with your documents in the Inference Flow (Step 2).

Need a text-only PDF to test with? Download the PDF below!  

Help Docs PDF

  1. Download the text-only files you will be using for the ingestion pipeline and put those in the same folder (you can name this folder whatever you would like). 

    See the Supported Formats below along with the Document Structure guidelines. Your file structure should look similar to this: 

     

     

  2. Open Navigator and on the Home tab → Featured Templates → Select RAG Document Ingestion Pipeline → Open 

  3. Click on the OCR element - (Optical Character Recognition) - to open that element's settings → click Select Directory under Data Path. This is where we will select the Folder we created in Step 1.

  4. Next click the Select Directory under Output Path. In the example below we are just using the same folder we created in Step 1. However, you could create a new folder for output data if you would like. 

    My settings for the OCR element now look like this: 

  5. Click on the Embedding element to open that elements settings and make sure that Is Ingestion is enabled here: 

  6. For now, we won't modify anything else within the Chunking, Embedding, or Vector Indexing element settings - but more information on what those elements do in our Element Registry.
  7. Click Run in the top right.

  8. Once this finishes we should now see a file within the location we selected in Step 4 with the title format of ExecutionName_[#]_[Date] which contains the metadata and Vector Database files. 

In the next stage we will create a pipeline that will reference the metadata and vector DB exported from this Document Ingestion Pipeline which should allow us to use a LLM chatbot to ask questions about the documents we provided!
 


Step 2: RAG Inference Pipeline

The Inference Pipeline allows you to chat with your documents using the Vector Database created in the Ingestion Pipeline (Step 1). It processes user queries and returns relevant information from your documents.

  1. Click Create in the top left → Featured Templates → RAG Inference Pipeline → Open 

  2. Click on the Embedding element → disable the Is Ingestion setting → Click Update under Trained Artifact Path → Select the Run_[#]_[Date] folder that we created in Step 1 → Open 

    In simpler terms - what we are telling Navigator in this Inference Pipeline is - use the Vector Database that we created from the documents provided (this was created from the Ingestion Pipeline in Step 1) and allow me to ask an LLM inside of Navigator about those documents.

  3. Click on the Large Language Model Chat element and select a Base Model Architecture - I'm going to simply use Gemma 1.1 2 Bit.

    Unsure what an LLM (Large Language Model) is? Check out our Terminology page or see our Supported LLM Base Models. 

    If we needed to provide more guidance to this LLM - we can click the Show Advanced Settings dropdown → provide a more detailed Model System Detail prompt - but the provided settings should do just fine.

  4. Click Run in the top right

  5. Once this completes you should see a chat bot open up inside of Navigator. To test if our Inference Pipeline is working - let's ask a very specific question about our document for example: 
     

    "What are the suggested percentages for organizing data for object detection?"
     

  6. As we can see in the screenshot below displaying the actual document we provided and what the LLM Chat produced - this RAG Inference Pipeline is accurately referencing our documents! 

The bigger picture here is that these LLM's (Large Language Models) don't know anything about our documentation unless we provide that information to them. So with this RAG Inference Pipeline - we can now input our documents and easily ask questions about these documents, locally from our machine without providing any information to the cloud. Safe, secure, and private AI that you control!

Document Structure

Your ingestion pipeline processes various formats:

documents/
├── pdfs/
│   ├── manual1.pdf
│   ├── guide2.pdf
│   └── ...
├── text_files/
│   ├── doc1.txt
│   ├── readme.md
│   └── ...

Supported Formats:

  • PDF Files → Text extraction with structure preservation
  • Microsoft Word → .docx
  • Text Files → .md, .csv, .json formats
  • Web Content → HTML and web page processing

Next Steps