Language Models and Chatbots LLM Dataset Generation

LLM Dataset Generation

The LLM Dataset Generator is the essential first step in creating a custom language model. This tool prepares the training data needed to build either a local expert model or a custom model for use with the Document QnA element.

Gather all relevant documents you want your model to learn from.

Documents must be in one of these supported formats (PDF, text, or DOCX) and collected in a single folder.

Creating Your Dataset

Step 1: Set Up Your Canvas

  1. Create a new Canvas
  2. Open the Elements Drawer
  3. Drag the LLM Dataset Generator element onto your Canvas

Step 2: Configure the Generator

Open the element settings by clicking the ... button, then configure:

  • Topic: Enter a descriptive name for your dataset
  • References folder path: Use the Select Directory button to choose the folder containing your source documents
  • Output folder path: Use the Select Directory button to choose where you want to save the generated dataset
  • Dataset size: Specify the number of topics for your dataset

    Recommendation: Start with 5 topics for initial testing. This generates results quickly but produces a less accurate model than larger datasets. Increase this number for more comprehensive training.

    Larger dataset sizes produce more accurate models but take longer to generate. Large datasets can take several hours to complete.

  • API Keys: Enter your Groq, GPT, Claude, or Gemini API keys

    If using Groq, a single API key is sufficient. If using other providers (OpenAI, Claude, or Gemini), you must provide at least two different API keys to enable all evaluation benchmarks.

    Need a free API key? Get one from Groq:

    Groq API Key

Step 3: Generate Your Dataset

  1. Click the Run button to start the generation process

    Dependencies will be installed the first time this flow runs, which may take some time to complete

  2. A folder named dataset_[your_topic_name]_[timestamp] will be created in your specified output location

Next Steps

This generated dataset folder will be used in the next phase of creating your custom LLM. You'll connect it to the Dataset Folder Path in the LLM Trainer Element during the training step.