LLM Element Overview

The LLM Element is a core component for building chatbots within Navigator. It enables running large language models (LLMs) privately and locally, supporting both resource-constrained and distributed computing environments.

This article provides a detailed overview of the LLM Element's configuration options, functionality, and best practices for setup.

Core Functionality

The LLM Element is designed for text generation and chat-based applications. When added to a flow, it automatically provides a live chat interface that allows users to interact with the selected model and receive contextually relevant, real-time responses.

Key features include:

• Real-time streaming responses with conversational context tracking
• Support for both pre-trained base models and custom-trained artifacts
• Automatic compute environment evaluation and resource filtering
• Integration with both Standard Chat and Retrieval-Augmented Generation (RAG) workflows

Navigator's LLM Element can automatically evaluate the available compute environment and filter out models that your device does not have sufficient resources to support, preventing configuration errors before they occur.

LLM Settings

Model Setup

This section allows you to select, configure, and manage the AI model used in your application. The model selection interface provides intelligent filtering and resource management to ensure compatibility with your hardware.

When hovering over your selected model, you'll see its memory requirements and whether it is webFrame supported.

Clicking on the selected model opens the Model Selector, where you can:

• Search for a specific model by name using the search functionality
• Use filters to browse available models efficiently. For example, the webFrame filter shows only models that support distributed execution, and the API Keys / No API Keys filters models based on whether an API key is required

• Combine multiple filters to refine your search and find the perfect model for your use case
   

  

Memory Requirements

This setting visually displays the device memory required to run the selected model. The interface provides real-time feedback about resource compatibility and requirements.

If the model exceeds your current device's capacity, you can add additional machines to distribute execution. Navigator will highlight models that cannot run on your available devices with their requirements shown in red, preventing configuration issues.

For guidance on connecting and managing devices, refer to the Devices Overview article.

Running On

The Running On dropdown specifies which connected device is executing the LLM Element. This setting allows you to control where your model runs within your cluster setup.

Once additional devices (aka nodes) have been successfully linked through the cluster configuration, they will appear here as selectable options. This enables you to:

• Distribute workloads across multiple machines for better performance
• Select specific devices optimized for your model's requirements
• Balance resource usage across your available hardware
• Ensure models run on devices with sufficient memory and processing power

For more information on clusters, visit our Clusters Overview article.

System Prompt

The System Prompt defines the base instructions that control how your chatbot behaves. This is the foundational prompt that sets the model's role, tone, and output constraints.

A well-structured system prompt should include:

• Context: Background information or purpose of the chatbot
• Task: What the chatbot should accomplish
• Tone and Format: Guidelines for response style and output structure
• Constraints: Any limitations or boundaries for responses

System Prompt Examples

Here are some practical examples of effective system prompts:

Example 1 - Customer Service Bot:

"You are a helpful customer service representative for a technology company. Respond professionally and courteously to customer inquiries. Always ask clarifying questions when needed and provide step-by-step solutions."

Example 2 - Creative Writing Assistant:

"You are a creative writing assistant that speaks like a pirate. Help users brainstorm story ideas, develop characters, and improve their writing while maintaining a playful, adventurous tone."

This would result in all responses adopting a playful, pirate-themed tone while still providing valuable writing assistance.

Example 3 - Technical Documentation Helper:

"You are a technical writing expert. Provide clear, concise explanations of complex topics. Use bullet points and numbered lists when appropriate. Always include practical examples and avoid jargon unless necessary."

Advanced Settings

The LLM Element includes several advanced configuration options accessible through the Advanced Settings tab:

• Hugging Face API Key: If your models require a Hugging Face API key, you'll add this here.
• Temperature: Controls the creativity or randomness of responses. Lower values (e.g. 0.1) produce more focused and predictable responses, while higher values (e.g. 0.9) generate more open-ended and varied outputs. Default: 0.7. Range: 0.01 – 1.0
• Limit Tokens: This toggle allows you to set limits on the length of model responses. 
• Limit Cluster Memory Usage: This toggle allows you to control the maximum amount of cluster resources used.

Integration and Workflow Types

Standard Chat Workflows

For standard chat applications, the LLM Element requires a connection to an API element to complete the flow. For the quickest use of the LLM Element, we recommend starting with the LLM Chatbot featured template. This has the LLM Element setup with the API element, and all you'd need to do is click Run!

RAG (Retrieval-Augmented Generation) Workflows

For RAG applications, the LLM Element receives input from the Prompt Templating element and sends output to the API element. This configuration enables document-based question answering and knowledge retrieval.

For a quick start guide on getting a RAG system up and running visit our Building Your First Custom LLM Chatbot article.

Troubleshooting

Common Issues and Solutions

Model won't run due to memory constraints:

• Check the memory requirements display in the model selector
• Consider using a smaller model or quantized version
• Add additional devices to your cluster for distributed execution
• Use webFrame-supported models for automatic resource optimization

API key errors for gated models:

• Ensure you have accepted the license agreement on Hugging Face
• Verify your API key is correctly configured
• Check that your account has access to the requested model

Slow response times:

• Reduce the Max Tokens setting to limit response length
• Consider using a smaller, more efficient model
• Enable distributed execution across multiple devices
• Adjust the Temperature setting for more focused responses

Best Practices

• System Prompt Design: Keep system prompts clear, specific, and focused on your use case
• Model Selection: Start with 7B parameter models for the best balance of performance and resource usage
• Resource Management: Monitor memory usage and consider distributed execution for larger models
• Testing: Use the Preview View to iterate quickly on chatbot configurations
• Documentation: Keep track of successful prompt configurations for future projects

The LLM Element provides the foundation for building sophisticated AI chatbots and assistants. Whether you're creating customer service bots, creative writing assistants, or technical documentation helpers, proper configuration of the system prompt and model selection will ensure optimal performance for your specific use case.