Ollama

info

This is only helpful for self-hosted users. If you're using Khoj Cloud, you can use our first-party supported models.

info

Khoj can directly run local LLMs available on HuggingFace in GGUF format. The integration with Ollama is useful to run Khoj on Docker and have the chat models use your GPU or to try new models via CLI.

Ollama allows you to run many popular open-source LLMs locally from your terminal. For folks comfortable with the terminal, Ollama's terminal based flows can ease setup and management of chat models.

Ollama exposes a local OpenAI API compatible server. This makes it possible to use chat models from Ollama with Khoj.

Setup

info

Restart your Khoj server after first run or update to the settings below to ensure all settings are applied correctly.

First Run

Docker

1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,

   ollama pull llama3.1
3. Uncomment OPENAI_API_BASE environment variable in your downloaded Khoj docker-compose.yml
4. Start Khoj docker for the first time to automatically integrate and load models from the Ollama running on your host machine

   # run below command in the directory where you downloaded the Khoj docker-compose.yml
   docker-compose up

Pip

1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,

   ollama pull llama3.1
3. Set OPENAI_API_BASE environment variable to http://localhost:11434/v1/ in your shell before starting Khoj for the first time

   export OPENAI_API_BASE="http://localhost:11434/v1/"
   khoj --anonymous-mode

Update

1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,

   ollama pull llama3.1
3. Create a new AI Model API on your Khoj admin panel
   • Name: ollama
   • Api Key: any string
   • Api Base Url: http://localhost:11434/v1/ (default for Ollama)
4. Create a new Chat Model on your Khoj admin panel.
   • Name: llama3.1 (replace with the name of your local model)
   • Model Type: Openai
   • Openai Config: <the ollama config you created in step 3>
   • Max prompt size: 20000 (replace with the max prompt size of your model)
5. Go to your config and select the model you just created in the chat model dropdown.

If you want to add additional models running on Ollama, repeat step 4 for each model.

That's it! You should now be able to chat with your Ollama model from Khoj.