Ollama
This is only helpful for self-hosted users. If you're using Khoj Cloud, you can use our first-party supported models.
Khoj can directly run local LLMs available on HuggingFace in GGUF format. The integration with Ollama is useful to run Khoj on Docker and have the chat models use your GPU or to try new models via CLI.
Ollama allows you to run many popular open-source LLMs locally from your terminal. For folks comfortable with the terminal, Ollama's terminal based flows can ease setup and management of chat models.
Ollama exposes a local OpenAI API compatible server. This makes it possible to use chat models from Ollama with Khoj.
Setup
Restart your Khoj server after first run or update to the settings below to ensure all settings are applied correctly.
- First Run
- Update
- Docker
- Pip
- Setup Ollama: https://ollama.com/
- Download your preferred chat model with Ollama. For example,
ollama pull llama3.1
- Uncomment
OPENAI_API_BASE
environment variable in your downloaded Khoj docker-compose.yml - Start Khoj docker for the first time to automatically integrate and load models from the Ollama running on your host machine
# run below command in the directory where you downloaded the Khoj docker-compose.yml
docker-compose up
- Setup Ollama: https://ollama.com/
- Download your preferred chat model with Ollama. For example,
ollama pull llama3.1
- Set
OPENAI_API_BASE
environment variable tohttp://localhost:11434/v1/
in your shell before starting Khoj for the first timeexport OPENAI_API_BASE="http://localhost:11434/v1/"
khoj --anonymous-mode
- Setup Ollama: https://ollama.com/
- Download your preferred chat model with Ollama. For example,
ollama pull llama3.1
- Create a new OpenAI Processor Conversation Config on your Khoj admin panel
- Name:
ollama
- Api Key:
any string
- Api Base Url:
http://localhost:11434/v1/
(default for Ollama)
- Name:
- Create a new Chat Model Option on your Khoj admin panel.
- Name:
llama3.1
(replace with the name of your local model) - Model Type:
Openai
- Openai Config:
<the ollama config you created in step 3>
- Max prompt size:
20000
(replace with the max prompt size of your model)
- Name:
- Go to your config and select the model you just created in the chat model dropdown.
If you want to add additional models running on Ollama, repeat step 4 for each model.
That's it! You should now be able to chat with your Ollama model from Khoj.