Google Vertex AI

info

This is only helpful for self-hosted users. If you're using Khoj Cloud, you can directly use any of the pre-configured AI models.

Khoj can use Google's Gemini and Anthropic's Claude family of AI models from Vertex AI on Google Cloud. Explore Anthropic and Gemini AI models available on Vertex AI's Model Garden.

Setup

1. Follow these instructions to use models on GCP Vertex AI.
2. Create Service Account credentials.
   • Download the credentials keyfile in json format.
   • Base64 encode the credentials json keyfile. For example by running the following command from your terminal: base64 -i <service_account_credentials_keyfile.json>
3. Create a new API Model API on your Khoj admin panel.
   • Name: Google Vertex (or whatever friendly name you prefer).
   • Api Key: base64 encoded json keyfile from step 2.
   • Api Base Url: https://{MODEL_GCP_REGION}-aiplatform.googleapis.com/v1/projects/{YOUR_GCP_PROJECT_ID}
     • MODEL_GCP_REGION: A region the AI model is available in. For example us-east5 works for Claude.
     • YOUR_GCP_PROJECT_ID: Get your project id from the Google cloud dashboard
4. Create a new Chat Model on your Khoj admin panel.
   • Name: claude-3-7-sonnet@20250219. Any Claude or Gemini model on Vertex's Model Garden should work.
   • Model Type: Anthropic or Google
   • Ai Model API: the Google Vertex Ai Model API you created in step 3
   • Max prompt size: 60000 (replace with the max prompt size of your model)
   • Tokenizer: Do not set
5. Select the chat model on your settings page and start a conversation.