1. What is Ollama & Why Local AI

Ollama lets you run large language models locally. No API costs, no data leaving your machine, full privacy and data sovereignty. Uses llama.cpp as backend and GGUF model format.

Why Local AI?

Privacy — Data never leaves your machine
Speed — No network latency for inference
No API costs — Run unlimited queries
Offline — Works without internet
Data sovereignty — Comply with regulations
Customizable — Fine-tune and create custom models

Installation

# Linux / macOS
curl -fsSL https://ollama.com/install.sh | sh

# macOS (Homebrew)
brew install ollama

# Windows
# Download from https://ollama.com/download

# Docker
docker run -d -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama \
  ollama/ollama

Architecture

Ollama wraps llama.cpp with a REST API server. Models are stored in GGUF format (quantized weights). The model library at ollama.com/library hosts curated model manifests that pull the right GGUF files.

2. CLI Essentials

# Start the server (runs in background on install)
ollama serve

# Run a model (pulls if not present)
ollama run llama3.1
ollama run llama3.1:70b

# Pull a model without running
ollama pull mistral
ollama pull llama3.1:8b-instruct-q4_K_M

# List downloaded models
ollama list

# Show model details (parameters, template, license)
ollama show llama3.1

# Copy a model (create alias)
ollama cp llama3.1 my-llama

# Remove a model
ollama rm mistral

# Create custom model from Modelfile
ollama create my-model -f Modelfile

# List running models
ollama ps

Model Tags

:latest — Default tag
:7b — 7 billion parameters
:70b-q4_K_M — Size + quantization
:8b-instruct-fp16 — Full precision

Interactive vs Pipe Mode

# Interactive chat
ollama run llama3.1

# Pipe mode (single prompt)
echo "Explain Docker" | ollama run llama3.1

# With system prompt
ollama run llama3.1 "Translate to German: Hello"

# Multiline input
ollama run llama3.1 <<EOF
Summarize this text:
$(cat article.txt)
EOF

3. Model Library & Selection Guide

Model	Sizes	Best For	VRAM (Q4)
llama3.1	8B / 70B / 405B	General purpose, multilingual	~5 / ~40 / ~240 GB
codellama	7B / 13B / 34B	Code generation & completion	~4 / ~8 / ~20 GB
mistral	7B	Fast general purpose	~4 GB
mixtral	8x7B / 8x22B	MoE, high quality	~26 / ~80 GB
phi-3	3.8B / 14B	Small & efficient	~2 / ~8 GB
gemma2	2B / 9B / 27B	Google, multilingual	~2 / ~6 / ~16 GB
qwen2.5	0.5B–72B	Multilingual, coding	~0.5–~42 GB
deepseek-coder	1.3B / 6.7B / 33B	Code specialized	~1 / ~4 / ~20 GB
llava	7B / 13B / 34B	Vision + text (multimodal)	~5 / ~8 / ~20 GB
nomic-embed-text	137M	Text embeddings	~0.3 GB

Quantization Levels

Level	Bits/Weight	Quality	Use Case
Q4_K_M	~4.8	★★★☆	Best balance of size vs quality (recommended)
Q5_K_M	~5.7	★★★★	Higher quality, moderate size increase
Q8_0	~8.0	★★★★★	Near-lossless, 2x size of Q4
F16	16.0	★★★★★	Full precision, largest, research use

4. REST API

Generate (completion)

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain Kubernetes in one paragraph",
  "stream": false
}'

Chat (multi-turn conversation)

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Docker?"}
  ],
  "stream": false
}'

JSON Mode

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "List 3 colors as JSON"}],
  "format": "json",
  "stream": false
}'

Embeddings

curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "Ollama is a local AI tool"
}'

Management Endpoints

# List models
curl http://localhost:11434/api/tags

# Show model info
curl http://localhost:11434/api/show -d '{"name": "llama3.1"}'

# Pull a model
curl http://localhost:11434/api/pull -d '{"name": "mistral", "stream": false}'

# Copy a model
curl http://localhost:11434/api/copy -d '{"source": "llama3.1", "destination": "my-llama"}'

# Delete a model
curl -X DELETE http://localhost:11434/api/delete -d '{"name": "my-llama"}'

# List running models
curl http://localhost:11434/api/ps

# Create from Modelfile
curl http://localhost:11434/api/create -d '{
  "name": "my-model",
  "modelfile": "FROM llama3.1\nSYSTEM You are a pirate."
}'

5. Modelfile — Custom Models

Modelfile Instructions

Instruction	Description
FROM	Base model (required) — e.g. FROM llama3.1
PARAMETER	Set model parameters (temperature, top_p, etc.)
SYSTEM	Set the system prompt
TEMPLATE	Override the prompt template (Go template syntax)
ADAPTER	Apply a LoRA or QLoRA adapter
LICENSE	Specify a license for the model

Coding Assistant

FROM codellama:13b
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM """
You are an expert software engineer. Write clean, well-documented code.
Always include error handling. Prefer TypeScript for web projects.
When asked to review code, check for bugs, security issues, and performance.
"""

Swiss German Translator

FROM llama3.1:8b
PARAMETER temperature 0.3
PARAMETER repeat_penalty 1.1
SYSTEM """
You are a Swiss German (Züridütsch) translator. Translate any input
text to Swiss German dialect. Keep the meaning and tone intact.
If the input is already in Swiss German, translate it to Standard German.
Always respond only with the translation, no explanations.
"""

JSON-Only Responder

FROM llama3.1
PARAMETER temperature 0.1
PARAMETER top_k 10
SYSTEM """
You are a JSON API. You ONLY respond with valid JSON objects.
Never include explanations, markdown, or any text outside the JSON.
Every response must be a single valid JSON object.
"""

Build & Run

# Create the model
ollama create my-coder -f Modelfile

# Run it
ollama run my-coder "Write a Python FastAPI endpoint"

# Verify parameters
ollama show my-coder

6. GPU & Performance Tuning

GPU Setup by Platform

# NVIDIA (CUDA) — install drivers + toolkit
sudo apt install nvidia-driver-535
sudo apt install nvidia-cuda-toolkit
nvidia-smi  # verify

# AMD (ROCm) — Linux only
sudo apt install rocm-libs
rocm-smi    # verify

# Apple Metal — automatic on macOS
# No extra setup needed, uses Metal by default

Environment Variables

# Number of GPU layers to offload
OLLAMA_NUM_GPU=999

# Max models loaded simultaneously
OLLAMA_MAX_LOADED_MODELS=2

# Parallel request handling
OLLAMA_NUM_PARALLEL=4

# Enable flash attention (faster)
OLLAMA_FLASH_ATTENTION=1

# Bind to all interfaces
OLLAMA_HOST=0.0.0.0:11434

# Custom model storage
OLLAMA_MODELS=/data/ollama/models

VRAM Estimation Formula

VRAM ≈ (Parameters × Bits_per_weight / 8) + (num_ctx × 0.5 MB per 1K tokens)

Example: 7B model Q4_K_M (~4.8 bits) with 4096 ctx ≈ (7B × 4.8 / 8) + (4 × 0.5) ≈ 4.2 + 2.0 ≈ 6.2 GB

Benchmark: Approximate Tokens/sec

Model	RTX 3090 (24GB)	RTX 4090 (24GB)	M2 Max (32GB)
7B Q4	~80 t/s	~120 t/s	~50 t/s
13B Q4	~45 t/s	~70 t/s	~30 t/s
70B Q4	~10 t/s (offload)	~15 t/s (offload)	~12 t/s

7. Python Integration

Official Python Library

pip install ollama

import ollama

# Simple chat
response = ollama.chat(model='llama3.1', messages=[
    {'role': 'user', 'content': 'What is Python?'}
])
print(response['message']['content'])

# Generate (completion)
response = ollama.generate(model='llama3.1',
    prompt='Write a haiku about coding')
print(response['response'])

# Embeddings
response = ollama.embeddings(model='nomic-embed-text',
    prompt='Machine learning is fascinating')
print(len(response['embedding']))  # 768 dimensions

Streaming

import ollama

# Streaming chat
stream = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Tell me a story'}],
    stream=True
)
for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

Async Client

import asyncio
import ollama

async def main():
    client = ollama.AsyncClient()
    response = await client.chat(model='llama3.1', messages=[
        {'role': 'user', 'content': 'Hello!'}
    ])
    print(response['message']['content'])

asyncio.run(main())

LangChain + Ollama

pip install langchain-ollama

from langchain_ollama import ChatOllama, OllamaEmbeddings

# Chat model
llm = ChatOllama(model="llama3.1", temperature=0.7)
response = llm.invoke("Explain microservices")
print(response.content)

# Embeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectors = embeddings.embed_documents(["Hello", "World"])
print(f"Dimensions: {len(vectors[0])}")

RAG Example with Ollama

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA

# 1. Load & chunk documents
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50)
docs = text_splitter.create_documents(["Your document text here..."])

# 2. Create vector store with Ollama embeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(docs, embeddings)

# 3. Create RAG chain
llm = ChatOllama(model="llama3.1")
qa = RetrievalQA.from_chain_type(
    llm=llm, retriever=vectorstore.as_retriever())

# 4. Query
result = qa.invoke("What does the document say?")
print(result["result"])

8. JavaScript/TypeScript Integration

Official npm Package

npm install ollama

import ollama from 'ollama';

// Simple chat
const response = await ollama.chat({
  model: 'llama3.1',
  messages: [{ role: 'user', content: 'What is Node.js?' }],
});
console.log(response.message.content);

// Generate
const gen = await ollama.generate({
  model: 'llama3.1',
  prompt: 'Write a TypeScript interface for a User',
});
console.log(gen.response);

Streaming with Async Iterators

import ollama from 'ollama';

const stream = await ollama.chat({
  model: 'llama3.1',
  messages: [{ role: 'user', content: 'Tell me a joke' }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.message.content);
}

Next.js API Route

// app/api/chat/route.ts
import ollama from 'ollama';
import { NextRequest } from 'next/server';

export async function POST(req: NextRequest) {
  const { message } = await req.json();

  const response = await ollama.chat({
    model: 'llama3.1',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: message },
    ],
  });

  return Response.json({
    reply: response.message.content,
  });
}

Browser Fetch (CORS required)

// Requires OLLAMA_ORIGINS=* or specific origin
const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  body: JSON.stringify({
    model: 'llama3.1',
    messages: [{ role: 'user', content: 'Hello' }],
    stream: false,
  }),
});
const data = await response.json();
console.log(data.message.content);

9. Docker Deployment

Basic Docker Run (GPU)

# CPU only
docker run -d -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU
docker run -d --gpus all -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama

# AMD GPU (ROCm)
docker run -d --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama -p 11434:11434 \
  --name ollama ollama/ollama:rocm

Docker Compose: Ollama + Open WebUI + n8n

version: "3.8"
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

  n8n:
    image: n8nio/n8n
    container_name: n8n
    ports:
      - "5678:5678"
    environment:
      - N8N_OLLAMA_URL=http://ollama:11434
    volumes:
      - n8n_data:/home/node/.n8n
    restart: unless-stopped

volumes:
  ollama_data:
  webui_data:
  n8n_data:

Nginx Reverse Proxy

# /etc/nginx/sites-available/ollama
server {
    listen 443 ssl;
    server_name ollama.example.com;

    ssl_certificate /etc/letsencrypt/live/ollama.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.example.com/privkey.pem;

    location / {
        proxy_pass http://localhost:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 600s;
        proxy_buffering off;  # Important for streaming
    }
}

10. OpenAI-Compatible API

Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions. This is a drop-in replacement for the OpenAI SDK in most tools.

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any string works
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Kubernetes?"},
    ],
    temperature=0.7,
)
print(response.choices[0].message.content)

curl

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Hello"}]
}'

Works With

LangChain — ChatOpenAI(base_url="http://localhost:11434/v1")
AutoGen — Set base_url in config list
CrewAI — Use ollama/llama3.1 as model identifier
Continue (VS Code) — Point to localhost:11434
Any OpenAI-compatible tool — Just change the base URL

⚠️ Limitations vs Real OpenAI API

No function calling / tool use (model-dependent)
No image inputs via the completions endpoint (use native API)
No fine-tuning API
No usage/billing endpoints
Rate limiting depends on your hardware, not plan tier

11. RAG & Embeddings

Embedding Models

Model	Dimensions	Best For
nomic-embed-text	768	General purpose, fast, small
mxbai-embed-large	1024	Higher quality, multilingual

Complete RAG Pipeline (Python)

import ollama
import chromadb
import os

# 1. Initialize ChromaDB
client = chromadb.Client()
collection = client.create_collection("docs")

# 2. Load and chunk documents
def chunk_text(text, size=500, overlap=50):
    chunks = []
    for i in range(0, len(text), size - overlap):
        chunks.append(text[i:i + size])
    return chunks

# Read documents from a directory
documents = []
for filename in os.listdir("./docs"):
    with open(f"./docs/{filename}") as f:
        text = f.read()
        chunks = chunk_text(text)
        for i, chunk in enumerate(chunks):
            documents.append({
                "id": f"{filename}_{i}",
                "text": chunk,
                "source": filename
            })

# 3. Embed and store
for doc in documents:
    embedding = ollama.embeddings(
        model="nomic-embed-text",
        prompt=doc["text"]
    )
    collection.add(
        ids=[doc["id"]],
        embeddings=[embedding["embedding"]],
        documents=[doc["text"]],
        metadatas=[{"source": doc["source"]}]
    )

# 4. Query
def rag_query(question):
    # Embed the question
    q_embedding = ollama.embeddings(
        model="nomic-embed-text",
        prompt=question
    )

    # Search for relevant chunks
    results = collection.query(
        query_embeddings=[q_embedding["embedding"]],
        n_results=3
    )

    # Build context from results
    context = "\n\n".join(results["documents"][0])

    # Generate answer with context
    response = ollama.chat(model="llama3.1", messages=[
        {"role": "system", "content": f"""Answer based on this context:
{context}

If the context doesn't contain the answer, say so."""},
        {"role": "user", "content": question}
    ])
    return response["message"]["content"]

# Usage
answer = rag_query("How do I deploy to production?")
print(answer)

RAG Pipeline Flow

Load docs → Chunk (500 chars) → Embed (nomic-embed-text) → Store (ChromaDB/pgvector) → Query → Retrieve top-k → Generate (llama3.1)

12. Production & Monitoring

Systemd Service

# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama LLM Service
After=network.target

[Service]
Type=simple
User=ollama
Group=ollama
ExecStart=/usr/local/bin/ollama serve
Restart=always
RestartSec=5
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_FLASH_ATTENTION=1"

[Install]
WantedBy=multi-user.target

Health Check & Preloading

# Health check
curl -f http://localhost:11434/api/tags || exit 1

# Preload models on startup (add to ExecStartPost or init script)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1", "prompt": "", "keep_alive": "24h"
}'

# Check running models
curl http://localhost:11434/api/ps | jq .

# Log management
journalctl -u ollama -f --no-pager
journalctl -u ollama --since "1 hour ago"

Load Balancing Multiple Instances

# Nginx upstream for multiple Ollama instances
upstream ollama_cluster {
    least_conn;
    server gpu-node-1:11434;
    server gpu-node-2:11434;
    server gpu-node-3:11434;
}

server {
    listen 443 ssl;
    server_name ai.example.com;

    location / {
        proxy_pass http://ollama_cluster;
        proxy_read_timeout 600s;
        proxy_buffering off;
    }
}

Security

Bind to 127.0.0.1 by default
Use reverse proxy + TLS for remote access
Firewall port 11434 from public access
Set OLLAMA_ORIGINS for CORS
No built-in auth — add at proxy layer

Troubleshooting

OOM — Reduce num_ctx or use smaller quantization
Slow inference — Check GPU offload with ollama ps
Model loading — Verify disk space, check ~/.ollama/models
GPU not detected — Verify drivers with nvidia-smi
Port in use — Change with OLLAMA_HOST