# Technical Notes - ht-docker-ai

## Architecture

This project uses **Ollama** as the runtime framework for serving AI models. This provides:

- Automatic model download and caching
- Unified REST API (compatible with OpenAI format)
- Built-in quantization support
- GPU/CPU auto-detection

## Model Details

### MiniCPM-V 4.5

- **Source**: OpenBMB (https://github.com/OpenBMB/MiniCPM-V)
- **Base Models**: Qwen3-8B + SigLIP2-400M
- **Total Parameters**: 8B
- **Ollama Model Name**: `minicpm-v`

### VRAM Usage

| Mode | VRAM Required |
|------|---------------|
| Full precision (bf16) | 18GB |
| int4 quantized | 9GB |
| GGUF (CPU) | 8GB RAM |

## Container Startup Flow

1. `docker-entrypoint.sh` starts Ollama server in background
2. Waits for server to be ready
3. Checks if model already exists in volume
4. Pulls model if not present
5. Keeps container running

## Volume Persistence

Mount `/root/.ollama` to persist downloaded models:

```bash
-v ollama-data:/root/.ollama
```

Without this volume, the model will be re-downloaded on each container start (~5GB download).

## API Endpoints

All endpoints follow the Ollama API specification:

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/tags` | GET | List available models |
| `/api/generate` | POST | Generate completion |
| `/api/chat` | POST | Chat completion |
| `/api/pull` | POST | Pull a model |
| `/api/show` | POST | Show model info |

## GPU Detection

The GPU variant uses Ollama's automatic GPU detection. For CPU-only mode, we set:

```dockerfile
ENV CUDA_VISIBLE_DEVICES=""
```

This forces Ollama to use CPU inference even if GPU is available.

## Health Checks

Both variants include Docker health checks:

```dockerfile
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:11434/api/tags || exit 1
```

CPU variant has longer `start-period` (120s) due to slower startup.

## Adding New Models

To add a new model variant:

1. Create `Dockerfile_<modelname>`
2. Set `MODEL_NAME` environment variable
3. Update `build-images.sh` with new build target
4. Add documentation to `readme.md`

## Troubleshooting

### Model download hangs

Check container logs:
```bash
docker logs -f <container-name>
```

The model download is ~5GB and may take several minutes.

### Out of memory

- GPU: Use int4 quantized version or add more VRAM
- CPU: Increase container memory limit: `--memory=16g`

### API not responding

1. Check if container is healthy: `docker ps`
2. Check logs for errors: `docker logs <container>`
3. Verify port mapping: `curl localhost:11434/api/tags`

## CI/CD Integration

Build and push using npmci:

```bash
npmci docker login
npmci docker build
npmci docker push code.foss.global
```

## Related Resources

- [Ollama Documentation](https://ollama.ai/docs)
- [MiniCPM-V GitHub](https://github.com/OpenBMB/MiniCPM-V)
- [Ollama API Reference](https://github.com/ollama/ollama/blob/main/docs/api.md)