ModelGrid is a root-level daemon that turns any GPU-equipped machine into a vLLM-serving node. It manages single-model vLLM deployments across NVIDIA, AMD, and Intel GPUs, exposes a unified OpenAI-compatible API, and resolves deployable models from list.modelgrid.com.

┌─────────────────────────────────────────────────────────────────────────┐
│                          ModelGrid Daemon                                │
│  ┌───────────────┐   ┌───────────────┐   ┌───────────────────────────┐  │
│  │  GPU Manager  │   │  vLLM Deploy  │   │   OpenAI-Compatible API   │  │
│  │ NVIDIA/AMD/   │──▶│   Scheduler    │──▶│   /v1/chat/completions    │  │
│  │ Intel Arc     │   │ + Cluster Base │   │   /v1/models              │  │
│  └───────────────┘   └───────────────┘   │   /v1/embeddings          │  │
│           │                  │            └───────────────────────────┘  │
│           └──── list.modelgrid.com catalog + deployment metadata ───────┘
└─────────────────────────────────────────────────────────────────────────┘

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

✨ Features

🎯 OpenAI-Compatible API — Drop-in replacement for OpenAI's API. Works with existing tools, SDKs, and applications
🖥️ Multi-GPU Support — Auto-detect and manage NVIDIA (CUDA), AMD (ROCm), and Intel Arc (oneAPI) GPUs
📦 vLLM Deployments — Launch model-specific vLLM runtimes instead of hand-managing containers
📚 OSS Model Catalog — Resolve supported models from list.modelgrid.com
🕸️ Cluster Foundation — Cluster-aware config surface for standalone, control-plane, and worker roles
⚡ Streaming Support — Real-time token streaming via Server-Sent Events
🔄 Auto-Recovery — Health monitoring with automatic container restart
🐳 Docker Native — Full Docker/Podman integration with isolated networking
📊 Prometheus Metrics — Built-in /metrics endpoint for monitoring
🖥️ Cross-Platform — Pre-compiled binaries for Linux, macOS, and Windows

📥 Installation

Via Installer Script (Recommended)

curl -fsSL https://code.foss.global/modelgrid.com/modelgrid/releases/download/latest/install.sh | sudo bash

The installer is an asset of every release from 2.0.1 on. sudo modelgrid update installs the newest release with that release's installer; on 2.0.0 it fails, so update a 2.0.0 host once with the command above.

Manual Binary Download

Download the appropriate binary for your platform from releases:

Platform	Binary
Linux x64	`modelgrid-linux-x64`
Linux ARM64	`modelgrid-linux-arm64`
macOS Intel	`modelgrid-macos-x64`
macOS Apple Silicon	`modelgrid-macos-arm64`
Windows x64	`modelgrid-windows-x64.exe`

chmod +x modelgrid-linux-x64
sudo mv modelgrid-linux-x64 /usr/local/bin/modelgrid

The executables are Node.js 26 single executable applications. Linux hosts need glibc 2.28 or later, libstdc++6 and libatomic1 (libatomic.so.1); Alpine/musl is not supported. The Windows executable is built but not tested.

🚀 Quick Start

# 1. Check your GPUs
sudo modelgrid gpu list

# 2. Initialize configuration
sudo modelgrid config init

# 3. Add an API key
sudo modelgrid config apikey add

# 4. Browse the public catalog
sudo modelgrid model list

# 5. Deploy a model
sudo modelgrid run meta-llama/Llama-3.1-8B-Instruct

# 6. Enable and start the service
sudo modelgrid service enable
sudo modelgrid service start

# 7. Test the API
curl http://localhost:8080/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

That's it! Your GPU server is now serving AI models with an OpenAI-compatible API. 🎉

📖 API Reference

ModelGrid exposes a fully OpenAI-compatible API on port 8080 (configurable).

Authentication

All API endpoints require Bearer token authentication:

curl -H "Authorization: Bearer YOUR_API_KEY" http://localhost:8080/v1/models

Chat Completions

POST /v1/chat/completions

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is machine learning?"}
    ],
    "temperature": 0.7,
    "max_tokens": 1024,
    "stream": false
  }'

Streaming Response:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "messages": [{"role": "user", "content": "Write a poem about AI"}],
    "stream": true
  }' --no-buffer

List Models

GET /v1/models

Returns all available models across all containers:

{
  "object": "list",
  "data": [
    {
      "id": "llama3:8b",
      "object": "model",
      "owned_by": "modelgrid",
      "created": 1706745600
    }
  ]
}

Embeddings

POST /v1/embeddings

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "input": "Hello, world!"
  }'

Health Check (No Auth Required)

GET /health

{
  "status": "ok",
  "uptime": 3600,
  "containers": { "total": 2, "running": 2 },
  "models": 5,
  "gpus": 2
}

Prometheus Metrics (No Auth Required)

GET /metrics

# HELP modelgrid_uptime_seconds Server uptime in seconds
modelgrid_uptime_seconds 3600
# HELP modelgrid_containers_total Total configured containers
modelgrid_containers_total 2
# HELP modelgrid_containers_running Running containers
modelgrid_containers_running 2

🔧 CLI Commands

Service Management

modelgrid service enable       # Install and enable systemd service
modelgrid service disable      # Stop and disable service
modelgrid service start        # Start the daemon
modelgrid service stop         # Stop the daemon
modelgrid service restart      # Restart the daemon
modelgrid service status       # Show service status with GPU/container info
modelgrid service logs         # Tail live service logs

GPU Management

modelgrid gpu list             # List all detected GPUs with VRAM info
modelgrid gpu status           # Show real-time GPU utilization
modelgrid gpu drivers          # Check driver status for all GPUs
modelgrid gpu install          # Install GPU drivers (interactive)

Example output:

GPU Devices (2):
┌────────────┬──────────────────────┬────────┬─────────────┬────────────┐
│ ID         │ Model                │ VRAM   │ Driver      │ Status     │
├────────────┼──────────────────────┼────────┼─────────────┼────────────┤
│ nvidia-0   │ NVIDIA RTX 4090      │ 24 GB  │ 535.154.05  │ Ready      │
│ nvidia-1   │ NVIDIA RTX 4090      │ 24 GB  │ 535.154.05  │ In Use     │
└────────────┴──────────────────────┴────────┴─────────────┴────────────┘

Deployment Management

modelgrid ps                   # List active deployments
modelgrid run MODEL            # Deploy a model from the registry
modelgrid container list       # List all configured deployments
modelgrid container add        # Interactive deployment setup wizard
modelgrid container remove ID  # Remove a deployment
modelgrid container start [ID] # Start deployment(s)
modelgrid container stop [ID]  # Stop deployment(s)
modelgrid container logs ID    # Show deployment logs

Model Management

modelgrid model list           # List available/loaded models
modelgrid model pull NAME      # Deploy a model from the registry
modelgrid model remove NAME    # Remove a model deployment
modelgrid model status         # Show model recommendations with VRAM analysis
modelgrid model refresh        # Refresh registry cache

Configuration

modelgrid config show          # Display current configuration
modelgrid config init          # Initialize default configuration
modelgrid config apikey list   # List configured API keys
modelgrid config apikey add    # Generate and add new API key
modelgrid config apikey remove # Remove an API key

Cluster

modelgrid cluster status       # Show cluster state
modelgrid cluster nodes        # List registered nodes
modelgrid cluster models       # Show model locations across nodes
modelgrid cluster desired      # Show desired deployment targets
modelgrid cluster ensure NAME  # Ask control plane to schedule a model
modelgrid cluster scale NAME 3 # Set desired replica count
modelgrid cluster clear NAME   # Remove desired deployment target
modelgrid cluster cordon NODE  # Prevent new placements on a node
modelgrid cluster drain NODE   # Mark a node for evacuation
modelgrid cluster activate NODE # Mark a node active again

Global Options

--debug, -d    # Enable debug mode (verbose logging)
--version, -v  # Show version information
--help, -h     # Show help message

📦 Supported Runtime

vLLM

High-performance inference with PagedAttention and continuous batching.

{
  "id": "vllm-1",
  "type": "vllm",
  "name": "vLLM Server",
  "gpuIds": ["nvidia-0", "nvidia-1"], // Tensor parallelism
  "port": 8000,
  "env": {
    "HF_TOKEN": "your-huggingface-token" // For gated models
  }
}

Best for: Production workloads, multi-GPU tensor parallelism, OpenAI-compatible serving

🎯 GPU Support

NVIDIA (CUDA)

Requirements:

NVIDIA Driver 470+
CUDA Toolkit 11.0+
NVIDIA Container Toolkit (nvidia-docker2)

# Check status
modelgrid gpu drivers

modelgrid gpu install follows the vendor guides:

Ubuntu (Ubuntu's NVIDIA driver guide): the server (-server) driver series that ubuntu-drivers list --gpgpu --recommended names for the GPU, from Ubuntu's archive with Ubuntu's pre-built, signed kernel modules (Secure Boot works). The graphics-drivers PPA is not added. The installation stops when ubuntu-drivers recommends no server driver or apt cannot install one of the packages.
Debian 12/13 (Debian's NVIDIA guide): the release's packaged driver; a backports kernel is refused.
RHEL, Rocky Linux, AlmaLinux 8/9/10 and Fedora 44 (NVIDIA's driver installation guide): EPEL and its prerequisite repositories (RHEL: CodeReady Builder; Rocky/Alma: CRB or PowerTools), the kernel development packages, NVIDIA's CUDA repository and nvidia-open (open kernel modules; the nvidia-driver:open-dkms module stream on release 8 and 9).

The same driver by hand on Ubuntu (the series and flavour are the ones ubuntu-drivers prints; modules first, then the driver, as Ubuntu's guide describes):

sudo apt-get update && sudo apt-get install -y ubuntu-drivers-common &&
sudo ubuntu-drivers list --gpgpu --recommended
# for example: nvidia-driver-580-server linux-modules-nvidia-580-server-generic
sudo apt-get install -y linux-modules-nvidia-580-server-generic "linux-modules-nvidia-580-server-$(uname -r)" &&
sudo apt-get install -y nvidia-driver-580-server

Then add the NVIDIA Container Toolkit repository; every download is saved and checked before anything is installed, and each step runs only when the one before succeeded:

cd "$(mktemp -d)" &&
curl -fsSLo toolkit.key https://nvidia.github.io/libnvidia-container/gpgkey &&
gpg --show-keys toolkit.key &&
sudo gpg --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg --dearmor toolkit.key &&
curl -fsSLo toolkit.list https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list &&
grep '^deb https://' toolkit.list &&
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' toolkit.list > signed.list &&
sudo install -m 0644 signed.list /etc/apt/sources.list.d/nvidia-container-toolkit.list &&
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit &&
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

AMD (ROCm)

Requirements:

ROCm 5.0+
AMD GPU with ROCm support (RX 6000+, MI series)

Manual install on Ubuntu, as modelgrid gpu install does it (ROCm 6.0 publishes Ubuntu 22.04 jammy; the Release checks stop the steps for a release a repository does not publish):

. /etc/os-release && CODENAME="${UBUNTU_CODENAME:-$VERSION_CODENAME}" && ROCM=6.0 &&
cd "$(mktemp -d)" &&
curl -fsSLo rocm.Release "https://repo.radeon.com/rocm/apt/$ROCM/dists/$CODENAME/Release" &&
grep -Ex "Suite: $CODENAME" rocm.Release &&
curl -fsSLo amdgpu.Release "https://repo.radeon.com/amdgpu/$ROCM/ubuntu/dists/$CODENAME/Release" &&
grep -Ex "Suite: $CODENAME" amdgpu.Release &&
curl -fsSLo rocm.key https://repo.radeon.com/rocm/rocm.gpg.key &&
gpg --show-keys rocm.key &&
sudo gpg --yes -o /usr/share/keyrings/rocm.gpg --dearmor rocm.key &&
printf 'deb [arch=amd64 signed-by=/usr/share/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/%s %s main\n' "$ROCM" "$CODENAME" > rocm.list &&
printf 'deb [arch=amd64 signed-by=/usr/share/keyrings/rocm.gpg] https://repo.radeon.com/amdgpu/%s/ubuntu %s main\n' "$ROCM" "$CODENAME" > amdgpu.list &&
sudo install -m 0644 rocm.list amdgpu.list /etc/apt/sources.list.d/ &&
sudo apt-get update && sudo apt-get install -y amdgpu-dkms rocm-hip-sdk &&
sudo usermod -a -G video,render "$USER"

Intel Arc (oneAPI)

Requirements:

Intel oneAPI Base Toolkit
Intel Arc A-series GPU (A770, A750, A380)

Manual oneAPI install on Ubuntu, as modelgrid gpu install does it:

cd "$(mktemp -d)" &&
curl -fsSLo oneapi.key https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB &&
gpg --show-keys oneapi.key &&
sudo gpg --yes -o /usr/share/keyrings/oneapi-archive-keyring.gpg --dearmor oneapi.key &&
printf 'deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main\n' > oneAPI.list &&
sudo install -m 0644 oneAPI.list /etc/apt/sources.list.d/oneAPI.list &&
sudo apt-get update && sudo apt-get install -y intel-basekit &&
sudo usermod -a -G render "$USER"

⚙️ Configuration

Configuration is stored at /etc/modelgrid/config.json:

{
  "version": "1.0",
  "api": {
    "port": 8080,
    "host": "0.0.0.0",
    "apiKeys": ["sk-your-api-key-here"],
    "cors": false,
    "corsOrigins": ["*"],
    "rateLimit": 60
  },
  "docker": {
    "networkName": "modelgrid",
    "runtime": "docker",
    "socketPath": "/var/run/docker.sock"
  },
  "gpus": {
    "autoDetect": true,
    "assignments": {}
  },
  "containers": [
    {
      "id": "vllm-llama31-8b",
      "type": "vllm",
      "name": "Primary vLLM",
      "image": "vllm/vllm-openai:latest",
      "gpuIds": ["nvidia-0"],
      "port": 8000,
      "models": ["meta-llama/Llama-3.1-8B-Instruct"],
      "env": {},
      "volumes": []
    }
  ],
  "models": {
    "registryUrl": "https://list.modelgrid.com/catalog/models.json",
    "autoDeploy": true,
    "defaultEngine": "vllm",
    "autoLoad": ["meta-llama/Llama-3.1-8B-Instruct"]
  },
  "cluster": {
    "enabled": false,
    "nodeName": "modelgrid-local",
    "role": "standalone",
    "bindHost": "0.0.0.0",
    "gossipPort": 7946,
    "sharedSecret": "",
    "advertiseUrl": "http://127.0.0.1:8080",
    "heartbeatIntervalMs": 5000,
    "seedNodes": []
  },
  "checkInterval": 30000
}

Configuration Options

Option	Description	Default
`api.port`	API server port	`8080`
`api.host`	Bind address	`0.0.0.0`
`api.apiKeys`	Valid API keys	`[]`
`api.rateLimit`	Requests per minute	`60`
`docker.runtime`	Container runtime	`docker`
`gpus.autoDetect`	Auto-detect GPUs	`true`
`models.autoDeploy`	Auto-start deployments on demand	`true`
`models.autoLoad`	Models to preload on start	`[]`
`cluster.role`	Cluster mode	`standalone`
`cluster.sharedSecret`	Shared secret for `/_cluster/*`	unset
`cluster.advertiseUrl`	URL advertised to other nodes	`http://127.0.0.1:8080`
`cluster.controlPlaneUrl`	Control-plane URL for workers	unset
`checkInterval`	Health check interval (ms)	`30000`

🕸️ Clustering

Cluster mode uses ModelGrid's internal control-plane endpoints to:

register worker nodes
advertise locally deployed models
persist desired deployment targets separately from live heartbeats
schedule new deployments onto healthy nodes with enough VRAM
proxy OpenAI-compatible requests to the selected node gateway
exclude cordoned or draining nodes from new placements

Minimal setup:

{
  "cluster": {
    "enabled": true,
    "nodeName": "worker-a",
    "role": "worker",
    "sharedSecret": "replace-me-with-a-random-secret",
    "advertiseUrl": "http://worker-a.internal:8080",
    "controlPlaneUrl": "http://control.internal:8080",
    "heartbeatIntervalMs": 5000
  }
}

For the control plane, set role to control-plane and advertiseUrl to its reachable API URL. Set the same cluster.sharedSecret on every node to protect internal cluster endpoints.

Runtime state files:

/var/lib/modelgrid/cluster-state.json for live node heartbeats
/var/lib/modelgrid/cluster-control-state.json for desired deployments and node lifecycle state

📚 Model Catalog

ModelGrid resolves deployable models from list.modelgrid.com. The catalog is public, versioned, and describes:

canonical model IDs
aliases
minimum VRAM and GPU count
vLLM launch defaults
desired replica counts
capabilities like chat, completions, and embeddings

Default catalog includes:

Qwen/Qwen2.5-7B-Instruct
meta-llama/Llama-3.1-8B-Instruct
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
BAAI/bge-m3

Custom registry source:

// models.json
{
  "version": "1.0",
  "generatedAt": "2026-04-20T00:00:00.000Z",
  "models": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "engine": "vllm",
      "source": { "repo": "meta-llama/Llama-3.1-8B-Instruct" },
      "capabilities": { "chat": true },
      "requirements": { "minVramGb": 18 },
      "launchDefaults": { "replicas": 2 }
    }
  ]
}

Configure with:

{
  "models": {
    "registryUrl": "https://your-server.com/models.json"
  }
}

🏗️ Development

Building from Source

# Clone repository
git clone https://code.foss.global/modelgrid.com/modelgrid.git
cd modelgrid
pnpm install

# Run from source
pnpm run dev help

# Build (tsbuild, dist_ts/) and type-check the tests
pnpm build
pnpm exec tsbuild check 'test/**/*'

# Run the test suite
pnpm test

# Build the five executables into dist/binaries/ (tsbundle, then tssea)
pnpm run build:binary

# Check the UI the linux-x64 executable serves from its embedded asset tree (needs Docker)
pnpm run check:embedded-ui

Project Structure

modelgrid/
├── ts/
│   ├── main.ts               # Executable entry point
│   ├── plugins.ts            # Dependency imports
│   ├── cli.ts                # CLI command routing
│   ├── modelgrid.ts          # Main coordinator class
│   ├── daemon.ts             # Background daemon process
│   ├── systemd.ts            # Systemd service integration
│   ├── constants.ts          # Configuration constants
│   ├── logger.ts             # Logging utilities
│   ├── interfaces/           # TypeScript interfaces
│   ├── hardware/             # GPU detection (NVIDIA/AMD/Intel)
│   ├── drivers/              # Driver management
│   ├── docker/               # Docker management
│   ├── containers/           # Container orchestration
│   │   ├── vllm.ts           # vLLM implementation
│   │   └── tgi.ts            # TGI implementation
│   ├── api/                  # OpenAI-compatible API
│   │   ├── server.ts         # HTTP server
│   │   ├── router.ts         # Request routing
│   │   └── handlers/         # Endpoint handlers
│   ├── models/               # Model management
│   ├── ui/                   # Operations console server
│   └── cli/                  # CLI handlers
├── ts_web/                   # Operations console assets (embedded in the executable)
├── test/                     # tstest tests (test.*.node.ts)
└── scripts/                  # Manual checks and demos

Focused seam tests live alongside test/test.core.node.ts:

test/test.api-router.node.ts covers routing, auth failures, and request-size handling
test/test.api-server.node.ts covers health, metrics, and authenticated model listing
test/test.modelgrid-config.node.ts covers config normalization and ignored-key warnings
test/test.model-registry.node.ts covers fallback and file-backed catalog loading
test/test.cluster-manager-persistence.node.ts covers persisted cluster state loading and pruning
test/test.ui-server.node.ts covers the console's assets, SPA shell and overview

🗑️ Uninstallation

# Stop and remove service
sudo modelgrid service disable

# Uninstall via script
sudo modelgrid uninstall

# Or run the uninstaller the installer placed beside the executable
sudo bash /opt/modelgrid/uninstall.sh

# Or manual removal
sudo rm /usr/local/bin/modelgrid
sudo rm -rf /etc/modelgrid
sudo rm -rf /opt/modelgrid
sudo rm /etc/systemd/system/modelgrid.service
sudo systemctl daemon-reload

📚 Resources

Repository: https://code.foss.global/modelgrid.com/modelgrid
Issues: https://community.foss.global/
Releases: https://code.foss.global/modelgrid.com/modelgrid/releases

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the LICENSE file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

🚀 ModelGrid

Issue Reporting and Security

✨ Features

📥 Installation

Via Installer Script (Recommended)

Manual Binary Download

🚀 Quick Start

📖 API Reference

Authentication

Chat Completions

List Models

Embeddings

Health Check (No Auth Required)

Prometheus Metrics (No Auth Required)

🔧 CLI Commands

Service Management

GPU Management

Deployment Management

Model Management

Configuration

Cluster

Global Options

📦 Supported Runtime

vLLM

🎯 GPU Support

NVIDIA (CUDA)

AMD (ROCm)

Intel Arc (oneAPI)

⚙️ Configuration

Configuration Options

🕸️ Clustering

📚 Model Catalog

🏗️ Development

Building from Source

Project Structure

🗑️ Uninstallation

📚 Resources

License and Legal Information

Trademarks

Company Information