Skip to main content
The Models page is split into two lists:
  • Installed — models already on disk, ready to launch. Shows quantization, size, and architecture per model.
  • Browse — the live catalog: curated picks plus live Hugging Face search.

​
Downloading models

In Browse, add a model three ways:
  1. Catalog — curated models (including frontier MoE and NVFP4/AWQ picks for GB10 clusters), click to download.
  2. Search — search Hugging Face live; results show FITS GPU / availability badges computed from your cluster’s aggregate VRAM.
  3. Custom — paste any HF repo id (e.g. microsoft/Phi-4).
Downloads run in the background with a progress bar. Click ✕ to cancel. Already-downloaded models show as Installed instead of offering a re-download.

​
Launching a model

In Installed, click a model → ▶ Launch. On a single node AINode serves it directly; on a cluster you can pick which nodes span it (see Distributed Inference) or serve different models on different nodes (see Federated Serving).

​
Unloading a model

Each running instance has an Unload control that stops just that instance and frees its memory. (Unload force-clears a dead or phantom instance too, so a node that stopped responding doesn’t stay stuck “loaded”.) Unload a node’s models before starting a quantization job — quantization needs the full unified memory.

​
Serving from on-disk weights

AINode serves directly from the weights in ~/.ainode/models/<slug>. A model you downloaded once — or produced with a quantization or fine-tuning job — is immediately launchable; no re-download.

​
Gated models

Llama, Gemma, and other gated repos require a Hugging Face read token: 
ainode config --hf-token hf_xxxxxxxxxxxxxxxx
See Tokens & Secrets for read vs write tokens.

​
Supported models

Any model compatible with vLLM works. Tested families include:
  • Meta Llama 3 / 3.1 (8B, 70B, 405B)
  • Qwen 2.5 and Qwen3 / Qwen3.5 (incl. frontier MoE, e.g. Qwen3-235B-A22B)
  • Mistral / Mixtral
  • Google Gemma 2
  • DeepSeek V3
  • Phi-3 / Phi-4
  • Command R+
AWQ (awq_marlin) and NVFP4 quantized checkpoints serve natively on GB10.

​
Shared storage across cluster

To avoid downloading the same model on every node, mount a shared NFS path: 
// ~/.ainode/config.json on each node
{
  "models_dir": "/mnt/shared-models"
}
See Cluster Setup for NFS configuration details.