HF Gradio + llama.cpp Bootstrap

Battle-tested rules for Hugging Face Gradio Spaces that run local GGUF models with llama.cpp.

When to use it
Any time you are deploying or debugging a Gradio Space that needs a local GGUF — companions, chairman models, private inference endpoints, L.I.P.S.-style agents, or anything else that must load a GGUF on ZeroGPU or paid GPU hardware.

What it fixes

• The #1 failure mode: Gradio / other imports running before llama is ready
• ZeroGPU vs paid-GPU confusion (they require completely different load patterns)
• Pip ResolutionImpossible from conflicting Gradio version pins
• Model loads on CPU / OOM when the GGUF is initialized at import time on ZeroGPU
• Missing-module crashes after a clean build
• Repeated multi-day rediscovery of the same bootstrap order

Core Rule (non-negotiable)

llama must start before anything else.

In app.py the very first import (before Gradio, before any internal modules) must be the bootstrap:

# ═══════════════════════════════════════════════════════════════
# STEP 0 — llama.cpp MUST init before ANY other import
# ═══════════════════════════════════════════════════════════════
import llama_bootstrap  # noqa: F401, E402

Then the rest of the imports. Never put Gradio or heavy modules above this line.

Two Hardware Shapes — Never Confuse Them

Shape	Hardware	How GPU appears	Correct load pattern
Paid GPU (L4, A10, etc.)	suggested_hardware: l4x1	Persistent for the container lifetime	Bootstrap llama-server or llama-cpp-python at process start
ZeroGPU	Free Gradio ZeroGPU	Only inside @spaces.GPU decorated functions	Load GGUF inside the decorated function on first call. Do not start a long-lived server at import time

ZeroGPU will OOM or hang if you try to load a 16–30 GB GGUF at module import. The factory does not give you a GPU until the decorated function runs.

Requirements.txt Rules

• Never pin multiple conflicting Gradio versions.
• Factory already installs a Gradio + spaces combination. Prefer not listing gradio or spaces in requirements unless you have a hard reason.
• For ZeroGPU + llama-cpp-python:

httpx>=0.27.0
huggingface_hub>=0.30.0
llama-cpp-python==0.3.16

• For paid GPU with prebuilt wheels, prefer installing the correct CUDA wheel inside llama_bootstrap.py rather than baking a huge wheel into requirements.

README.md YAML (Gradio SDK)

---
title: Your Space Name
emoji: 🧠
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.33.0   # or current factory version
python_version: "3.10"  # or 3.11
app_file: app.py
pinned: false
startup_duration_timeout: 1h
# For paid GPU only:
# suggested_hardware: l4x1
---

Match sdk_version to what the factory is currently installing or you will get pip ResolutionImpossible.

Bootstrap Patterns

Pattern A — Paid GPU / L4 (server or in-process)

llama_bootstrap.py (or equivalent) runs at import, detects CUDA, installs the right wheel if needed, starts llama-server or loads Llama(), and sets env vars the rest of the app expects (LLM_BASE_URL, N_GPU_LAYERS, etc.).

app.py imports it first, then builds the UI.

Pattern B — ZeroGPU (the path that commonly fails)

1. Do not start a server at import.
2. Keep supporting modules (engine.py, affect/state modules, pose modules, etc.) as separate files.
3. In the model-access module, expose a stable public API (complete, status, loaded, prefetch_default, has_gpu).
4. Inside the function that actually talks to the model (decorated with @spaces.GPU(duration=180)), call an _ensure_direct() helper which:
   • Finds the GGUF under /data (bucket mount)
   • Instantiates llama_cpp.Llama with n_gpu_layers=-1 only when CUDA is visible
   • Caches the instance for the life of the container
5. Prefer server mode only if http://127.0.0.1:8080/health already answers (Docker path).

Common Failure Modes (and the fix)

Symptom	Root cause	Fix
from __future__ import annotations SyntaxError	Future import not first statement	Put it as absolute first line after the module docstring
pip ResolutionImpossible (gradio 5.x vs 6.x)	requirements.txt + factory both pinning different Gradio	Remove gradio/spaces pins from requirements
ModuleNotFoundError for local modules	Only app.py uploaded	Upload every .py module the app imports
Model loads on CPU / OOM on ZeroGPU	Prefetch or Llama() at module level	Move load inside @spaces.GPU
Build succeeds but first chat hangs	Waiting for a server that never started	Use direct llama-cpp-python path on ZeroGPU
System prompt or adult/companion tone stripped	Over-cautious content filtering	Adult content between consenting adults is allowed. Preserve the intended tone

File Checklist for a Working Local-GGUF Space

Minimum root files for ZeroGPU + local GGUF:

• app.py (imports bootstrap first if using Pattern A; otherwise clean Gradio entry)
• Model-access module (e.g. engine.py) — ZeroGPU-safe version
• Any supporting modules the app imports
• requirements.txt (minimal)
• README.md (correct YAML)

Optional: portrait/assets, packages.txt only if you truly need build-essential/cmake.

When Creating or Fixing a Space

1. Ask which hardware shape (ZeroGPU vs paid L4/A10).
2. Confirm the GGUF is already in the bucket mounted at /data.
3. Enforce the bootstrap order for paid GPU.
4. For ZeroGPU, enforce load-inside-@spaces.GPU.
5. Preserve the intended system prompt and tone unless the user explicitly requests changes.
6. After the first successful build, the next error is almost always a missing module or the GPU load path — not another pip fight.

Usage

Use this skill whenever an agent starts arguing about import order, Gradio vs Docker, or "why won't the model load." Hand them this document and stop re-deriving it.

---

Credits
Skill authored by Grok (xAI) from hard-won deployment lessons. Shared under MIT so the community stops losing days to the same bootstrap failures.