AutoGineer
Generated banner for Huggingface Local Models
Generated logo for Huggingface Local Models

Huggingface Local Models

Hugging FaceOfficialPublished by Hugging Face

  • Skill
  • Tool integration
  • Software development
  • Apache-2.0
  • Low risk
Download ZIP

From the author

Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.
— the description by Hugging Face

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to running LLMs locally on CPU, Apple Silicon, or GPU using llama.cpp. It covers the end-to-end process of finding compatible GGUF models on the Hugging Face Hub, selecting the appropriate quantization for your hardware, and launching models via llama-cli or llama-server. It emphasizes using Hugging Face's native local-app discovery tools and API endpoints to ensure accurate model selection and configuration.

What it does

  • Searches the Hugging Face Hub for llama.cpp-compatible GGUF repositories.
  • Extracts quantization recommendations and hardware compatibility data from Hugging Face.
  • Verifies exact model filenames using the Hugging Face tree API.
  • Provides commands for launching models via llama-cli and llama-server.
  • Offers guidance on converting non-GGUF models to the GGUF format.
  • Includes instructions for setting up local OpenAI-compatible API servers.

When to use it

  • When you need to run an LLM locally on your own hardware.
  • When you are unsure which quantization level is appropriate for your available RAM or VRAM.
  • When you need to serve a local model via an OpenAI-compatible API endpoint.
  • When you have a model in Transformers format and need to convert it to GGUF for llama.cpp.

Advantages

  • Leverages official Hugging Face discovery tools for reliable model selection.
  • Supports multiple hardware backends including CPU, Metal, CUDA, and ROCm.
  • Provides clear workflows for both quick-start and custom file-specific configurations.
  • Reduces ambiguity in quantization selection by prioritizing repo-native labels.

Usage examples

“How do I run the Qwen3.6-35B model locally using llama.cpp?”
The agent will guide you to the appropriate Hugging Face repo, help you select the correct GGUF quantization, and provide the exact llama-server command to launch it.
“I have a model repo without GGUF files. How can I convert it?”
The agent will provide the steps to download the model, use the conversion script to create an f16 GGUF file, and then quantize it to your desired format.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/huggingface/huggingface-local-models/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip huggingface-local-models.zip -d .agents/skills

Documentation by Hugging Face

Hugging Face Local Models

Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.

Default Workflow

  1. Search the Hub with apps=llama.cpp.

  2. Open https://huggingface.co/<repo>?local-app=llama.cpp.

  3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.

  4. Confirm exact .gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.

  5. Launch with llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.

  6. Fall back to --hf-repo plus --hf-file when the repo uses custom file naming.

  7. Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start

Install llama.cpp
brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make
Authenticate for gated repos
hf auth login
Search the Hub
https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
Run directly from the Hub
llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
Run an exact GGUF file
llama-server \
    --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
    --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -c 4096
Convert only when no GGUF is available
hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
    --outfile model-f16.gguf \
    --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
Smoke test a local server
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about exception handling"}
    ]
  }'

Quant Choice

  • Prefer the exact quant that HF marks as compatible on the ?local-app=llama.cpp page.

  • Keep repo-native labels such as UD-Q4_K_M instead of normalizing them.

  • Default to Q4_K_M unless the repo page or hardware profile suggests otherwise.

  • Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows.

  • Consider Q3_K_M, Q4_K_S, or repo-specific IQ / UD-* variants for tighter RAM or VRAM budgets.

  • Treat mmproj-*.gguf files as projector weights, not the main checkpoint.

Load References

  • Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.

  • Read quantization.md for format tables, model scaling, quality tradeoffs, and imatrix.

  • Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

Resources

  • llama.cpp: https://github.com/ggml-org/llama.cpp

  • Hugging Face GGUF + llama.cpp docs: https://huggingface.co/docs/hub/gguf-llamacpp

  • Hugging Face Local Apps docs: https://huggingface.co/docs/hub/main/local-apps

  • Hugging Face Local Agents docs: https://huggingface.co/docs/hub/agents-local

  • GGUF converter Space: https://huggingface.co/spaces/ggml-org/gguf-my-repo

Files

references/hub-discovery.md

Hugging Face URL Workflows for llama.cpp

Use URL-only workflows first. Do not require hf or API clients just to find GGUF files, choose a quant, or build a llama-server command.

Contents

  • Core URLs

  • Search for llama.cpp-compatible models

  • Use the local-app page for the recommended quant

  • Confirm exact files from the tree API

  • Build the command

  • Example: unsloth/Qwen3.6-35B-A3B-GGUF

  • Notes

Core URLs

Search:
https://huggingface.co/models?apps=llama.cpp&sort=trending

Search with text:
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending

Search with size bounds:
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

Repo local-app view:
https://huggingface.co/<repo>?local-app=llama.cpp

Repo tree API:
https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Repo file tree:
https://huggingface.co/<repo>/tree/main

1. Search for llama.cpp-compatible models

Start from the models page with apps=llama.cpp.

Use:

  • search=<term> for model family names such as Qwen, Gemma, Phi, or Mistral

  • num_parameters=min:0,max:24B or similar if the user has hardware limits

  • sort=trending when the user wants popular repos right now

Do not start with random GGUF repos if the user has not chosen a model family yet. Search first, shortlist second.

Example: https://huggingface.co/models?search=Qwen&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

2. Use the local-app page for the recommended quant

Open:

https://huggingface.co/<repo>?local-app=llama.cpp

Extract, in order:

  1. The exact Use this model snippet, if it is visible as text

  2. The Hardware compatibility section from the fetched page text or HTML:

    • quant label

    • file size

    • bit-depth grouping

  3. Any extra launch flags shown in the snippet, such as --jinja

Treat the HF local-app snippet as the source of truth when it is visible.

Do this by reading the URL itself, not by assuming the UI rendered in a browser. If the fetched page source does not expose Hardware compatibility, say that the section was not text-visible and fall back to the tree API plus generic guidance from quantization.md.

3. Confirm exact files from the tree API

Open:

https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Treat the JSON response as the source of truth for repo inventory.

Keep entries where:

  • type is file

  • path ends with .gguf

Use these fields:

  • path for the filename and subdirectory

  • size for the byte size

  • optionally lfs.size to confirm the LFS payload size

Separate files into:

  • quantized single-file checkpoints, for example Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

  • projector weights, usually mmproj-*.gguf

  • BF16 shard files, usually under BF16/

  • everything else

Ignore unless the user asks:

  • README.md

  • imatrix or calibration blobs

Use https://huggingface.co/<repo>/tree/main only as a human fallback if the API endpoint fails or the user wants the web view.

4. Build the command

Preferred order:

  1. Copy the exact HF snippet from the local-app page

  2. If the page gives a clean quant label, use shorthand selection:

llama-server -hf <repo>:<QUANT>

  1. If you need an exact file from the tree API, use the file-specific form:

llama-server --hf-repo <repo> --hf-file <filename.gguf>

  1. For CLI usage instead of a server, use:

llama-cli -hf <repo>:<QUANT>

Use the exact-file form when the repo uses custom labels or nonstandard naming that could make :<QUANT> ambiguous.

5. Example: unsloth/Qwen3.6-35B-A3B-GGUF

Use these URLs:

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main

On the local-app page, the hardware compatibility section can expose entries such as:

  • UD-IQ4_XS - 17.7 GB

  • UD-Q4_K_S - 20.9 GB

  • UD-Q4_K_M - 22.1 GB

  • UD-Q5_K_M - 26.5 GB

  • UD-Q6_K - 29.3 GB

  • Q8_0 - 36.9 GB

On the tree API, you can confirm exact filenames such as:

  • Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

  • Qwen3.6-35B-A3B-UD-Q5_K_M.gguf

  • Qwen3.6-35B-A3B-UD-Q6_K.gguf

  • Qwen3.6-35B-A3B-Q8_0.gguf

  • mmproj-F16.gguf

Good final output for this repo:

Repo: unsloth/Qwen3.6-35B-A3B-GGUF
Recommended quant from HF: UD-Q4_K_M (22.1 GB)
llama-server: llama-server --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
Other GGUFs:
- Qwen3.6-35B-A3B-UD-Q5_K_M.gguf - 26.5 GB
- Qwen3.6-35B-A3B-UD-Q6_K.gguf - 29.3 GB
- Qwen3.6-35B-A3B-Q8_0.gguf - 36.9 GB
Projector:
- mmproj-F16.gguf - 899 MB

Notes

  • Repo-specific quant labels matter. Do not rewrite UD-Q4_K_M to Q4_K_M unless the page itself does.

  • mmproj files are projector weights for multimodal models, not the main language model checkpoint.

  • If the HF hardware compatibility panel is missing because the user has no hardware profile configured, or because the fetched page source did not expose it, still use the tree API plus generic quant guidance from quantization.md.

  • If the repo already has GGUFs, do not jump straight to conversion workflows.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs a command-line tool, Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Hugging Face
Publisher
Hugging Face (Official)
Source
huggingface/skills · skills/huggingface-local-models · commit 25910b3
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Hugging Face or Hugging Face.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download huggingface-local-models.zip