GitHub - hyperionaidev/smolmaxxing: maxxing the ability to run different LLMs on the smallest devices possible
maxxing the ability to run different LLMs on the smallest devices possible - hyperionaidev/smolmaxxing
maxxing the ability to run different LLMs on the smallest devices possible - hyperionaidev/smolmaxxing
Run the biggest language model you can on the smallest hardware you own.
TV boxes, old phones and tablets, single-board computers and routers all have a CPU, some RAM and flash storage that sit idle most of the day. smolmaxxing finds the biggest model each of them can really run, then pushes it further with custom kernels, careful memory use and weights streamed from flash. Every number in this repo was measured on the device itself.
| 📺 Xiaomi TV Box S (3rd Gen) | 📱 iPad Air (2013) | 📲 OnePlus Nord CE 2 (2022) | |
|---|---|---|---|
| Chip | Amlogic S7D, 4× Cortex-A55 | Apple A7, 2 cores | MediaTek Dimensity 900, 2× Cortex-A78 + 6× A55 |
| RAM | 1.9 GB (about 1 GB usable) | 1 GB (about 700 MB usable) | 7.4 GB (about 3 GB free) |
| OS | Android 14, no root | iOS 12.5.7, jailbroken | Android 13, no root |
| Everyday model | Qwen3.5-2B at 7.7 tok/s | Bonsai-1.7B (1-bit) at 24-26 tok/s | Qwen3.6-35B-A3B at 4.9-5.7 tok/s |
| Biggest model | Ling-mini-2.0, 16B, at 5.1 tok/s | Ling-mini-2.0, 16B, at 2.6-4.5 tok/s | Qwen3-Next-80B-A3B at 2.6-3.1 tok/s |
| Full results | docs/tv-box.md | ipad-air/README.md | docs/phone.md |
The TV box and the iPad run 8B and 16B mixture-of-experts models on about 1 GB of RAM, and the phone runs 35B and 80B ones with about 3 GB free. They all keep only part of each model in memory and read the rest from flash as it's needed.
The chart below compares the two 1 GB devices size class by size class; the phone plays in a bigger league, and its results are in docs/phone.md.
More devices (single-board computers, routers, more phones) are coming. The techniques aren't tied to one box: the 1-bit kernel needs only basic NEON, and the flash streaming works on any Linux or Android device with more model than RAM.
Got a small device? Run a model on it and submit your numbers: we'll add them to the results.
All of it is a patch on llama.cpp. The main ideas:
docs/how-it-works.md has every technique with its measured gain.
The quickest start is a ready-made model file from Hugging Face. The TV box and iPad files need the patched llama.cpp (stock llama.cpp can't load their pruned heads); the phone files load anywhere, but their speeds need the patch's expert pager.
| Page | What's in it |
|---|---|
| docs/tv-box.md | device #1: every model run on the TV box, speed and quality |
| ipad-air/README.md | device #2: the iPad Air results, setup and scripts |
| docs/phone.md | device #3: the OnePlus phone, from 16B to an 80B model |
| docs/how-it-works.md | the kernels, memory tricks, expert pager and streaming engine |
| docs/models.md | downloads for all three devices, rebuild targets and checksums |
| docs/lessons.md | what we learned, including what didn't work |
| EXPERIMENT.md | the full lab notebook: 165 experiments, failures included |
| Path | What |
|---|---|
patches/ |
every change to llama.cpp for the TV box and the phone (the iPad's patch is in ipad-air/patches/) |
tools/ |
model rebuilding, GGUF rewriting, benchmarks and kernel checks |
tv-llm-ui/ |
the live token-stream chat page and its launcher |
ipad-air/ |
device #2: iOS 12 build, smolchat, deploy and benchmark scripts |
bench/ |
every chat benchmark (results.jsonl) and the raw logs |
data/ |
vocabulary lists and evaluation texts |
tiny-llm-web/ |
the first experiment: a 260K-parameter story model |
zigcc/ |
zig wrappers for cross-compiling static aarch64 binaries |
Other people running language models on hardware that was never meant for them:
| Project | Hardware | What it runs |
|---|---|---|
| On-chip LLM on a KV260 | AMD Kria KV260 FPGA ($250) | a 3.16M-parameter INT4 transformer held entirely in on-chip memory, about 60K tok/s |
| ps2-llm | PlayStation 2 (294 MHz MIPS, 32 MB RAM) | a 10M-parameter instruct model |
| iiama2.c | Nintendo Wii (729 MHz PowerPC, 88 MB RAM) | llama2.c with stories15M, 61.9 tok/s on the GPU |
| esp32-ai | ESP32-S3 microcontroller | a 28.9M-parameter model at about 9.9 tok/s |
| go-pherence | CPUs with AVX2, NEON or RVV, and NVIDIA GPUs | pure-Go inference for language, speech and vision models |
| Ryan Seddon on X | Sega Dreamcast, via Windows CE | a language model running under the Dreamcast's Windows CE |
MIT. The llama.cpp patches are derivatives of llama.cpp (MIT). Model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts (CC BY-SA 4.0).
Built on llama.cpp and llama2.c. Models from Qwen, unsloth (GGUFs and importance matrices), PrismML (Bonsai), inclusionAI (Ling-mini-2.0), tablinum (Moon-Beam), bartowski and mradermacher (GGUFs), Google (Gemma) and stories260K. The iPad is jailbroken with Amethyst. The banner uses Archivo (SIL OFL) and is rendered by tools/banner/render.sh. The 🤏 logo is the pinching-hand emoji from Google's Noto Emoji (Apache 2.0).
Texto de github.com tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en github.com · Volver a la noticia de El Autómata