← 返回列表
🔗 原文
28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s
📰 CNX Software 📅 2026-08-03 10:40 ✓ 已读

Slava S. (slvDev) has optimized a 28.9M-parameter LLM running locally on an ESP32-S3 development board at around 9 tokens/s while generating text, or more exactly, telling short stories.

We’ve written about LLMs and ESP32 in the past, but usually the boards are used as low-power gateways to interact with online LLMs hosted on powerful servers in data centers. Such products or projects include HiWonderLLM “smart chat module”, the ESP-Claw framework, or the ESP32 Agent Dev Kit. Slava’s “esp32-ai” project is quite different, as everything runs on the ESP32-S3 microcontroller.

ESP32-S3 local LLM

This won’t work on all ESP32-S3 as you’ll need enough memory and storage to store the 14.9MB (4-bit) model file, and Slava tested his demo on an ~$8 ESP32-S3 board with 512KB SRAM+8MB SPRAM and 16MB flash connected to a small I2C display to display short stories.

Dave Bennett already did something similar with the ESP32-LLM project, but it only relied on a 260K-parameter model, far from the 28.9M-parameter model used in this project. The trick was to stop putting the model in “fast memory” (the 512KB SRAM), as most of a language model’s parameters sit in an embedding table with around 25 million rows, which can be read from slow flash. So only the few rows each token needs, or about 450 bytes, need to be loaded and processed in the SRAM.

Slave explains that the idea is Google’s Per-Layer Embeddings used in Gemma 3n and Gemma 4. This is what the memory layout on the ESP32-S3 looks like:

This is more of an engineering feat than a useful LLM project, since the model was trained on TinyStories to write short, simple stories. It will not answer questions, follow instructions, write code, or anything you’d expect from a typical LLM. Here’s a demo below.

You’ll find the source code, instructions, and more details about the results on GitHub, a few additional tidbits may be found on Reddit, and in the video below if you prefer a visual explanation of how it all works.

Via Tom’s Hardware

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress. We also use affiliate links in articles to earn commissions if you make a purchase after clicking on those links.

Radxa Dragon Q8B Edge AI SBC with Snapdragon 8cx Gen3 SoC

AI 总结尚未生成

Slava S. (slvDev) has optimized a 28.9M-parameter LLM running locally on an ESP32-S3 development board at around 9 tokens/s while generating text, or more exactly, telling short stories.

We’ve written about LLMs and ESP32 in the past, but usually the boards are used as low-power gateways to interact with online LLMs hosted on powerful servers in data centers. Such products or projects include HiWonderLLM “smart chat module”, the ESP-Claw framework, or the ESP32 Agent Dev Kit. Slava’s “esp32-ai” project is quite different, as everything runs on the ESP32-S3 microcontroller.

ESP32-S3 local LLM

This won’t work on all ESP32-S3 as you’ll need enough memory and storage to store the 14.9MB (4-bit) model file, and Slava tested his demo on an ~$8 ESP32-S3 board with 512KB SRAM+8MB SPRAM and 16MB flash connected to a small I2C display to display short stories.

Dave Bennett already did something similar with the ESP32-LLM project, but it only relied on a 260K-parameter model, far from the 28.9M-parameter model used in this project. The trick was to stop putting the model in “fast memory” (the 512KB SRAM), as most of a language model’s parameters sit in an embedding table with around 25 million rows, which can be read from slow flash. So only the few rows each token needs, or about 450 bytes, need to be loaded and processed in the SRAM.

Slave explains that the idea is Google’s Per-Layer Embeddings used in Gemma 3n and Gemma 4. This is what the memory layout on the ESP32-S3 looks like:

This is more of an engineering feat than a useful LLM project, since the model was trained on TinyStories to write short, simple stories. It will not answer questions, follow instructions, write code, or anything you’d expect from a typical LLM. Here’s a demo below.

You’ll find the source code, instructions, and more details about the results on GitHub, a few additional tidbits may be found on Reddit, and in the video below if you prefer a visual explanation of how it all works.

Via Tom’s Hardware

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress. We also use affiliate links in articles to earn commissions if you make a purchase after clicking on those links.

Radxa Dragon Q8B Edge AI SBC with Snapdragon 8cx Gen3 SoC
原文
28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s

Slava S. (slvDev) has optimized a 28.9M-parameter LLM running locally on an ESP32-S3 development board at around 9 tokens/s while generating text, or more exactly, telling short stories.

We’ve written about LLMs and ESP32 in the past, but usually the boards are used as low-power gateways to interact with online LLMs hosted on powerful servers in data centers. Such products or projects include HiWonderLLM “smart chat module”, the ESP-Claw framework, or the ESP32 Agent Dev Kit. Slava’s “esp32-ai” project is quite different, as everything runs on the ESP32-S3 microcontroller.

ESP32-S3 local LLM

This won’t work on all ESP32-S3 as you’ll need enough memory and storage to store the 14.9MB (4-bit) model file, and Slava tested his demo on an ~$8 ESP32-S3 board with 512KB SRAM+8MB SPRAM and 16MB flash connected to a small I2C display to display short stories.

Dave Bennett already did something similar with the ESP32-LLM project, but it only relied on a 260K-parameter model, far from the 28.9M-parameter model used in this project. The trick was to stop putting the model in “fast memory” (the 512KB SRAM), as most of a language model’s parameters sit in an embedding table with around 25 million rows, which can be read from slow flash. So only the few rows each token needs, or about 450 bytes, need to be loaded and processed in the SRAM.

Slave explains that the idea is Google’s Per-Layer Embeddings used in Gemma 3n and Gemma 4. This is what the memory layout on the ESP32-S3 looks like:

This is more of an engineering feat than a useful LLM project, since the model was trained on TinyStories to write short, simple stories. It will not answer questions, follow instructions, write code, or anything you’d expect from a typical LLM. Here’s a demo below.

You’ll find the source code, instructions, and more details about the results on GitHub, a few additional tidbits may be found on Reddit, and in the video below if you prefer a visual explanation of how it all works.

Via Tom’s Hardware

Share this:

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress. We also use affiliate links in articles to earn commissions if you make a purchase after clicking on those links.

Radxa Dragon Q8B Edge AI SBC with Snapdragon 8cx Gen3 SoC
中文翻译
28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s

翻译尚未生成

$299.99 ASUS UGen300 USB ...
ChronoWatch X2040 – A DIY...