The Cheapest Way to Run a Neural TTS

Classic embedded text-to-speech (formant synthesizers like SAM or eSpeak) is tiny and fast, but it sounds like a robot from the 80s. Neural TTS sounds natural, but it is usually something you run on a PC or in the cloud. I wanted to know: what is the cheapest hardware that can run a neural text-to-speech model locally, faster than real time?

To find out I ported tiny-tts to C++ as TinyTTS. It is a VITS-style, end-to-end model with only ~1.6 million parameters, and the port is plain hand-written C++ with no inference runtime (no TFLite, no ONNX Runtime). That makes it easy to run the exact same code on very different hardware.

Here is what “Hello world!” sounds like: ▶️ hello_world.wav

Microcontrollers: enough memory, not enough speed

My first target was the ESP32. Memory turned out not to be a problem at all: the model, a slimmed pronunciation dictionary and a small neural G2P model for unknown words need about 4.2 MB of flash, and only about 743 KB of PSRAM after startup.

Speed is a different story. Even after INT8 weight quantization, weight caching and SIMD dot products, the numbers for speak("Hello world!") (1.49 s of audio) are:

It works, and it is a nice proof of concept, but you would not want to wait 20 seconds for your doorbell to say “Hello”.

Where does the time go?

Profiling shows the same picture on every CPU: the decoder (a HiFi-GAN-style vocoder that turns the spectral representation into audio samples) takes 87-90% of the time. Most of the rest goes to the flow. The text encoder, the duration predictor and the grapheme-to-phoneme conversion are cheap.

So the question is not “which CPU is fast enough?” but “which cheap board has something that can do convolutions in parallel?” That points to a GPU or an NPU.

The winner: Milk-V Duo 256M

The Milk-V Duo 256M is a tiny Linux board around the size of a Raspberry Pi Pico. It sells for roughly ten dollars, and its SG2002 SoC has:

• a 1 GHz T-Head C906 RISC-V core running Linux
• 256 MB of RAM
• a TPU rated at 1 TOPS INT8, which also does BF16

On the CPU alone it is already better than the ESP32s, but still far from real time. Moving the flow and the decoder to the TPU changes everything:

That is about 2.4x faster than real time, including program startup and loading five models, on a board that costs about as much as a pizza.

How it works

The TPU is programmed with Sophgo’s TPU-MLIR compiler, which only accepts models with static shapes. Speech has a different length for every sentence, so I had to work around that:

• Decoder: it is purely convolutional, so it runs in overlapping fixed-size windows of 128 frames, with 12 frames of context on each side. The stitched result is practically identical to running the whole sentence at once (108 dB SNR in the ONNX check).
• Flow: it uses attention over the whole sentence, so windowing is not possible. Instead I compile four versions for 128, 256, 512 and 1,024 frames and pad the input to the next size. 1,024 frames is about 12 s of speech; longer sentences fall back to the CPU.
• Precision: the models run in BF16, not INT8. INT8 would be faster in theory, but the decoder’s output turned into noise, and even mixed precision with 47 layers kept in BF16 could not fix it. BF16 is as close to float as the CPU version.

The small, variable-length stages (G2P, text encoder, duration predictor) stay on the CPU. TinyTTS has setFlow() and setVocoder() hooks so that an external implementation can take over these two stages; the Milk-V build plugs the TPU in there.

The workflow is:

1. On the PC, export the decoder and the flow as ONNX.
2. Compile them with TPU-MLIR into .cvimodel files.
3. Cross-compile the tinytts command line program with the T-Head musl toolchain.
4. Copy the program and the models to the board with scp and run it:

export LD_LIBRARY_PATH=/mnt/system/lib
./tinytts --tpu-model decoder_44100hz_w128_bf16.cvimodel \
  --tpu-flow flow_44100hz_l128_bf16.cvimodel --tpu-flow flow_44100hz_l256_bf16.cvimodel \
  --tpu-flow flow_44100hz_l512_bf16.cvimodel --tpu-flow flow_44100hz_l1024_bf16.cvimodel \
  -o hello.wav "Hello world!"

All the details are in docs/milkv-duo.md.

Getting sound out of a $10 board

The Duo has no usable audio output with the default image. There are a few options:

• Play it on your PC. tinytts --stdout writes a WAV to standard output, so you can run it on the board over SSH and play it locally:

  ssh root@192.168.42.1 './tinytts --stdout "Hello world!"' | aplay
  

  The duo-say.sh script in the repository wraps this up with the TPU. A 2.6 s sentence takes about 1 s from command to WAV, SSH included.

• An I2S amplifier (MAX98357A). None of the header pins offers I2S, but I2S2 is available on the Ethernet pads. With a rebuilt kernel (scripts and patches are in research/milkv_duo/i2s/), the board gets a sound card and tinytts plays on it directly at 16 and 44.1 kHz. The sound card and playback work; I still have to confirm the actual sound on a connected amplifier, so treat the wiring as unverified for now.
• A USB speaker. This works too, but the Duo has only one USB-C port, so you lose the USB network and SSH.
• The built-in DAC and PWM are not really usable for speech.

The runner-up: Raspberry Pi Zero W

If you already have a Raspberry Pi Zero W in a drawer, it can do the same trick with its VideoCore IV GPU. With hand-written QPU kernels for the flow and the decoder, “Hello world!” drops from 15.7 s on the CPU to 0.58 s (about 1.0 s with startup), also faster than real time. See docs/rpi-zero-gpu.md.

The Milk-V Duo wins on price, and its toolchain (ONNX in, compiled model out) is a lot more pleasant than writing GPU assembly.

Summary

My conclusions:

• Memory is no longer the problem. A small neural TTS model fits easily into a microcontroller with PSRAM.
• CPUs in this price class are too slow, because the vocoder dominates everything.
• A small accelerator is enough. A few dollars’ worth of TPU or an old mobile GPU turns “23x slower than real time” into “faster than real time”.

So the cheapest way to run a neural TTS today is a Milk-V Duo 256M with the flow and the decoder on its TPU, at about ten dollars for the board.

The code, the models and the documentation are on GitHub: pschatzmann/TinyTTS. You can also train your own voice, or a model at a lower sample rate (see docs/training.md).