TinySTT: Speech-to-Text on a Microcontroller – a Proof of Concept

If you look for speech recognition examples for microcontrollers, you almost always end up with wake word detection or keyword spotting: the micro_speech example of TensorFlow Lite Micro, which recognizes “yes” and “no”; Espressif’s ESP-SR with WakeNet for the wake word and MultiNet for a list of commands; keyword spotting models trained with Edge Impulse; or Picovoice’s Porcupine and Rhino. They work well for what they do, but they all classify a small, fixed set Read more

By pschatzmann, ago

Arduino Audio Tools: Streaming Video and Audio via HTTP

If you have ever streamed video from an ESP32, you have most likely used the CameraWebServer example: the camera delivers JPEG images, which the web server pushes one after the other to the browser. This works, but there is no sound, the bandwidth is high, and the result is not a real video stream that you could record or play in VLC. A real stream needs a video codec such as H.264, an audio codec Read more

By pschatzmann, ago

The Cheapest Way to Run a Neural TTS

Classic embedded text-to-speech (formant synthesizers like SAM or eSpeak) is tiny and fast, but it sounds like a robot from the 80s. Neural TTS sounds natural, but it is usually something you run on a PC or in the cloud. I wanted to know: what is the cheapest hardware that can run a neural text-to-speech model locally, faster than real time? To find out I ported tiny-tts to C++ as TinyTTS. It is a VITS-style, Read more

By pschatzmann, ago

Neural TTS on a Microcontroller ?

To produce the most natural-sounding speech, modern Text-To-Speach (TTS) systems rely on neural networks. TinyTTS is a proof of concept: It asks whether a VITS-style neural TTS model can run on a microcontroller at all. It is a header-only C++ port of tiny-tts. The model has about 1.6M parameters. It has four stages: text encoder, flow, duration predictor and a HiFi-GAN-style vocoder. There is no inference runtime such as TFLite Micro. Everything is hand-written C++ Read more

By pschatzmann, ago

AudioTools: The VideoPlayer

A follow-up to AudioTools video support, which covers container/codec selection (AVI vs. MP4 vs. MPG, MJPEG vs. H.264). This post is the other half: the pipeline that actually plays a file, a minimal sketch, and every class involved. Why this exists On a desktop, “play this video” hands the problem to an OS: a media framework demuxes the container, decodes on a scheduler-managed thread, and a compositor times frames to the display’s refresh. An ESP32 Read more

By pschatzmann, ago

AudioTools: Supporting Videos

I added video support to my AudioTools library: Different container formats can be used to store video on audio together in the same file. Here is a quick overivew of the provided functionality that is relevant for Microcontrollers: Container Formats The simplest and therefore most recommended container format is AVI (Audio Video Interleave). It is simple to both encode and decode and you can easily mix audio and video together. It supports both different audio Read more

By pschatzmann, ago

Arduino LCD TFT Boards and Material Design

Material Design is an open-source design system created by Google in 2014 to help developers and designers build consistent, high-quality digital interfaces across Android, iOS, the web, and other platforms. I did not find any existing Arduino Library which would allow me to easily implement some Material Design Applications on Arduino, so I decided to implement my own TinyMaterialDesign library. I also added a comprehensive tutorial which explains the library in detail. Further information can Read more

By pschatzmann, ago

LCD Boards with Audio

I started to work on extending the AudioTools library to support different video formats. There are quite a few boards with LDC display that have a built in support for audio output. The cool thing is that they also provide an SD drive, so that you have plenty of room to store audio and videos. In a first step, I tested the audio and the LCD output. Here is an overview of the boards that Read more

By pschatzmann, ago

Receiving Audio via USB: a Deep Dive

In one of my last blogs, I gave an overview of the USB Class 2 Audio functionality provided by the AudioTools library Many users were struggeling to get the process of receiving data via USB and writing to I2S stable and working properly. In this blog I am providing a deep dive of the releated challenges and the solution approaches: Standard Processing Logic Here is a simple basic sketch that demonstrates the standard processing logic: Read more

By pschatzmann, ago