AudioTools: The VideoPlayer

A follow-up to AudioTools video support, which covers container/codec selection (AVI vs. MP4 vs. MPG, MJPEG vs. H.264). This post is the other half: the pipeline that actually plays a file, a minimal sketch, and every class involved.

Why this exists

On a desktop, “play this video” hands the problem to an OS: a media framework demuxes the container, decodes on a scheduler-managed thread, and a compositor times frames to the display’s refresh. An ESP32 has none of that. Reading the container, decoding a frame, and pushing pixels to a panel are three jobs competing for the same loop, and if you decode as fast as the CPU allows, frames arrive whenever they’re ready — not when the video’s own frame rate says they should.

VideoPlayer (AudioTools/Video/VideoPlayer.h) is the video counterpart of this library’s AudioPlayer: one object that owns a container demuxer, a codec decoder, a background-task frame queue that paces output to real time, and — optionally — an audio decode chain kept in sync against it. Nothing is registered by default; you wire in exactly the container and codec your content uses, which keeps a sketch that only plays Motion-JPEG AVI files from pulling in an H.264 decoder it will never call.

The pipeline

One copy() call per loop() iteration reads a chunk from your source and feeds it through the demuxer, which splits it into a video track and an optional audio track. Video is queued and paced by a dedicated task; audio, if present, becomes the clock the video schedule is checked against.

Stream&  --->  Demuxer
                 │
        ┌────────┴─────────┐
        │                  │
   video track         audio track (optional)
        │                  │
PacedVideoOutput      AudioDecoder (MP3/AAC/WAV)
   (queue + task)           │
        │            AudioTimeSourceStream (the clock)
   VideoDecoder              │
 (H264/MJPEG/MPG)      AudioOutput (I2S/board)
        │
   VideoOutput
  (panel/GPU)

The audio branch doesn’t just play sound — by default it becomes the clock the video branch schedules against, so video stays locked to audio even when the two decode at slightly different rates. Drop the audio output entirely and VideoPlayer falls back to a free-running timer, which is all a silent or video-only file needs.

Minimal example

Video-only playback of a Motion-JPEG AVI from an SD card to a TinyGPU-driven panel, no audio track involved. This is close to the smallest sketch that plays a real file: one demuxer, one decoder, one output, wired through the two-argument VideoPlayer constructor.

#include "AudioTools.h"
#include "AudioTools/AudioCodecs/ContainerAVI.h"
#include "AudioTools/Video/CodecJPEG.h"
#include "AudioTools/Video/OutputTinyGPU.h"
#include "AudioTools/Video/VideoPlayer.h"
#include "TinyGPU/Boards.h"
#include <SD_MMC.h>

// container, panel, and the player that wires them together
DemuxerAVI aviDemuxer;
LCDBoardESP32S3_2_8Display board;
OutputTinyGPU tftOutput(board);
VideoPlayer player(aviDemuxer, tftOutput);   // video only: no audio arg
MJPEGDecoder mjpegDecoder;
File file;

const char* path = "/Videos/clip176x144-mjpeg.avi";

void setup() {
  Serial.begin(115200);
  SD_MMC.begin();
  board.begin();
  tftOutput.setScaleToFit(true);   // upscale to fill the panel

  player.addVideoDecoder(mjpegDecoder);  // registered by codec fourcc

  file = SD_MMC.open(path);
  if (!file || !player.begin(file)) {
    Serial.println("could not start playback");
    while (true) delay(1000);
  }
}

void loop() {
  if (player.copy() == 0) {      // 0 == source exhausted
    file.close();
    while (true) delay(1000);
  }
}
Code language: PHP (php)

Adding audio: pass a third argument — an AudioOutput, AudioStream, or plain Print — to the VideoPlayer constructor, register an audio decoder with addAudioDecoder(decoder, mime), and playback locks video to that output’s own audio clock automatically. Call setUseAudioClock(false) only if the track never actually delivers bytes (a silent or empty stream would otherwise stall the video schedule forever).

Classes and dependencies

Every piece below lives under AudioTools/Video/ or AudioTools/AudioCodecs/ and is opt-in — VideoPlayer starts empty; VideoPlayerFull pre-registers the whole set at the cost of linking every dependency at once.

Pipeline core

Class	Role	Header	External dependency
VideoPlayer	Owns demuxer + decoder + sync task; one copy() per loop	Video/VideoPlayer.h	none
VideoPlayerFull	VideoPlayer with every demuxer/codec pre-registered	Video/VideoPlayerFull.h	everything below, unconditionally
PacedVideoOutput	Frame queue + background task that paces decode to real time, drops/resyncs on backlog	Video/PacedVideoOutput.h	FreeRTOS task/queue
AudioTimeSourceStream	Wraps the audio output as the clock video is scheduled against	CoreAudio/AudioIO.h	none
EncodedAudioStream	Audio decode chain feeding the real audio output	AudioCodecs/AudioEncoded.h	none

Containers (demux)

Class	Role	Header	External dependency
Demuxer (interface)	Container parsing contract; exposes video/audio tracks + mime	AudioCodecs/ContainerCommon.h	none
DemuxerAVI / MuxerAVI	AVI (RIFF) container — audio codec may be PCM/MP3/AAC	AudioCodecs/ContainerAVI.h	none
DemuxerMP4 / MuxerMP4	MP4/MOV (ISO BMFF) container	AudioCodecs/ContainerMP4.h	none
DemuxerMPG	MPEG program stream container	AudioCodecs/ContainerMPG.h	none
MultiVideoDemuxer	Auto-selects a registered demuxer by mime type	Video/MultiVideoDemuxer.h	none

Video codecs (decode)

Class	Role	Header	External dependency
VideoDecoder (interface)	Decode contract; extends VideoOutput so decoders chain directly to a display	Video/CodecVideo.h	none
MJPEGDecoder	Motion-JPEG, every frame independently decodable	Video/CodecJPEG.h	TinyJPEG (pschatzmann/TinyJPEG)
H264Decoder	Software H.264 decode	Video/CodecH264.h	external H.264 software decoder
H264DecoderESP32S3	Hardware-assisted H.264 decode on ESP32-S3	Video/CodecH264ESP32S3.h	ESP32-S3 HW codec (ESP-IDF)
H264EncoderESP32S3 / H264EncoderESP32P4	Hardware H.264 encode (camera → stream use cases)	Video/CodecH264ESP32{S3,P4}.h	ESP32-S3/P4 HW codec (ESP-IDF)
MPGDecoder / MPGEncoder	MPEG-1 video decode/encode	Video/CodecMPG.h	external MPEG-1 codec
MultiVideoDecoder	Auto-selects a registered decoder by codec fourcc	Video/MultiVideoDecoder.h	none

Display outputs

Class	Role	Header	External dependency
VideoOutput (interface)	Display-sink contract every decoder/output implements	Video/VideoOutput.h	none
OutputTinyGPU	GPU-accelerated scale/blit to a supported panel	Video/OutputTinyGPU.h	TinyGPU (pschatzmann/TinyGPU)
OutputTFT_eSPI	Renders through the TFT_eSPI panel driver	Video/OutputTFT_eSPI.h	Bodmer/TFT_eSPI
OutputOpenCV	Preview window for desktop/simulator builds	Video/OutputOpenCV.h	OpenCV

Audio side (optional track)

Class	Role	Header	External dependency
MultiDecoder	Auto-selects a registered audio decoder by mime	AudioCodecs/MultiDecoder.h	none
MP3DecoderHelix / AACDecoderHelix / WAVDecoder	Common AVI/MP4 audio tracks	AudioCodecs/CodecMP3Helix.h etc.	libhelix (bundled)
AudioBoardStream / I2SStream	Final audio sink (speaker/DAC)	AudioLibs / CoreAudio	arduino-audio-driver (board-dependent)

Tuning knobs worth knowing about

PacedVideoOutput does the actual scheduling, and VideoPlayer forwards its controls: setSchedulingDelayMs() compensates for your audio output’s own buffering latency (audio “plays” the instant it’s written, not when the DAC emits it), setTaskParameters() pins the render task to its own CPU core so a slow frame never stalls the loop doing SD reads and demuxing, and setResyncThresholdMs() / setMaxQueuedIFrames() decide how playback recovers once it falls behind — dropping frames slows a backlog’s growth, but only a forward jump in the schedule actually clears one.

Transcoding Videos

Please note that none of this plays raw camera footage — everything needs to be downscaled and re-encoded first. `176×144` at 7–8 fps is the practical baseline; going higher roughly triples decode+render cost per frame.

In additioin, PSRAM is required, not optional — decoded-picture buffers and the frame queue don’t fit in internal SRAM. Full ffmpeg variants and measured fps per codec can be found in the Video Playback wiki page.

---

AudioTools/Video · github.com/pschatzmann/arduino-audio-tools