Walking an Entire Arduino SD Card Directory Tree: A Performance Guide

I was searching for the most efficient generic way to process the whole SD directory tree in Arduino: this can be helpful if you want to search for files or if you want to store the whole file system in a tree in RAM.

The Standard SD API Way

When using the API provided by the SD library, we need to open each file/directory to find it’s children. So to process all files in the directory tree we can use a recursive function call:

#include <SPI.h>
#include <SD.h>

#define PIN_AUDIO_KIT_SD_CARD_CS 13
#define PIN_AUDIO_KIT_SD_CARD_MISO 2
#define PIN_AUDIO_KIT_SD_CARD_MOSI 15
#define PIN_AUDIO_KIT_SD_CARD_CLK 14

int count = 0;

class NullPrintCls : public Print {
  size_t write(uint8_t c) override {
    return 1;
  }
} NullPrint;

void printDirectory(Print& out, File dir, int level) {
  while (true) {
    File entry = dir.openNextFile();
    if (!entry) {
      // No more files
      break;
    }

    // Print indentation
    for (int i = 0; i < level; i++) {
      out.print("  ");
    }

    // Print name
    out.print(entry.name());
    count++;

    if (entry.isDirectory()) {
      out.println("/");
      printDirectory(out, entry, level);
    } 
    entry.close();
  }
}

void setup() {
  Serial.begin(115200);
  delay(2000);

  Serial.println("Initializing SD card...");
  SPI.begin(PIN_AUDIO_KIT_SD_CARD_CLK, PIN_AUDIO_KIT_SD_CARD_MISO, PIN_AUDIO_KIT_SD_CARD_MOSI, PIN_AUDIO_KIT_SD_CARD_CS);

  if (!SD.begin(PIN_AUDIO_KIT_SD_CARD_CS)) {
    Serial.println("SD card initialization failed!");
    return;
  }
  Serial.println("SD card initialized.\n");

  auto start_ms = millis();
  File root = SD.open("/");
  printDirectory(NullPrint, root, 0);
  root.close();

  Serial.print("Runtime in ms: ");
  Serial.println(millis() - start_ms);
  Serial.print("Number of files: ");
  Serial.println(count);
}

void loop() {}
Code language: HTML, XML (xml)

I am not interested in timing the output, so I use a custom NullPrint object which does nothing. If you want to print the result, just replace the NullPrint with Serial. Here is the result:

09:33:16.645 -> Runtime in ms: 95170
09:33:16.645 -> Number of files: 2208
Code language: CSS (css)

This result is, as expected, really bad since we are forced to open each file just to process it’s directory information. Please note that in some Arduino variants the stack is quite limited: so you might need to replace this simple recursive approach with your own stack logic which is using the heap.

You can improve the processing time by using a more efficient API compatible library: e.g. on the ESP32 you can use SDMMC: however the overall handicap coming from the inefficient directory functions remains!

Using the fatfs library

In the next approach, I am using the arduino-fatfs library. It is providing a recursive_directory_iterator which is using the dedicated directory API provided by the fatfs library. So this should be much more efficient. This method provides you the file including the full path, so we need to do some additional logic to extract the file name w/o path and do the indentation.

Here is the Arduino sketch:

#include <SPI.h>
#include "fatfs.h"
#include "filesystem.h"

// pins for SD card
#define MISO 2
#define MOSI 15
#define SCLK 14
#define CS 13

int count = 0;

class NullPrintCls : public Print {
  size_t write(uint8_t c) override {
    return 1;
  }
} NullPrint;

// Helper function to count directory depth for indentation
int getDepth(const std::string& path) {
  int depth = 0;
  for (char c : path) {
    if (c == '/') depth++;
  }
  return depth;
}

void printDirectory(Print& out) {
  // Use recursive_directory_iterator - it automatically traverses subdirectories!
  for (auto it = recursive_directory_iterator("/");
       it != recursive_directory_iterator::end();
       ++it) {
    auto entry = *it;
    count++;

    // Calculate indentation based on path depth
    int depth = getDepth(entry.path);
    std::string indent;
    for (int i = 0; i < depth; i++) {
      indent += "  ";
    }

    // Extract filename from path
    const char* name = entry.path.c_str();
    const char* lastSlash = strrchr(name, '/');
    const char* filename = lastSlash ? lastSlash + 1 : name;

    out.print(indent.c_str());
    out.println(filename);
  }
}

void setup() {
  Serial.begin(115200);
  delay(3000);

  // Initialize hardware SPI for SD card
  Serial.print("Initializing SD card...\n");
  SPI.begin(SCLK, MISO, MOSI);
  delay(100);

  if (!SD.begin(CS, SPI)) {
    Serial.print("SD card initialization failed!\n");
    while (true)
      ;
  }

  auto start_ms = millis();
  count = 0;

  printDirectory(NullPrint);

  Serial.print("Runtime in ms: ");
  Serial.println(millis() - start_ms);
  Serial.print("Number of files: ");
  Serial.println(count);
}

void loop() {}
Code language: HTML, XML (xml)

This is, as expected, much faster and in real live we can get rid of the unneeded functionality to make it even more efficient. I kept the NullPrint logic, so that you can call the method with Serial to check the output.

09:28:30.608 -> Runtime in ms: 7301
09:28:30.609 -> Number of files: 2208
Code language: CSS (css)

Using ls() with the SdFat library

The SdFat library is usually much more efficient than the SD library. In addition it povides the ls() function that is incredible fast. The only challenge: the output goes to a Print object, so we need to write our own logic to capture the output and do some parsing.

Here is the complete sketch

#include <SPI.h>
#include "SdFat.h"
#include <string>

// pins for audiokit
#define MISO 2
#define MOSI 15
#define SCLK 14
#define CS 13

SdFs sd;
int count = 0;

// Dummy output
class NullPrintCls : public Print {
  size_t write(uint8_t c) override {
    return 1;
  }
} NullPrint;

// File info
struct SdFatFileInfo {
  std::string name;
  bool is_directory;
  int level;
};

// A simple parser that provides SdFatFileInfo via a callback
class SdFatParserCls : public Print {
public:
  SdFatParserCls() {
    name.reserve(80);
  }

  size_t write(uint8_t c) override {
    switch (c) {
      case '\n':
        parse();
        break;
      case '\t':
        break;
      default:
        name += c;
        break;
    }
    return 1;
  }

  void setCallback(void (*cb)(SdFatFileInfo&, void* ref), void* ref = nullptr) {
    this->ref = ref;
    this->cb = cb;
  }

protected:
  std::string name;
  void (*cb)(SdFatFileInfo&, void* ref);
  void* ref;
  SdFatFileInfo info;

  int spaceCount() {
    for (int j = 0; j < name.size(); j++) {
      if (name[j] != ' ') return j;
    }
    return 0;
  }

  void parse() {
    int spaces = spaceCount();
    info.level = spaces / 2;
    info.name = name.erase(0, spaces);
    info.is_directory = info.name[info.name.size() - 1] == '/';
    if (cb) cb(info, ref);
    name.clear();
  }
} SdFatParser;

// output the file
void processInfo(SdFatFileInfo& info, void* ref) {
  Print *p_out = (Print*) ref;
  count++;
  // indent
  for (int j = 0; j < info.level; j++) {
    p_out->print("  ");
  }
  // print name
  p_out->println(info.name.c_str());
}

void setup() {
  Serial.begin(115200);
  delay(3000);
  Serial.println("starting...");

  // start SPI and setup pins
  SPI.begin(SCLK, MISO, MOSI);

  if (!sd.begin(CS, SD_SCK_MHZ(4))) {
    Serial.println("sd.begin() failed");
  }

  // setup callback
  SdFatParser.setCallback(processInfo, &NullPrint);
  
  auto start_ms = millis();
  
  // process ls
  sd.ls(&SdFatParser, LS_R);
  
  Serial.print("Runtime in ms: ");
  Serial.println(millis() - start_ms);
  Serial.print("Number of files: ");
  Serial.println(count);
}

void loop() {}

The LS_R flag is specifying that we want to do the processing recursively. Here as well, you can replace the NullPrint with Serial if you want to print the result. The only disadvantage with this functionality is, that you can’t abort the processing and you are forced to process the complete directory tree.

This gives by far the best result:

11:26:45.407 -> Runtime in ms: 2396
11:26:45.407 -> Number of files: 2208
Code language: CSS (css)

Result Overview

I was running the test on an AI Thinker AudioKit which uses an ESP32 and provides a built in SD drive. Here is the final summary which is giving the times to process 2208 files and directories:

So using the SdFat library for this task seems to be the recommended approach!