Introducing the FPGA Drive Recorder

In this post I’m going to introduce our new reference design, the FPGA Drive Recorder. It records a data stream generated in the FPGA fabric to a file on one or two NVMe SSDs, and it can play those recordings back into the fabric. The SSDs connect to the FPGA through our FPGA Drive FMC Gen4 (OP063) or M.2 M-key Stack FMC (OP073), and the design runs embedded Linux built with the AMD Yocto / Embedded Development Framework (EDF) 2025.2.

Over the years, a lot of customers have asked me for something like this. Our fpga-drive-aximm-pcie reference design gets the SSDs working as Linux block devices, but it doesn’t get your data from the fabric onto them, and I never had a working design that I could hand over for that. Almost two years ago, at the end of the Versal PL NVMe SSD speed test post, I wrote that I wanted to build something closer to a real application: generate samples in the PL, store them on the SSD, and validate the data. This is that design, and I’m really happy to finally deliver it.

This post covers what the design does, where the line sits between the FPGA and Linux, the performance we measured, and some use cases. For the fine detail (register map, driver, file format, full benchmark tables) see the user guide.

What it does

Linux runs on the processor of the target board, and it owns the SSDs and the filesystem. The recording is zero-copy:

• An AMD AXI DMA in the fabric writes the data stream into a ring of buffers in the processor’s DDR memory. The buffers are 2 MB hugepages that the recorder app allocates and the fdrec kernel driver pins for the DMA (the default ring is 32 x 8 MB).
• When a buffer is full, the app writes it to a file with an O_DIRECT write, and the NVMe controller of the SSD reads the data straight out of that same buffer.
• The CPU never reads, writes or copies the data. It only does the bookkeeping: DMA descriptors, cache maintenance and the I/O requests.

The data source is a block-design hierarchy called user_data_source with a 128-bit AXI4-Stream output. In the reference design it holds a test pattern generator that models an ADC: it produces data at a programmable rate and it cannot be stalled. That’s the point where you drop in your own ADC, sensor or processing logic. If the SSDs can’t keep up, the data that doesn’t fit is dropped and counted in the fabric. It’s never silently lost: every recording reports its drop count in the file header, and a valid recording has a drop count of zero. The fdverify tool then checks a test-pattern recording beat by beat.

Playback is the same path in reverse. The app reads a recording into the hugepage buffers with O_DIRECT reads, and the DMA streams them into a second hierarchy, user_data_sink. In the reference design that’s a hardware checker that consumes the stream at a programmable rate, like a DAC would, and verifies every beat. You can replace it with your own logic too.

The image below shows the Zynq UltraScale+ version of the design.

The PCIe part of the design (one PCIe Root Port per M.2 slot) is the design of fpga-drive-aximm-pcie. The recorder adds the data source and sink, the fdrec_core FIFOs and registers, and the AXI DMA. The datapath is 128 bits wide at 250 MHz (dp_clk) on the DMA side, which is 4 GB/s, and the source and sink run at 200 MHz (src_clk), so the test pattern generator tops out at 3.2 GB/s. The Versal version uses exactly the same datapath; only the platform around it changes (CIPS, the NoC and QDMA Root Ports).

The Linux image comes with these tools:

Recordings are ordinary files: a 4096-byte header (with the start and stop times and the drop count) followed by the raw 16-byte beats (see the file format).

What it is not: complete NVMe offload

I want to be clear about this up front: this design does not implement complete NVMe offload. It is not an NVMe host engine in the fabric. Here is where the line sits:

• In the PL: your data source and sink, the FIFOs, the AXI DMA that moves data between the fabric and DDR, and the PCIe Root Ports (the integrated PCIe blocks in AXI Bridge mode).
• In the PS, under Linux: everything NVMe. The standard Linux NVMe driver, the block layer and the filesystem (XFS by default) handle every command, queue and completion. The recorder app issues the writes with io_uring, and the fdrec driver manages the DMA descriptors and the cache maintenance.
• In the SSD: its NVMe controller DMAs the data directly out of (or into) the DDR buffers through the Root Port. That’s what makes it zero-copy.

The diagram below, from the user guide, shows who touches each buffer and when.

Keeping Linux in charge has real advantages: recordings are normal files, you can use RAID0, fio, nvme-cli and every other Linux tool, and the design needs no paid IP. The trade-off is that you need a running Linux system, and the rate you can sustain is set by the SSDs and by the Linux I/O path.

If you need CPU-less operation, or rates beyond what Linux can sustain, then you need the NVMe host in hardware. Opsero is partnered with Missing Link Electronics (MLE), and their NVMe Streamer IP core does complete NVMe offload: the NVMe protocol is handled in the fabric rather than by a processor running Linux. The FPGA Drive Recorder is a free, open-source starting point for the many applications that fit within what Linux can do, and NVMe Streamer is for the ones that don’t.

Performance

All of these figures were measured on hardware on 2-3 October 2026 with two SSDs, a Samsung 970 EVO 250GB and a Samsung 950 PRO 256GB, both linked at PCIe Gen3 x4, used alone and as a RAID0 pair (512 KB chunks, XFS). The boards were the UltraZed-EV Carrier (four Cortex-A53 cores at 1.1 GHz), the AMD ZCU106 (four Cortex-A53 cores at 1.2 GHz) and the AMD VCK190 (two Cortex-A72 cores at 1.4 GHz). The benchmarks page has every measurement and the method.

The pass criterion is stricter than a typical SSD benchmark: a recording rate passes only if fdrec drops zero beats over the whole recording and fdverify reports a clean file. The source can’t be stalled, so a pass means the whole chain (DMA, driver, filesystem, SSD) kept up from the first byte to the last. Playback is clean when the hardware checker sees no underflows and no errors.

Measured vs. the manufacturers’ ratings

Here’s how the recorder’s results compare to what Samsung claims for these drives. “Burst” is the highest clean rate for a 4000 MiB recording, which fits inside the 970 EVO’s SLC write cache. “Sustained” is the highest clean rate for a 32 GB recording, found with fdbench.sh in 250 MB/s steps. Playback is the highest clean fdplay rate for 4000 MiB (that sweep stops at 3000, the last step below the 3.2 GB/s sink maximum). All figures are in MB/s.

A few things jump out:

• Burst writes hit the rating. The recorder records cleanly at 1500 MB/s to the 970 EVO on all three boards, which is the drive’s rated write speed. The 950 PRO actually writes at about 955 MB/s, slightly above its rating.
• Sustained writes are much lower. For long recordings the 970 EVO sustains 250 MB/s (300 MB/s passed over 40 GB), the 950 PRO 750 MB/s (800 MB/s passed over 40 GB), and the RAID0 pair 500 MB/s.
• Reads depend on the host. On the two Zynq UltraScale+ boards, playback tops out at about 1.5 GB/s per drive. The VCK190 reads the same drives 2-3 times faster, and in 10 GB playbacks the RAID0 pair is clean at 3.2 GB/s, the maximum rate of the sink in this design.
• The sustained rates are identical on all three boards (250 / 750 / 500 MB/s). They are set by the SSDs, not by the FPGA or the processor.

Why you don’t get the rated speed

The docs have a whole section on where the bottlenecks are, but here’s the short version, in the order the limits usually bite.

1. The SSD’s SLC cache. Most consumer SSDs write into a fast pseudo-SLC cache first and move the data to the slower TLC flash later. While the cache lasts, the drive writes at its datasheet rate; once it’s full, the rate drops. The 970 EVO 250GB writes at up to ~1.5 GB/s for about 13-14 GB on an empty, trimmed drive, then at about 330 MB/s. Samsung is upfront about this (300 MB/s after TurboWrite), but it’s easy to miss. A recorder writes at a constant rate for as long as the recording lasts, so the post-cache rate is the one that counts. The 950 PRO has no comparable cliff, which is why its sustained rate is so much closer to its burst rate.

2. The drive’s worst moment, not its average. A recording passes only if no beat is dropped over its whole length, so the rate is set by the drive’s slowest stretch: garbage collection, the cache running out, a write-latency spike. That’s why the sustained recording rates are below every average throughput figure on the benchmarks page, fio included. If you test a drive with a short dd or fio run, you’re measuring its cache and overstating what it can sustain.

3. The Linux I/O path and the CPU. Even without a data copy, every write goes through the filesystem, the block layer and the NVMe driver, and that costs CPU time. On the UltraZed-EV, fdrec uses about 2.5-3 % of one 1.1 GHz Cortex-A53 core per 100 MB/s, almost all of it system time. That’s about half the CPU of a dd write, which has to produce every byte and runs one request at a time. Reads are where the host really shows: on the Zynq UltraScale+ boards the read path limits playback to roughly 1.3-1.6 GB/s per drive, while the VCK190’s Cortex-A72 cores read the same drives close to or above their ratings with fio (3217 MB/s from the 970 EVO).

4. NVMe queues and RAID0. On every board we tried, the Linux NVMe driver set up the 970 EVO 250GB with a single I/O queue (the 950 PRO got two to four), so all of its completions are handled by one core. RAID0 stripes every chunk across both drives in turn, so the array runs at twice the slower drive’s rate. With the 970 EVO in the pair, that’s why the RAID0 sustains 500 MB/s: twice the 970 EVO’s 250 MB/s.

What is not the bottleneck at these rates: the PCIe link (Gen3 x4 carries about 3.3 GB/s of payload, Gen4 x4 about 6.5 GB/s), the PS DDR, and the fabric datapath (4 GB/s). I speculated about the PCIe link in earlier posts, so it’s nice to have that settled with data.

To get more, use drives designed for sustained writes (enterprise or industrial, MLC or SLC flash, or larger capacities with a bigger cache), RAID0 of two identical drives, and SSDs that expose several NVMe I/O queues. Also watch the heat: many SSDs throttle when they get hot during long recordings.

What you can do with that throughput

Even the most conservative figure, 250 MB/s sustained with a budget consumer drive, opens up a lot of applications, and the better drive or a RAID0 pair raises that to 500-750 MB/s:

• ADC capture. 750 MB/s sustained is, for example, a continuous stream of 16-bit samples at 375 MSPS, or two 16-bit channels at 187.5 MSPS, for as long as the drive has space. At that rate a 2 TB drive fills in about 44 minutes.
• Short, fast bursts. If your event fits in the SSD’s cache, you can record bursts at 1.5 GB/s to a single drive, or up to about 2 GB/s to the RAID0 pair.
• Waveform playback. Reads have no cache cliff, so playback rates hold for long files. A DAC or a transmitter in user_data_sink can be fed at 1.5 GB/s from a single drive on Zynq UltraScale+, and at up to 3.2 GB/s from a RAID0 pair on the VCK190.
• Network capture. A fully loaded 1G Ethernet link carries 125 MB/s in each direction, well within the sustained rate of even the 970 EVO. More on that below.

Supported boards

At the time of writing, the design supports these target boards:

The VCK190 needs a Vivado Enterprise license because the XCVC1902 device isn’t supported by the free Standard Edition. That’s a device license; no design needs an IP license. More Linux-capable Zynq UltraScale+ and Versal targets of fpga-drive-aximm-pcie are being added.

Requirements

• Vivado 2025.2 and Vitis 2025.2 (Vitis provides sdtgen/xsct for the Yocto flow)
• A native Linux machine for the Yocto / EDF 2025.2 build (the Vivado part also builds on Windows)
• FPGA Drive FMC Gen4 (OP063) or M.2 M-key Stack FMC (OP073)
• One or two M.2 NVMe PCIe SSDs
• One of the supported boards above, with its USB-UART cable, and a micro-SD card of 8 GB or larger

There’s no PetaLinux flow (PetaLinux is being retired, see my Yocto/EDF migration post) and no bare-metal application. The recorder needs Linux.

Build and run

Clone the repo (with its submodules) and build everything for your target. The example below is for the UltraZed-EV; use zcu106_hpc0 or vck190_fmcp1 for the other boards. This builds the Vivado project and XSA, then the Yocto image, and gathers the boot files into bootimages/:

git clone --recursive https://github.com/fpgadeveloper/fpga-drive-recorder.git
cd fpga-drive-recorder
./build.sh all --target uzev

Flash Yocto/uzev/images/linux/rootfs.wic.xz to the SD card, copy BOOT.BIN onto its first (FAT) partition, and boot the board with one or two SSDs on the mezzanine card. Log in as amd-edf (you choose the password on the first login). Then:

# 1. Make a filesystem on the SSD(s) and mount it at /mnt/rec -- ERASES THE DRIVES.
#    Both M.2 slots populated: RAID0 across the two SSDs
sudo fdsetup-raid0.sh
#    or a single SSD:
sudo fdsetup-raid0.sh --single /dev/nvme0n1

# 2. Record 10 GB of the test pattern at 1 GB/s (exit code 0 = no dropped beats)
sudo fdrec --rate 1000MB --size 10G /mnt/rec/test.dat

# 3. Verify every beat of the recording
sudo fdverify /mnt/rec/test.dat

# 4. Play it back into the fabric, where the hardware checker consumes it at 1 GB/s
sudo fdplay --rate 1000MB /mnt/rec/test.dat

# Find the sustained recording rate of your SSD(s): 32 GB per step
sudo fdbench.sh

Run fdbench.sh with your own SSDs before you design around a rate. As the table above shows, the drive matters far more than the board. The test procedure in the user guide shows the expected output of each step.

To record your own data, replace the test pattern generator in user_data_source (see custom data source) and run fdrec --no-tpg. Keep in mind the contract: the source is never stalled, the stream is 128 bits wide, and the recorder adds no framing or timestamps to your data. If you need timestamps or frame markers, put them into the stream in your source logic.

Use cases

Here are a few applications that I think are a good fit for this design:

• Data acquisition. Record ADC or sensor data continuously, for later analysis on a PC. The drop count tells you whether anything was lost.
• Record and replay for testing. Capture a real-world signal once, then play it back into your processing logic at the original rate, as often as you like, while you develop and debug.
• Arbitrary waveform playback. Stream waveforms from the SSD into a DAC or transmitter, far longer than you could hold in DDR.
• Logging a processing pipeline. The source can be anything in the fabric that produces a stream, not just an ADC.

What I’m building with it: an Ethernet packet recorder

I’m currently using the FPGA Drive Recorder to build an Ethernet packet recorder. It’s a work in progress, but here’s the idea. The recorder taps a 1G Ethernet link and stores the packets directly to the NVMe drive, as a file that you can open in Wireshark. It’s all controlled through a web interface.

The hardware is an AMD ZCU106 with the FPGA Drive FMC Gen4 on HPC1 and the Ethernet FMC Max (OP080) on HPC0. The ZCU106’s on-board Ethernet port is used for the web interface, so the control traffic never shares a port with the link being tapped.

Because every packet is timestamped as it’s captured, the recorder can also do the reverse: it can replay Ethernet traffic that was recorded earlier, with the exact original timing. That maps nicely onto the two halves of this design, with the capture logic as the data source and the replay logic as the data sink. I see a lot of uses for that in testing network equipment: record a problem once, then reproduce it on the bench as many times as you need.

I’ll write more about it when it’s ready.

Wrap up

The FPGA Drive Recorder gives you a free, open-source, zero-copy path from the FPGA fabric to a file on an NVMe SSD and back, with Linux in charge of the drives. It’s not a hardware NVMe host, and it doesn’t try to be one. If you need complete NVMe offload, look at MLE’s NVMe Streamer. But for a lot of applications, the rates that Linux and a good SSD can sustain are plenty.

• Source code: FPGA Drive Recorder on GitHub
• User guide: recorder.fpgadrive.com
• Hardware: FPGA Drive FMC Gen4 (OP063), available from the Opsero store, or the M.2 M-key Stack FMC (OP073)

If you build something with it, test it with SSDs that aren’t on our list, or get it working on another board, I’d really like to hear about it. And if you have any questions, feel free to reach out.