VDO Under Proxmox Local LVM: Fewer Bytes Reach the Drive

Fewer Bytes Reach the Drive

VDO is the Linux kernel’s deduplication and compression layer. It sits under a volume group and works in 4 KiB blocks. Every block it has seen before, it stores once.

What that means for the hardware is simple. Fewer bytes reach the drive. I wrote a mix that was 75% duplicate and 50% compressible through VDO for 30 seconds at a time, and the drive’s own counters said it received 23 to 28% of what the guest wrote. Six full clones of a template disk added 8 data blocks between them. Thirty-two kilobytes.

Fewer bytes is less wear, and less work for any cache on the drive or the controller in front of it. It costs CPU and latency to get there, and on a fast drive that cost is measured below against the same Optane with no VDO at all. On slow spinning disks the same cost disappears into the seek times, which is where this gets interesting.

Two rules shape everything that follows. The stack is thick on thick: nothing above VDO, and VDO itself, can promise more space than the drive has. And VDO’s own metadata, its index and its journals, can live on NVMe while the data lives somewhere slower.

Every number here comes from one Proxmox VE 9.2 host, a Ryzen 7 5700G with 16 CPUs, kernel 7.0.14-25-pve, LVM 2.03.31 and the Debian vdo 8.3.1.1 tools, on two Intel Optane drives that were wiped afterwards.

What VDO Is and Where It Sits

The kernel describes dm-vdo as a device-mapper target that “provides block-level deduplication, compression, and thin provisioning”¹. Each incoming 4 KiB block is hashed. If an identical block is already stored, VDO records a reference instead of a copy. Blocks that are not duplicates go through LZ4 and get packed several to a physical block, and blocks of all zeroes are not stored at all².

LVM drives the whole thing. lvcreate --type vdo makes a VDO pool, which holds the real blocks, and a VDO LV on top of it with whatever virtual size you ask for². A thick volume group goes on the VDO LV, and Proxmox uses it as ordinary lvm storage.

Thick on thick, or not at all

VDO is a thin provisioner in its own right: the VDO LV can be given any virtual size. Make it bigger than the pool can hold and the volume group on top is thick on a promise, which is thin on thin moved down a layer. I did exactly that on the first build, a 120 GiB VDO LV on a 40 GiB pool, and Proxmox showed a third of the storage free while VDO was full.

So size the VDO LV to what the pool can actually address, no more. lvcreate prints that figure, “can address 36.00 GB” on this 40 GiB pool, and VDO’s block map for the logical space comes out of the same room, so I gave it 35 GiB.

Then I proved it the hard way. One VM disk taking every extent Proxmox would give, 34 GiB, filled end to end with random data that VDO can do nothing with:

# vdostats --human-readable
Device                    Size      Used Available Use% Space saving%
vdolab-vdopool-vpool     40.0G     38.1G      1.9G  95%            0%

The disk ran out at its own end, as a thick disk should. VDO still had 1.9 GiB free, more than the 1 GiB Proxmox had left to allocate. Proxmox’s figure is the real limit, and VDO cannot run out first.

What you give up is the overcommit: the space deduplication saves shows up as drive that never gets written, not as extra disks for Proxmox. That is the right trade for this post, which is about the hardware. And do not put an lvmthin pool on it either. Two thin layers is two sets of books³.

What It Costs Before You Store a Byte

VDO keeps a deduplication index, the UDS index, on the pool itself. An empty 40 GiB pool reported 4.0 GiB used and 36.0 GiB available. vdolistmetadata, from the same vdo package, lists every metadata block on the device, and accounts for nearly all of it:

The leftover line is worth knowing: slabs are 2 GiB, so size the pool in whole slabs plus the overhead or the remainder is simply unused.

RAM is the other bill. It is paid whether the pool is busy or idle:

The default dense index covers a deduplication window of roughly 256 GiB, since a dense index with 1 GiB of RAM “maintains a 1 TiB deduplication window”². The 128 MiB block map cache is sized for “efficient access to 100 GB of logical space at a time”, and the kernel documentation names it the most important value to adjust as the working set grows¹.

Building the Stack

Let LVM see a PV on an LV

The volume group sits on an LV, and LVM ignores LVs as physical volumes by default. devices/scan_lvs turns that on, and LVM’s reference carries the warning that goes with it: “Caution is required to avoid using PVs that belong to guest images stored on LVs”⁴. A guest that uses LVM inside its own disk would otherwise have its volume groups turn up on the host.

So the filter changes too. Same edit. The stock filter on this host only rejected ZFS zvols and RBD devices. I added a reject for every Proxmox VM disk:

# lvmconfig devices/global_filter devices/scan_lvs
global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/mapper/.*-(vm|base)--[0-9]+--disk--[0-9]+|"]
scan_lvs=1

The same reference says scan_lvs “does not enable autoactivation of layered VGs”⁴. Proxmox does not need it to. Its LVM plugin activates each disk with lvchange -aey when the VM starts⁵. I tested it the way a boot leaves things, everything deactivated and then the same pvscan udev runs: the VG under VDO came up, and qm start activated the VM’s disk itself. The catch is underneath. With the VG under VDO incomplete, the same qm start failed with “Volume group “vdovg” not found”.

The VDO pool, thick, with its settings on the creation line

The formatter is in Debian’s vdo package⁶:

# apt install vdo

Then the pool. -L is the real space, -V the virtual size, kept inside what the pool can address. The thread layout and the discard size go on the same line, because neither can be changed later without stopping every VM on the pool:

# pvcreate /dev/nvme8n1
  Physical volume "/dev/nvme8n1" successfully created.
# vgcreate vdolab /dev/nvme8n1
  Volume group "vdolab" successfully created
# lvcreate -y --type vdo -L 40G -V 35G -n vdolv vdolab/vdopool /dev/nvme8n1 --vdosettings "logical_threads=4 physical_threads=4 hash_zone_threads=2 cpu_threads=3 bio_threads=2 ack_threads=1 max_discard=4096"
  Wiping vdo signature on /dev/vdolab/vdopool.
    The VDO volume can address 36.00 GB in 18 data slabs, each 2.00 GB.
    It can grow to address at most 16.00 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "vdolv" created.
# dmsetup table vdolab-vdopool-vpool
0 73402368 vdo V4 /dev/dm-7 10485760 4096 32768 16380 deduplication on compression on maxDiscard 4096 ack 1 bio 2 bioRotationInterval 64 cpu 3 hash 2 logical 4 physical 4

The thread layout is the one measured further down for a 16-CPU host; max_discard=4096 has its own section.

The volume group and the Proxmox storage

# pvcreate /dev/vdolab/vdolv
  Physical volume "/dev/vdolab/vdolv" successfully created.
# vgcreate vdovg /dev/vdolab/vdolv
  Volume group "vdovg" successfully created
# pvesm add lvm vdo-lvm --vgname vdovg --content images,rootdir --on-volume-remove discard=1
# pvesm status | grep -E "Name|vdo-lvm"
Name             Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
vdo-lvm           lvm     active        36696064               0        36696064    0.00%

on-volume-remove discard=1 matters, and it is covered below.

What Deduplication Saved

I wrote things a Proxmox host actually stores and read VDO’s block counters after each. logical blocks used is what the layers above wrote; data blocks used is what VDO stored. Both are 4 KiB blocks.

Each clone wrote all 4 GiB, zeros included, because a full clone onto a thick LV copies every block. VDO stored the zeros as nothing and the rest as references.

VDO is very good at data it has seen before. And at nowt. It is useless on data that is already compressed or encrypted, which is why the first ISO, mostly squashfs, saved 1.2%.

With the template and its six clones in place, Proxmox showed the storage 80.01% used, 28 GiB of thick disks out of 35. The drive held 4.3 GiB, and most of that was the index. On thick storage that is the point: Proxmox counts what has been allocated, and the drive carries what is actually different.

What the Drive Actually Receives

Savings on disk are a snapshot. The hardware cares about the stream. So I measured the bytes reaching the drive while writing: fio for 30 seconds, the drive’s write counter from /sys/block before and after, the same Optane with and without VDO, twice each.

The counter is honest. Across the whole run it added up to 794 GB, and the drive’s own SMART “Data Units Written” moved by 794 GB too.

On a realistic mix, VDO kept 79 to 93% of the bare drive’s throughput and sent it a quarter of the bytes. On fully random data it saves nothing and costs a lot, and the two runs disagreed so widely that I am only giving the range.

The tail latency is VDO’s own processing, not the drive. A p99 of about a millisecond, against a quarter of one bare.

The Optane has no cache to relieve

These drives say so themselves: vwc 0 from nvme id-ctrl, write through in /sys/block. An Optane writes in place with no DRAM or SLC staging, so on this hardware the gain is the bytes and the wear, not a drive cache that fills more slowly. A NAND SSD with an SLC cache, or a RAID controller with a write-back cache, is where fewer bytes buys responsiveness as well. That is the RAID6 section below.

Matching VDO’s Threads to the CPUs

Out of the box LVM gives VDO ten threads: one logical, one physical, one hash zone, two CPU, four bio submission, one acknowledgement. On a 16-CPU host that leaves most of the machine idle while VDO’s block map work queues behind a single logical thread.

The kernel documentation explains the knobs¹. Logical threads each own “a disjoint section of the block map”, physical threads “a disjoint section of the data blocks”, and both “can increase throughput”, while excess threads “can waste resources and increase contention”. Logical threads go up to 60, physical up to 16, and the pool needs at least one slab per physical thread. At 18 slabs this one has room.

Each layout ran the same fio jobs twice, interleaved, with random data, plus all seven clone disks read at once the way seven VMs boot together:

Matched to the CPU count wins nearly everything: a quarter more parallel writes, half as much again on sequential writes, seven disks read 2.4 times faster, and a tighter tail. Doubling again is worse across the board. That is the contention the documentation warns about. The defaults were also the least stable, with the seven-disk read swinging from 6.1 to 8.8 s between runs.

As such, the rule is one VDO thread per CPU, spread across the types as above, with physical threads capped by the slab count.

What VDO Costs on Fast Hardware

The same Optane with no VDO, against VDO with its threads matched, random data, mean of two runs:

Read it plainly. A single random read takes more than twice as long through VDO. Parallel random writes lose more than a third, sequential writes a quarter. On a drive this fast, VDO’s 17 µs of extra work per read is most of the read.

The QD1 write being faster through VDO is not a free lunch. Far from it. VDO acknowledges a write before it is on the drive, and the kernel documentation is plain that “unflushed writes may or may not persist after a crash”¹. That number measures an acknowledgement. A guest that flushes, which every filesystem does, gets the durability it asks for.

The seven-disk line is the one VDO wins outright, and it is worth being clear why. Seven plain copies have to read 28 GiB of real bytes, zeros included. VDO reads one set of shared blocks and skips the zeros.

So on fast flash, VDO trades latency for bytes. Every one of those microseconds is fixed cost, though, and it does not grow when the drive underneath gets slower.

Putting VDO’s Metadata on NVMe

VDO has no separate metadata device. Everything lives on the one pool LV. But the dm-vdo source shows the layout is fixed, and an LV can be built from pieces on different drives.

The format code places the block map roots from the start of the data region, then the slab summary and the recovery journal from the end, and fills the middle with slabs⁷. The UDS index sits before all of it, straight after the geometry block. vdolistmetadata confirmed it on the lab pool:

So build the pool LV in three pieces: the head on NVMe, the middle on the slow drive, the tail on NVMe.

Then convert it to a VDO pool, which lvmvdo(7) documents for an existing LV². Here nvme6n1 stands in for the NVMe and nvme8n1 for the slow drive:

# lvcreate -y -L 3G -n splitpool vdolab /dev/nvme6n1
  Logical volume "splitpool" created.
# lvextend -l +9408 vdolab/splitpool /dev/nvme8n1
  Size of logical volume vdolab/splitpool changed from 3.00 GiB (768 extents) to 39.75 GiB (10176 extents).
  Logical volume vdolab/splitpool successfully resized.
# lvextend -l +64 vdolab/splitpool /dev/nvme6n1
  Size of logical volume vdolab/splitpool changed from 39.75 GiB (10176 extents) to 40.00 GiB (10240 extents).
  Logical volume vdolab/splitpool successfully resized.
# lvconvert -y --type vdo-pool -n splitlv -V 35G vdolab/splitpool --vdosettings "logical_threads=4 physical_threads=4 hash_zone_threads=2 cpu_threads=3 bio_threads=2 ack_threads=1 max_discard=4096"
  WARNING: Converting logical volume vdolab/splitpool to VDO pool volume with formatting.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
    The VDO volume can address 36.00 GB in 18 data slabs, each 2.00 GB.
    It can grow to address at most 16.00 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
  Logical volume "splitlv" created.
  Converted vdolab/splitpool to VDO pool volume and created virtual vdolab/splitlv VDO volume.

Then check that VDO put its metadata where the extents are:

# vdolistmetadata /dev/mapper/vdolab-splitpool_vdata | grep -v "slab [0-9]* \(reference\|journal\)"
0 .. 0: geometry block
1 .. 679127: index
679128 .. 679128: super block
679129 .. 679188: block map tree roots
10452928 .. 10485695: recovery journal
10485696 .. 10485759: slab summary

An extent is 1,024 blocks. The NVMe head is extents 0 to 767, blocks 0 to 786,431, which covers the index and roots, with about 420 MiB of the first slab’s data along for the ride. The NVMe tail is extents 10,176 to 10,239, blocks 10,420,224 onwards, which covers the recovery journal and slab summary.

How much of VDO’s work moves with it, per 30-second run, counted per device:

A tenth or so of the write operations: the recovery journal, index updates, and that slice of slab 0 on the NVMe head. They are small, scattered metadata writes, the kind a spinning disk handles worst. On NVMe they are nothing.

Two limits. The per-slab metadata, reference counts, slab journals and block map leaves, lives inside the slabs, and stays with the data. And I did not test growing a split pool. The journal is placed from the end, so after any lvextend, run vdolistmetadata again and see where it went before trusting it.

VDO on bcache: One NVMe, Two Jobs

bcache is the other way to put flash in front of slow disks. It is what I already run under Ceph on spinning drives, with every setting explained in the bcache post. So the question is what VDO does to a bcache cache when it sits on top of one.

One NVMe does both flash jobs, split into two partitions:

Both drives are Optanes again, so nvme6n1 is standing in for spinning disk. Keep that in mind for every figure below. The backing device here is fast, so this measures what reaches the cache, not how long a spindle takes to drain it.

The partitions and the cache set, with the settings from that post:

# sgdisk --zap-all /dev/nvme8n1
# sgdisk -n 1:0:+32G -c 1:bcache-cache -n 2:0:+8G -c 2:vdo-meta /dev/nvme8n1
# blockdev --rereadpt /dev/nvme8n1
# apt install bcache-tools
# make-bcache -C /dev/nvme8n1p1 -B /dev/nvme6n1 --writeback
# B=/sys/block/bcache0/bcache
# echo 0 > $B/sequential_cutoff
# echo 40 > $B/writeback_percent
# echo 81920 > $B/writeback_rate
# echo 20480 > $B/writeback_rate_minimum
# CS=/sys/fs/bcache/47bac36f-a1d5-4e8e-8695-2e7ff944172d
# echo 0 > $CS/congested_read_threshold_us
# echo 0 > $CS/congested_write_threshold_us

Those echo lines last until the next reboot. The udev rule in the bcache post is how they stick.

Then the pool, split the same way as above, with nvme8n1p2 for the two ends and bcache0 for the middle:

# pvcreate /dev/bcache0 /dev/nvme8n1p2
# vgcreate vdolab /dev/bcache0 /dev/nvme8n1p2
# lvcreate -y -L 3G -n vdopool vdolab /dev/nvme8n1p2
# lvextend -l +9408 vdolab/vdopool /dev/bcache0
# lvextend -l +64 vdolab/vdopool /dev/nvme8n1p2
# lvconvert -y --type vdo-pool -n vdolv -V 35G vdolab/vdopool --vdosettings "logical_threads=4 physical_threads=4 hash_zone_threads=2 cpu_threads=3 bio_threads=2 ack_threads=1 max_discard=4096"
# lvs -a --segments -o lv_name,lv_size,segtype,seg_start_pe,seg_size_pe,devices vdolab
  LV              LSize  Type     Start SSize Devices
  vdolv           35.00g vdo          0  8960 vdopool(0)
  vdopool         40.00g vdo-pool     0 10240 vdopool_vdata(0)
  [vdopool_vdata] 40.00g linear       0   768 /dev/nvme8n1p2(0)
  [vdopool_vdata] 40.00g linear     768  9408 /dev/bcache0(0)
  [vdopool_vdata] 40.00g linear   10176    64 /dev/nvme8n1p2(768)

vdolistmetadata printed the same block ranges as the split build, so the index and both journals sit on nvme8n1p2. vdovg and the Proxmox storage go on top exactly as before.

What reaches the cache

The comparison is an 8 GiB LV straight on bcache0 against an 8 GiB LV in vdovg, the same fio jobs for 30 seconds, twice, with the cache flushed clean between every run. The device counters in /sys/class/block say where the bytes went.

bcache on its own topped out at about 210,000 small writes a second, and it wrote every byte into the cache plus 3% of its own. Put VDO in front and the cache took a quarter of the bytes. On the first run the same cache then carried 390,206 writes a second, nearly twice as many, and still finished with less dirty data waiting for the slow drive. The cache was the limit. VDO moved it.

The second run is the honest one for a pool that has been in use. fio seeds its generators the same way every run by default⁸, so run two wrote data the index had already seen, pointing at blocks that had since been discarded. VDO treats the index as hints and reads each candidate block to check it is really a duplicate before sharing it⁹. During a write-only test, that second run read 7.11 GiB back out of the cache and 1.20 GiB from the data drive. The first read 0.12 GiB. Stale advice is not free, and on a pool where VMs come and go there will always be some.

Random data is the other side of it:

Nothing to save, so VDO is pure overhead, and on the stale-index run it more than halved the small writes. Every read in both columns came from the cache. VDO’s 12 µs on top is the same fixed cost it adds on bare Optane.

So where the data repeats, the two stack well. The cache gets a quarter of the bytes, rides out a burst for longer, and has less to write back to the spindles. Where it does not repeat, bcache is better off on its own.

On HDD Hardware RAID6: Reasoned, Not Measured

I have no HDD RAID6 host to test on, so this section is reasoning built on the measurements above. Measure your own array before relying on it.

The first two lines are the hardware answer. While the controller’s battery or flash-backed cache has room, the guest sees cache latency. When it fills, the guest sees the spindles. Send it a quarter of the bytes and the same cache gives about four times the headroom, and has a quarter as much to drain. The bcache section above is that measured with a software cache in place of the controller’s.

The third is why VDO belongs on slow disks more than fast ones. On the Optane, VDO doubled the read latency. On 7,200 rpm drives the same fixed cost disappears into one seek.

The last three are the costs. A boot storm of identical clones gets faster, but one clone read end to end gets slower, because deduplication scatters it. And VDO’s own small writes land on the array too, which is the case for the split layout above: index and journal on NVMe, slabs on the RAID6.

Deleting a Disk: Make Proxmox Discard It

Removing an LV frees its extents in LVM. It does not tell VDO the blocks are unused, so VDO keeps them. The kernel documentation says why that matters: “If the vdo is thinly provisioned, discarding unused blocks is essential to prevent the target from running out of space”¹. On thick storage it cannot run out, but undiscarded blocks are still drive space held for data nobody owns.

The on-volume-remove discard=1 option on the storage is the fix. Proxmox describes it as issuing “discard (TRIM) requests before removal”⁵:

It is slow on a big disk, though. Deleting the 34 GiB disk from the fill test took Proxmox 262.6 s, while blkdiscard on the same kind of data managed 15 GiB in 2.25 s. So for anything large, blkdiscard the disk first and then delete it.

The discard size matters too. VDO’s default max_discard is one 4 KiB block per request¹:

Inside the guests, turn on the drive’s Discard option so their TRIM reaches the storage¹⁰, and run fstrim on a schedule.

Where It Earns Its Place

None of that is hard. None of it is the default.

The Cheapest Write Is the One You Never Send

What strikes me about VDO is how badly it gets sold. It turns up as a way to put more on a disk than the disk holds, and that is the least useful thing it does. I did it by accident on the first build, and Proxmox showed a third of the storage free while VDO was full. Overcommit is a loan, and the drive calls it in at full write speed.

The real value is in the line most people never measure. A quarter of the bytes reaching the drive is a quarter of the wear and a controller cache that lasts four times as long. That is not capacity. It is the hardware doing less work for the same result, which is the only kind of saving that does not come back to bite you.

It costs CPU, and on fast flash it costs latency, and both are fixed. That is the trade, stated plainly: processor time spent so the slow, expensive part of the box does less. On spinning disks behind a RAID controller or a bcache, with the index and journal on an NVMe, that sum comes out well. On an all-Optane host it mostly does not.

And it will not look after itself. Every default that matters is wrong for this job: ten threads on sixteen CPUs, a discard size that burns a core, a virtual size you can set to anything, and nothing in the storage config telling Proxmox to give space back. None of that is hidden. It is all in the documentation.

So build it thick, once, properly, with the settings on the creation line and the metadata where it belongs. Do that and the drive does a quarter of the work. Skip it and you have bought another layer that promises more than it has.