FlashStore OSD on Proxmox VE: What It Took on One Node

A Rust OSD, On The Day It Shipped

croit put FlashStore OSD into public preview on 9th November 2026¹. It is a Ceph OSD written from scratch in Rust, with its own object store in place of BlueStore, and it speaks the Ceph wire protocol. The claim is that it drops into a cluster you already run: your monitors, your managers, your ceph CLI and your clients stay as they are, and you replace one drive at a time¹.

That claim matters more to a Proxmox shop than to most. On a hyperconverged Proxmox VE node, Ceph is not something you install separately. It comes out of the Proxmox GUI, from Proxmox’s own repositories, built and patched by Proxmox, and the whole thing is driven by pveceph. If an OSD from somebody else can sit inside that without breaking it, you get a new storage engine without giving up the tooling. If it cannot, you find out the hard way, on a node that is also running your VMs.

So I tried it.

I work for croit, who make FlashStore. That is why I tried it within two days of it shipping, and it is also the reason nothing below gets softened. It is a preview, the README says it is “not yet recommended for production data”¹, and this is a write-up of what it took on real kit, not a brochure.

Short answer: it works. Eight FlashStore OSDs are serving an RBD pool to a VM on a stock Proxmox VE 9.2 node as I write this, HEALTH_OK. The longer answer is that it took two fixes the documentation does not mention, one of them because of how Proxmox stores its Ceph config and one because of Proxmox’s kernel, and the rest of this post is those fixes and how they were found. If you only want it running, there is a script near the end that does all of it, and it has been run on this node.

The Box

The lab is a single node, pve2, on my home LAN, and it is a crappy Minisforum B550 mini PC from 2021. Single-node Ceph is not how you would run production, and the CRUSH rule further down is the price of it, but every step here is the same on a node that has neighbours.

The Optanes are there because FlashStore has two hard requirements a lot of drives fail. The drive must have power-loss protection, and the namespace must use 4096-byte blocks, as a whole namespace with no partitions or filesystem on it¹. FlashStore acknowledges a write when it is durable on the drive, so a drive that lies about durability with a volatile cache is refused at mkfs³. Every P4800X reports no volatile write cache in its Identify data (vwc 0), which is the drive itself saying the cache cannot lose an acknowledged write.

As such, the mini PC is the limiting factor for this deep dive, not FlashStore and not the drives. Eight cores means one per OSD and nothing spare, 58 GiB has to feed Ceph and the VMs alike, and eight Optanes share one PCIe 3.0 link because that is all the APU has to give. Read every number below as what this box allows.

Umbrella Or Nothing

FlashStore is built for one Ceph release at a time. The branch decides it: main speaks Ceph 21.2.0 Umbrella, and a v20.2.4 branch speaks 20.2.4 Tentacle⁴. The daemon checks the cluster’s release when it starts and refuses one outside its window⁴. The packages on the release page are built from main, and the binary says so itself: croit-flashstore-osd 21.2.0 umbrella.

Proxmox VE 9 offers three Ceph repositories, and only one of them lines up.

Tentacle is the Proxmox default, so is it the safe choice? It is the one that will not work. The Proxmox repository carries Tentacle up to 20.2.5-pve1, and FlashStore’s Tentacle branch targets 20.2.4 exactly. Umbrella in Proxmox’s repository is 21.2.0-pve1, the same release main targets.

Proxmox does not hide what Umbrella is. PVE/Ceph/Releases.pm lists it with unsupported => 1⁵, and the GUI wizard says so in plain words the moment you pick it.

“Technology Preview”, no enterprise repository, and the installer asks a second time in the terminal, continue (y/N)?, before it touches apt. That is the right amount of warning, and it is worth reading as a statement about the whole stack: a preview OSD on a preview Ceph release.

The install is stock Proxmox. Datacenter → Ceph → Install, version Umbrella, repository No-Subscription, then the public network and the first monitor.

Thirteen packages upgraded, 46 new, 107 MB downloaded. One thing on that list is new to Umbrella: ceph-osd-classic alongside ceph-osd. Neither gets used here, because no classic OSD is ever created. About a minute later mon.pve2 and mgr.pve2 were running, ceph versions reported 21.2.0 … umbrella, and health was OK with nothing to store yet.

Four Drives At 512 Bytes

Four of the eight Optanes had been passed through to a lab VM and were still in 512-byte format, with old ZFS labels on them. Two more had the right format and the same labels. The fix for both is plain: take them off the VM, format to 4 KiB, wipe the labels.

The P4800X offers 4096 bytes with no metadata as LBA format 3, which nvme id-ns reports as lbaf 3 : ms:0 lbads:12.

nvme format /dev/nvme0n1 --lbaf=3 --force --timeout=900000
wipefs -a /dev/nvme0n1
sgdisk --zap-all /dev/nvme0n1

The first attempt failed, and so did a later one. Both times nvme format came back with Interrupted system call and the drive was still at 512 bytes. The kernel log has the reason:

nvme nvme0: I/O tag 0 (0000) opcode 0x2 (Admin Cmd) QID 0 timeout, reset controller
nvme nvme1: I/O tag 9 (0009) opcode 0x6 (Admin Cmd) QID 0 timeout, reset controller

Opcode 0x2 is Get Log Page and 0x6 is Identify. Neither is the format. A P4800X formats the whole medium, and that took 345 to 351 seconds per drive here. While it runs, the controller does not answer other admin commands, so anything that asks the drive a question waits. After the kernel’s 60-second admin timeout it gives up and resets the controller, and the reset kills the format.

The fix is to run each format in a transient unit with a long timeout, one drive at a time, and leave the drives alone until it finishes:

systemd-run --unit=fmt4k /root/fmt4k.sh
journalctl -u fmt4k -f       # read progress from the journal, not from the drive

That way the only progress check is the journal, and nothing reaches the controller. All eight then read 4 KiB in nvme list.

The Package

The release page carries .deb and .rpm packages, with Debian 13 trixie among them, which is what Proxmox VE 9 is built on². One build serves every distribution, needing glibc 2.34 or newer⁶.

B=https://github.com/croit/ceph-rust/releases/download/snapshot-2026-11-10-0b6783237291
curl -LO "$B/flashstore_21.2.0+git.20261110063351.c5792750020e_debian-trixie_amd64.deb"
curl -LO "$B/sha256sums.txt"
sha256sum flashstore_*_debian-trixie_amd64.deb
grep debian-trixie/flashstore_ sha256sums.txt

Check it by hand. sha256sum -c cannot check this download, because the checksum file names it debian-trixie/flashstore_…_amd64.deb while the asset is called flashstore_…_debian-trixie_amd64.deb. The hashes match, ce1fb352…26c8 on both sides, so the file is fine and only the list is out of step with the assets.

apt install ./flashstore_*_debian-trixie_amd64.deb
croit-flashstore-osd --version

croit-flashstore-osd 21.2.0 umbrella (build 0.0.0) git c5792750020e
net tiers: kernel (+rdma upgrade)

It installed cleanly beside Proxmox’s own ceph-osd and ceph-osd-classic. The package installs a minimal ceph-osd.target only on hosts that lack one⁶, and Proxmox already has one, so the FlashStore units hang off the same target the classic ones would.

Three things about the package worth knowing before you script against it:

The middle row matters. The SPDK tier, the user-space NVMe driver that needs the drive bound to vfio-pci, is not in the published Debian package. What is in it is the kernel tier: the drive stays on the kernel’s own nvme driver and FlashStore talks to it through io_uring. The source shows it compiled in, because the knet feature pulls in kernel-nvme⁸, even though the host check’s summary line calls it “compiled out of this build”. That line is wrong. A reader who trusts it would stop right there.

What The Host Check Says

croit-flashstore-osd --check is read-only and runs before anything is written. It checks the build, CPU, memory, IOMMU, every NVMe controller, the NIC, the kernel and the config, and prints the exact fix for every finding⁹. On this node it ran 115 checks, and it is good: every finding comes with the command that fixes it, which saved a load of digging.

Plan for the second FAIL. The check counts every controller in the box, and the boot SSD has a write cache, so it fails the power-loss test, even though the same check correctly marks it “in use, not a candidate”. The tutorial says exit 0 means no FAIL⁹. On any host booting from a consumer SSD, which is a lot of Proxmox hosts, --check cannot exit 0, whatever you fix. Script against the per-check lines, not the exit code.

The tuning WARNs stayed. Every one of them needs a kernel command line change and a reboot, and this is a deployment, not a benchmark. They are the right list to work through before anybody quotes a number from this box.

Cores and hugepages for eight OSDs

FlashStore runs thread-per-core, and the check spells out what that means: “one core = one shard/PG owner”, at most 8 cores per OSD, and each OSD core on its own physical core rather than sharing one with an SMT sibling. Eight physical cores and eight OSDs makes that one core each, flashstore_lcores = 0 through 7, with the SMT siblings left to Proxmox and the VMs. Re-run with that in the config, the check marks every core plan OK.

Look again at the hugepages. The check asks for 2048 MiB per OSD, which is the daemon’s default --mem, so eight OSDs need 16 GiB reserved. Running, each OSD has used 512 MiB of its 2048. That matches croit’s own design note for the kernel backend, which budgets a quarter of --mem for its payload allocator¹⁰. So on this box, with all eight running, 12 GiB of RAM sits reserved as hugepages that nothing touches, on a host that also has VMs to feed. A storage node would shrug. On a Proxmox node, that memory was a VM, and further down it is trimmed back to what the OSDs use.

Can I Ask Why It Cannot Read ceph.conf?

With the drives formatted, the package in and the host check clear, the first OSD would not start.

croit-flashstore-tool: refused: /etc/ceph/ceph.conf: Permission denied (os error 13)

On Proxmox, /etc/ceph/ceph.conf is a symlink to /etc/pve/ceph.conf, and /etc/pve is pmxcfs, the cluster filesystem that keeps every node’s config in step. Files there are root:www-data 0640, and pmxcfs does not let you change that per file¹¹. Proxmox’s own Ceph daemons cope because they start as root, read the config, and then drop to the ceph user, which is what --setuser ceph --setgroup ceph on every ceph-osd@ unit does.

FlashStore’s unit does not start as root. It is User=ceph from the first instruction, ExecStartPre included¹², so it never gets to read the file.

Can I ask why a package built for Debian, for a drop-in product, was not tried against the most common Debian-based Ceph platform there is? I am not asking who. pveceph keeps its config on pmxcfs, and every FlashStore OSD on a pveceph-managed node will hit this on its first start.

The daemon already has the answer built in. Its source describes a classic --setuser/--setgroup drop: “started as root by a classic unit (--setuser ceph --setgroup ceph), the daemon becomes that user before it opens the store, a socket or a file”¹³. So a drop-in that starts the unit the way Proxmox starts its own OSDs should be enough.

It was not quite. Started as plain root, the next refusal was the drive.

activate: device kernel:/dev/ng0n1 not ready: /dev/nvme0n1: Permission denied (os error 13)

The unit’s CapabilityBoundingSet leaves out CAP_DAC_OVERRIDE¹², so root is held to ordinary file permissions, and the namespace nodes belong to ceph:ceph. Starting as root with the group set to ceph covers both: root owns the config file, the ceph group opens the drive, and the daemon drops to ceph before it does anything else.

# /etc/systemd/system/croit-flashstore-osd@.service.d/pve-ceph-conf.conf
[Service]
User=root
Group=ceph
ExecStart=
ExecStart=/usr/bin/croit-flashstore-osd run --id %i --cluster ${CLUSTER} -f --setuser ceph --setgroup ceph

The daemon confirms the drop in its own log: set uid:gid to 64045:64045 (ceph).

There is a shorter fix, SupplementaryGroups=www-data, and I did not use it. It would let the OSD read every group-readable file under /etc/pve, not just the Ceph config. An OSD has no business reading the rest of the cluster’s configuration, and the drop-in above gives it nowt more than Proxmox gives its own.

Then the device nodes. FlashStore’s docs say the namespace node must be writable by ceph, “(udev rule)”¹⁴, and the package ships none. This one matches the drives by model, so it cannot catch the boot drive whatever the numbering does:

# /etc/udev/rules.d/70-croit-flashstore-optane.rules
SUBSYSTEM=="nvme-generic", KERNEL=="ng*n*", ATTRS{model}=="Dell Express Flash NVMe P4800X*|INTEL SSDPE21*", OWNER="ceph", GROUP="ceph", MODE="0660"
SUBSYSTEM=="block", KERNEL=="nvme*n*", ENV{DEVTYPE}=="disk", ATTRS{model}=="Dell Express Flash NVMe P4800X*|INTEL SSDPE21*", OWNER="ceph", GROUP="ceph", MODE="0660"

After udevadm trigger, the sixteen Optane nodes were ceph:ceph 0660 and the SN750 stayed root.

The Passthrough This Kernel Will Not Do

The kernel tier has two modes, and the node you name picks between them. kernel:/dev/ngXnY sends NVMe commands straight to the drive through io_uring, which croit call kernel-cmd. kernel:/dev/nvmeXnY does O_DIRECT reads and writes on the block device, called kernel or direct¹⁰. croit measured kernel-cmd ahead in every cell they tested¹⁰, and the README’s own quick start uses it¹, so that is what I configured.

With the permissions sorted, the daemon dropped to ceph, read its config, and stopped on the first read:

croit-flashstore-osd: --dev kernel:/dev/ng0n1: /dev/ng0n1: kernel-cmd fixed-buffer probe read: Errno(13)

The same probe had passed a minute earlier in activate, which still had root’s capabilities. So the question is whether this kernel lets an unprivileged process send I/O commands down /dev/ngXnY. That is answerable without FlashStore, using the same read with nvme-cli, as ceph:

All eight Optanes gave the same answer, including two I had never formatted or reset, running Intel’s firmware rather than Dell’s. So it is not the drives and it is not the morning’s formats.

Upstream says it should work. In Linux 7.0, nvme_cmd_allowed() lets an unprivileged process send an I/O command when the drive’s Commands Supported and Effects log marks it supported with no unusual effects, and requires a writable open only for commands that write or change data¹⁵. The P4800X’s effects log marks Read as supported, 00000001, and nothing else. The read is allowed.

Proxmox’s kernel is not upstream’s. Its changelog for this build includes a backported fix on exactly this path, CVE-2026-90227: nvme/ioctl: check SUBMIT_IO with nvme_cmd_allowed(), which on its face covers a different ioctl from the one I tested. I have not pinned down which change in the 7.0.14-28-pve kernel makes the difference, and I will not guess. What is certain is the behaviour: on this kernel, unprivileged NVMe passthrough I/O is refused.

FlashStore has a fallback for refused passthrough, and it did not fire. croit’s design note says that when Identify is refused, a kernel-cmd device drops to direct I/O on its block node and says so at startup¹⁰. Here Identify is allowed and Read is not, so the fallback never triggers and the OSD stops instead.

Direct I/O does not care. It is one line per OSD:

flashstore_store_backend = kernel

With that, osd.0 came up.

croit-flashstore-osd: store mounted (1 shards, replayed 4 txns in 16 ms)
croit-flashstore-osd: set uid:gid to 64045:64045 (ceph)
[    1.398] crush location set (create-or-move updating item name 'osd.0' weight 0.3359 at location {host=pve2,root=default} to crush map)
[    2.406] osd.0 is now active (up_from 10, map 10)

Active in 2.4 seconds.

Preparing And Starting The OSDs

Each OSD gets a section in the Ceph config, which on Proxmox means editing /etc/pve/ceph.conf, and that is fine to do by hand.

[osd.0]
	flashstore_dev = spdk:0000:05:00.0
	flashstore_store_backend = kernel
	flashstore_lcores = 0

The device is named by PCI address, not by /dev/nvme0n1, and the spdk: prefix does not mean SPDK here. With a kernel backend set, the daemon looks the address up in sysfs at every start and finds the controller’s current nodes¹⁰.

That matters here. The SN750 boot drive was nvme7 in October and is nvme8 now, and the generic node minors do not follow drive order either, with ng1n1 at minor 0 and ng0n1 at minor 2. A config that says /dev/nvme4n1 can point at a different drive after a reboot. A config that says 0000:0b:00.0 cannot.

Then one command per drive, writing the store and registering the OSD with the monitor:

croit-flashstore-tool prepare --dev spdk:0000:05:00.0 --id 0

kernel-cmd /dev/ng0n1: fixed-buffer NVMe passthrough probe ok (IORING_URING_CMD_FIXED on)
mkfs: device lba 4096 B x 91573146 (375083606016 B), awupf 1 lba, vwc false, max_xfer 32 lba
mkfs: 1 store shards, WAL ring 4096 MiB each
osd.0 prepared in /var/lib/ceph/osd/ceph-0

Two seconds, as root. prepare reads the bootstrap key from /var/lib/ceph/bootstrap-osd/ceph.keyring¹⁶, which pveceph only writes the first time it creates an OSD of its own, so with no classic OSDs it has to be fetched first:

ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring

Six drives prepared first time. Two were refused:

croit-flashstore-tool: refused: device is not empty (non-zero data in the first 8192 bytes); --wipe destroys it
croit-flashstore-tool: refused: device is not empty (non-zero data at 65536 bytes); --wipe destroys it

lsblk and blkid had both reported those two drives as empty. What was on them was the text metadata area of an LVM volume group from an earlier lab, long since removed, with the label at the front already zeroed. FlashStore’s probe reads the first 8 KiB and the regions at 64 KiB and 256 KiB¹⁷, which is further than the usual tools look, and it was right. Refusing a drive that holds anything at all, unless told to --wipe, is exactly what an OSD tool should do. Wiping them is a decision for whoever owns the drives, not for the tool. Once that was decided, --wipe prepared both in about two seconds, and they joined after the install further down, which is why the panel screenshot and the install figures show six.

Then start them:

systemctl enable --now croit-flashstore-osd@0

ID  CLASS  WEIGHT   TYPE NAME      STATUS  REWEIGHT  PRI-AFF
-1         2.01535  root default
-3         2.01535      host pve2
 0   nvme  0.33589          osd.0      up   1.00000  1.00000
 1   nvme  0.33589          osd.1      up   1.00000  1.00000
 2   nvme  0.33589          osd.2      up   1.00000  1.00000
 3   nvme  0.33589          osd.3      up   1.00000  1.00000
 4   nvme  0.33589          osd.4      up   1.00000  1.00000
 5   nvme  0.33589          osd.5      up   1.00000  1.00000

Each OSD put itself under host=pve2 with device class nvme. What they tell the cluster about themselves is worth reading once:

The BlueStore answer is deliberate. flashstore_report_objectstore defaults to bluestore because classic monitors only enable some features when every OSD reports it, so the default keeps a mixed cluster working⁷. It is not true, and it is worth knowing before you read the cluster’s metadata and believe it.

What Proxmox Makes Of Them

The Proxmox OSD panel lists all six, up and in, class nvme, version 21.2.0, 344 GiB each. It also calls them FileStore.

FileStore is the object store BlueStore replaced. Proxmox works the label out in PVE/API2/Ceph/OSD.pm: if the OSD’s metadata has a bluefs key it is BlueStore, and anything else is FileStore¹⁸. FlashStore has no BlueFS, because there is no RocksDB under it, so it reports BlueStore in one field and gets labelled FileStore from another. Neither is right. And the panel is where most Proxmox admins look first.

The latency columns read 0 / 0 at idle. Under the install below, ceph osd perf reported commit latencies of 0 to 3 ms across the six.

Start, Stop and Restart in the panel drive ceph-osd@N, the classic unit, which is not what is running¹⁹. As such, the panel is a view of these OSDs, not a control for them. Use systemctl … croit-flashstore-osd@N and the CLI for anything that touches the daemons, and leave Create: OSD and Destroy well alone on these.

A Pool, A Storage And A Fisher-Price OS (Windows) VM

The first PG in the cluster, the manager’s .mgr pool, sat undersized+degraded as soon as the OSDs came up. Blame the single node. The default replicated_rule puts each copy on a different host, and there is one host.

ceph osd crush rule create-replicated replicated_osd default osd nvme
ceph osd pool set .mgr crush_rule replicated_osd

A rule with failure domain osd and device class nvme²⁰ puts each copy on a different OSD, and the PG went active+clean within seconds. On a real cluster with three or more hosts, keep the host rule and skip this.

Everything after that is the Proxmox GUI as normal. Ceph → Pools → Create: name flashstore, size 3, min size 2, rule replicated_osd, Add as Storage ticked. It went to size 2, min size 1 later, for the load test further down.

Proxmox created the pool and the RBD storage definition in one go, and flashstore appeared in the tree with 654 GiB usable, which is the 2 TiB raw divided by three copies, less overhead. 129 PGs went active+clean, and the autoscaler brought the new pool down from 128 to 32 shortly after.

The test guest is a Fisher-Price OS (Windows) VM. That is not a preference, it is what a large share of customers actually run on their Proxmox pools, so it is the load that counts: 16 GiB of memory, 4 cores, a 60 GiB disk on flashstore with discard and iothread on. It is also chatty to disk and inefficient with it, which is the point: a storage test wants the guest that leans on the disk hardest.

Sampling the pool every ten seconds for the first fifteen minutes of the install:

Those are an installer’s numbers, with the VM and the OSDs on the same host and the tuning WARNs untouched, and they say nothing about FlashStore’s speed. What they do say is that a classic Proxmox client drove an operating system install onto Rust OSDs, 35 GiB raw, without a fault, which is the claim this post set out to test.

What It Does Under Load

An install proves it works. It does not tell you what the VM is like to use.

So the VM got CrystalDiskMark 9.0.3²¹, three times: the Real World Performance [+Mix] profile, Read&Write [+Mix], a 16 GiB file on C: and a 70/30 read and write mix, once at three passes and twice at nine. That profile runs everything one request at a time on one thread, which is what a desktop or a small database actually does, not what a datasheet does.

By then the setup had moved on from the install. The VM had six cores, and the pool had been changed in the GUI to size 2, min size 1 and 256 PGs. Size 2 with min size 1 is a lab setting on one box: with one copy left, the pool keeps taking writes that a single failure can then lose. Elsewhere, keep 3 and 2.

The disk was on Proxmox’s default, no cache, which has been the default since Proxmox 2²².

Writes and small-block latency hold within about 1% from run to run. Sequential read climbs 45% over the three. That is the first sign the read column is not measuring what it looks like.

The host was sampled every five seconds through all three, from /proc/diskstats for the eight Optanes, ceph status for the pool and /proc for every process and thread, and the two views do not tell the same story.

Writes: the drive is the small part

The drive is under 5% of a small write. The other 393 µs is the trip: QEMU, librbd, the messenger into the OSD, the OSD itself, and the wait for the second copy. None of that was measured apart here, so I will not put a number on each piece. What it does say is that on this box the Optanes are not the limit for one-at-a-time I/O, and a faster drive would not move that 410 by much. The limit has moved to the client, the librbd that Proxmox ships, and there is more to come on that when croit’s own librustyrbd is deployed here.

Reads: the cache, not the Optanes

In the last run the 4 KiB random read pass did 5,700 reads a second across the whole 16 GiB file, and the eight Optanes read nothing at all. The sequential pass reached 2.2 GB/s at the pool while the drives’ own reads fell from 755 MB/s to zero. CrystalDiskMark bypasses the guest’s cache and the disk has none in QEMU, so the memory doing the work is in the OSDs. flashstore_obj_cache is on by default in this build, each OSD’s share of the file is about 2 GiB at size 2, and each OSD had grown to about 2.6 GiB. FlashStore does not report its hits, so call that near certain rather than proven. It is also why sequential read rose with every run: more of the file was already there. Treat the read column as a cache figure, not a store figure.

CPU: where the cores went

The VM’s three columns come from its threads: QEMU names the guest’s vCPUs, and librbd’s messenger and worker threads run inside the same process.

The OSD column barely moves. Eight polling OSDs cost 3.3 cores doing nothing and 3.9 flat out, so on a 16-thread box a fifth of the CPU is gone before a VM asks for anything. Nothing else came near its limit. The busiest single thread was a guest vCPU at 47%, and librbd’s busiest stayed under 15%. At one request at a time the limit is the wait, not the work, and a CPU figure cannot show a wait.

Memory: plan for the peak

The hugepage row is the budget section again, under load this time: 512 MiB per OSD, and the other 12 GiB never touched.

Ordinary memory is the number to watch, because the cache above is made of it. Once a read had pulled the file in, every OSD sat at about 2.5 GiB, and this is what Ceph held on a 58.7 GiB host:

It does come back, but not on any timetable I could see. It held at 19.9 GiB for 25 minutes, idle gaps included, then dropped to 13.9 GiB at the end of the last run, and the host had 10.2 GiB to hand out again. osd_memory_target sat at its default of 4 GiB per OSD for all of that, so eight OSDs were entitled to 32 GiB before a single hugepage is counted. On a node that is also meant to run VMs, the peak is the budget to work out before the drives, and it is where the 12 idle GiB of hugepages hurts most.

Three VMs at once, on a tighter budget

One VM leaves plenty of room. Three is where the budget has to be right. So two settings changed first.

ceph config set osd osd_memory_target 1073741824   # 1 GiB per OSD, was 4
sysctl -w vm.nr_hugepages=2048                     # what eight OSDs use, was 8192

The target needs nothing else: every OSD reported the new value within seconds, and they were restarted one at a time anyway, with all 257 PGs back to active+clean before the next. The hugepages have a catch. Before every OSD start, croit-flashstore-prep wants a whole --mem free, 2048 MiB, and raises the pool if it is short, but it never lowers it again. Restarting osd.7 after the trim took the pool straight back up:

croit-flashstore-prep[7]: hugepages 2048kB: free 256 < need 1024, nr_hugepages 2048 -> 2816

So the floor that survives a restart or a reboot is what the OSDs use plus one start’s margin: 2,816 pages, 5.5 GiB, against the 16 GiB the check asks for. The setup script above now reserves exactly that. Trimming back to 2,048 once all eight are up buys another 1.5 GiB until the next restart.

Then three clones of the VM, at 10 GiB each, ran CrystalDiskMark at the same moment: Real World Performance [+Mix], three passes, but on a 1 GiB file each rather than 16 GiB. That makes the read column a cache figure more than ever, so read the writes.

Three VMs got 2.3 times the small writes of one, and each paid about 27% in latency for it. The Optanes were still not the reason. They peaked at 20,900 writes a second at 17 µs each, the same 17 µs as one VM, and the busiest of the eight was 14% busy.

So FlashStore does hold itself to osd_memory_target. It went 15% over at the peak of a read pass and came back under, with the cache doing its work inside the budget instead of outside it. Between the target and the trim, 26 GiB came back to the VMs on the same box.

Booting three guests worked the CPU harder than benchmarking them. The storage underneath never came near it, which is the mini PC setting the limit, as it was always going to.

All Of It, In One Script

Everything above is in one script. Give it the PCI addresses of the drives you want as OSDs and it does the rest, in the order this post did it, with the two Proxmox fixes built in.

lspci -D -d ::0108                                    # find the drives
bash flashstore-pve-setup.sh --dry-run 0000:05:00.0   # show the plan, change nothing
bash flashstore-pve-setup.sh 0000:05:00.0 0000:06:00.0

Nothing is destroyed unless you ask. --format-4k reformats a drive that is not at 4 KiB, one at a time in its own unit so nothing interrupts it, and --wipe lets it overwrite a drive that holds data. --single-node adds the replicated_osd rule, and running it again skips drives that are already OSDs.

It was tested here, not just written. It refused the SN750 on its write cache, refused osd.7’s drive while it still held a store, and then, with osd.7 purged and its nodes handed back to root, rebuilt it from nothing in 10.5 seconds, choosing the kernel backend on its own. What it will not do is install Ceph. That step asks you to accept a Technology Preview, and that is your decision, not a script’s.

What Wants Fixing

None of this stops a determined admin, and all of it would stop a busy one. Each item is a statement croit makes to anyone who downloads the package, and each one I hit on the first try.

None of this is new work. The daemon already has the privilege drop, the direct backend and the fallback logic. They just have not been joined up for the platform most people will try it on first.

A Preview Is Where You Find This Out

A preview is the right place for every one of those findings, and the wrong place to leave them. That is what a preview is for: somebody takes it home, puts it on the kit they actually have, and writes down where it falls over. The parts that matter held. The store came up in milliseconds, the OSDs joined a cluster built by somebody else’s tooling, and an operating system install went onto them without a fault.

What tripped it was everything around the daemon. A config file on a filesystem the unit could not read, a kernel that says no to a call upstream allows, a boot drive the check would not stop counting. None of that is hard. All of it is the difference between software that works and software that works on your box, and the only way to close it is to test on the boxes people have rather than the ones you built for it.

For all that, this is a promising start. On a crappy mini PC, with the client end still the classic librbd and all eight drives on one PCIe 3.0 link, the Optanes are worked harder here than under BlueStore. As this develops I will come back to it and dig deeper, as more of the stack is replaced, and with a look at CephFS2 as well.

“Drop-in” is a promise made to an admin who has not got time to read the source. Proxmox’s monitors, Proxmox’s GUI, Proxmox’s kernel and a package from someone else is exactly the situation that promise is for. It nearly keeps it now. Nine small fixes and it will, and I would rather be the one who found them on a Wednesday morning than a customer who finds them on a Friday night.