Skip to content
Crimson, Ceph's own Seastar rewrite of the OSD with SeaStore, against croit FlashStore, a Rust rewrite with its own store

FlashStore Against Crimson: Two Rewrites Of The Same OSD

Crimson is Ceph’s own rewrite of the OSD, in C++ on the Seastar framework, with a new native store called SeaStore; FlashStore is croit’s rewrite, in Rust, with a store of its own. Both are in preview and both are aimed at NVMe and fast networks. This compares them as Ceph 21.2.0 Umbrella ships Crimson, from Ceph’s source and documentation and croit’s ceph-rust snapshot: where each comes from and under what licence, why Rust meets the Five Eyes memory-safety guidance and C++ does not, why Crimson needs its own pools and an experimental flag while FlashStore drops into existing pools, why Umbrella has no way to turn an existing pool into a Crimson pool so the data has to be copied, what each tells the monitor at boot, Seastar reactors against pinned polling cores, the fact that a default Crimson OSD still runs BlueStore, SeaStore’s onode, OMAP and LBA trees against FlashStore’s one tree per shard, the order each commits a write in, SeaStore’s segment cleaner, and what hardware and pools each will take.

11th November 2026 · 20 min · 4679 words · Damien Dye
Umbrella BlueStore, RocksDB on BlueFS on any drive, against croit FlashStore, a copy-on-write B+tree and WAL per core on NVMe with power-loss protection

FlashStore Against Umbrella BlueStore: What croit Changed Under The OSD

croit’s ceph-rust is a clean-room Rust rewrite of the Ceph OSD with its own object store, FlashStore, in public preview since 9th November 2026. This puts it side by side with BlueStore exactly as Ceph 21.2.0 Umbrella ships it, read from both projects’ source and documentation: what is in the repository and under which licences, where each store keeps its metadata, how a 4 KiB write and a write under 4 KiB reach the drive in each, Umbrella’s thread pools against FlashStore’s one thread per core, why FlashStore makes power-loss protection a hard requirement, what it will not run on, why it tells your monitors it is BlueStore, which BlueStore options it honours and which it ignores, and why the only way between the two is backfill.

11th November 2026 · 20 min · 4665 words · Damien Dye
A Proxmox VE node running Proxmox's own Ceph monitor and manager beside croit FlashStore OSDs on eight Optane drives

FlashStore OSD on Proxmox VE: What It Took on One Node

croit’s FlashStore OSD is a Rust replacement for ceph-osd and BlueStore, in public preview since 9th November 2026. This deploys it hyperconverged on a single Proxmox VE 9.2 node: Ceph Umbrella 21.2.0 from the PVE wizard for the monitor and manager, FlashStore OSDs on Intel Optane P4800X drives. It covers why Umbrella is the only Proxmox release that matches, formatting the drives to 4 KiB and the controller reset that aborts a format, what croit-flashstore-osd –check reports, why the shipped unit cannot read Proxmox’s ceph.conf and the drop-in that fixes it, the NVMe passthrough Proxmox’s kernel refuses and the direct backend that works, preparing OSDs by PCI address, what the PVE OSD panel makes of them, a pool, RBD storage and a Fisher-Price OS (Windows) VM installed onto it, CrystalDiskMark from inside that VM read against the drives’ own counters, Ceph’s memory with all eight OSDs active, three VMs at once on a 1 GiB memory target, and a setup script that does all of it, tested by rebuilding an OSD from nothing.

11th November 2026 · 35 min · 7721 words · Damien Dye
An NVMe drive entering a tri-mode adapter and leaving as a SCSI disk with a queue depth of 128

Tri-Mode Adapters Buy Flexibility With Your NVMe Queues

A tri-mode adapter lets any bay take SAS, SATA or NVMe — real flexibility, and the reason U.3 backplanes exist. What it does not advertise is that your NVMe drives stop being NVMe drives: they arrive in Linux as SCSI disks on mpt3sas, queue depth 128, sharing one tag pool and one x8 uplink. Two Gen4 drives saturate the card. A single Gen5 drive is already past it. Whether that trade still makes sense in 2026, and why the universal bays belong to the backplane rather than the controller.

13th August 2026 · 25 min · 5202 words · Damien Dye
Three storage tiers: Optane absorbing writes, NVMe holding Ceph metadata, and HDDs providing bulk capacity

Making HDD-Backed Proxmox Ceph Clusters Fast — NVMe Metadata, One Optane Per Spindle, and What It Costs You

Flash pricing has made all-NVMe hard to justify, and enterprise HDDs are worth another look. Getting acceptable latency out of them on hyper-converged Proxmox Ceph means moving RocksDB off the spindle, pairing each disk with its own Optane through bcache — Optane specifically, because NAND endurance is wrong for that job — and being honest about the failure modes it buys you.

7th August 2026 · 35 min · 7359 words · Damien Dye
An emulated NVMe controller beside a VirtIO SCSI path, with Proxmox unable to see the former

A Virtual NVMe Device in Proxmox — and the Three Things You Give Up

QEMU can emulate a real NVMe controller, so the guest uses its own in-box NVMe driver with no VirtIO media required. It is also unmigratable by design, invisible to Proxmox backups, and no faster than VirtIO SCSI. Here is how to add one and when it is worth it.

7th August 2026 · 10 min · 2025 words · Damien Dye
Eight 512-byte logical blocks mapped onto one 4K physical sector, beside a one-to-one 4K mapping

4Kn, 512e and 512n — Why Native 4K Wins, and What the Emulation Costs

512e drives present 512-byte sectors they do not have, and the firmware makes up the difference on every unaligned write. What that costs on the medium, in the host and in write amplification — why direct synchronous writes are the worst case — and how to convert a fleet to 4Kn.

7th August 2026 · 26 min · 5489 words · Damien Dye
A curve rising from 36 percent of bare metal at queue depth 1 to within a few percent by queue depth 32

PCIe Passthrough Performance on Proxmox VE — The IOMMU Tax and How to Minimise It

Why PCIe devices lose throughput when passed through to a VM via VFIO, and the practical tuning steps that claw most of it back.

7th August 2026 · 19 min · 3994 words · Damien Dye