FlashStore Against Umbrella BlueStore: What croit Changed Under The OSD

Disclosure up front: I work for croit, who wrote this. Everything below is read from the published source and documentation of both projects, linked at the foot of the page, and where croit’s own docs disagree with croit’s own code I say so.

Ceph’s OSD, Rewritten From Scratch

croit has published ceph-rust, a Ceph OSD written from nothing in Rust, with an object store of its own called FlashStore in place of BlueStore¹. It went into public preview on 9th November 2026, with general availability planned for the first quarter of 2027¹. It is not a fork. The project’s own rule is that Ceph’s C++ is read as a reference, written up as a specification in croit’s own words, and then implemented from the specification².

The claim that matters to anyone running Ceph is the drop-in one. Your monitors, your managers, the ceph CLI and your clients stay as they are. A FlashStore OSD joins the same pools and the same placement groups as the classic OSDs already there, and you change one drive at a time¹.

So the question for an operator is practical. If you swap a BlueStore OSD for a FlashStore one, what is different underneath it? Not in the abstract, and not against some BlueStore from a conference talk years ago. Against the BlueStore you actually run today if you are on Umbrella, Ceph 21.2.0, which is the release FlashStore is built and tested against¹.

So that is the comparison here, and only that. Umbrella’s BlueStore as its own source and documentation describe it, against FlashStore as its source and decision records describe it.

It touches anyone with NVMe Ceph clusters, and anyone who has looked at what an OSD costs them in cores. It does not touch HDD clusters at all, and the reason for that comes up below.

What Is In The Repository

The repository is a read-only snapshot of croit’s private development tree. No issue tracker, no pull requests; updates arrive as new snapshot commits¹. That is the preview stage, and it is there for community feedback while croit runs its production test cases internally before marking it safe for use. Open development follows the first production release, as it does on most corporate-sponsored open-source projects. This one is dated 10th November 2026 and carries about 700,000 lines of Rust and over 450 architecture decision records, the ADRs, which are the short documents that record why each design choice was made.

The ADRs are the useful part for a comparison like this. They are blunt, they carry measurements, and they record what was tried and thrown away as well as what stayed.

The licences are worth knowing before you build anything on it. The daemon and the tools are LGPL-3.0-only, the client libraries are MPL-2.0, and the documentation is CC-BY-SA-4.0³. The client libraries went MPL so a Rust program can link them statically without the relinking duties LGPL brings, which is a reasoned choice and written down as such³.

Preview means what it says. croit’s own README says it is for testing and evaluation and “not yet recommended for production data”¹. I would not put customer data on it yet either, and nothing in the rest of this post changes that.

Two Stores, Side By Side

An OSD has two halves. The top half speaks the Ceph protocol, owns placement groups, peers, recovers and scrubs. The bottom half, the object store, turns “write these bytes to this object” into blocks on a drive and promises they survive a power cut.

BlueStore is Umbrella’s bottom half. Its documentation says it “reads and writes to the devices directly in a ‘raw’ fashion”⁴, and that it consumes “raw block devices or partitions”⁵. Object data goes straight to the block device. Everything else, the object metadata, OMAP, and its own bookkeeping, goes into an embedded RocksDB key/value database, and RocksDB sits on BlueFS, a small file system BlueStore carries for it⁵⁶.

FlashStore drops all of that. One copy-on-write B+tree per store shard holds the lot: collections, object metadata, extended attributes, extent maps, OMAP, the PG log, reference counts and even the allocator’s own bitmap⁷. Next to it sits a redo write-ahead log ring and the data region. There is no RocksDB, no BlueFS, and no second database to tune.

The other big split is the drive itself. BlueStore goes through the kernel’s block layer to whatever you give it. FlashStore talks to NVMe either with SPDK, a userspace driver that takes the device away from the kernel through vfio, or through the kernel with io_uring⁸⁹. SPDK is the default backend. The io_uring tier is the one that works where vfio does not, and as such it is the one I ended up on in the Proxmox build.

Where The Metadata Lives

Start here. Most of the rest follows from it.

The defaults on the BlueStore side come straight from Umbrella’s options file: RocksDB as the key/value backend, the hybrid allocator, crc32c checksums⁶¹⁰¹¹. BlueStore’s metadata checksumming “is handled by RocksDB and uses the crc32c algorithm”, and data costs it “a checksum value (usually 4 bytes) for every 4 KB block of data”⁴. On checksums the two stores agree. FlashStore keeps a crc32c per 4 KiB block in the extent map and verifies it on read⁷.

Structure is where they part. BlueStore’s RocksDB is a log-structured merge tree, so it writes, then compacts in the background, rewriting data it already wrote. FlashStore’s tree is updated by copy-on-write and folded to disk at checkpoints: every five seconds while something is unpersisted, or when the WAL ring reaches half full⁷.

A further consequence is that FlashStore has no fast device for metadata, because it does not need one. On Umbrella you can put block.db on a faster drive, and BlueStore will put “as much metadata as it can on the DB device”⁴. FlashStore assumes every drive is already fast NVMe, and keeps its metadata beside the data. That is a fair assumption for the hardware it targets. It is also exactly why there is no HDD story.

OMAP is the one place FlashStore hedges. The tree is the default, but a store can be prepared with an LSM engine of croit’s own, or with RocksDB, behind the same interface¹². The OMAP engine is chosen when the store is created. You do not switch it on a running OSD.

One 4 KiB Write, Two Ways

Here is the difference you would feel, if you feel any.

Take a 4 KiB write to an object on Umbrella with default settings on flash. bluestore_min_alloc_size_ssd is 4 KiB and bluestore_prefer_deferred_size_ssd is 0, so a whole 4 KiB write goes to a newly allocated 4 KiB¹³¹⁴. BlueStore submits that data write, and the transaction sits in STATE_AIO_WAIT until the drive says it is done¹⁵. Only then is the metadata queued for RocksDB, and the RocksDB sync thread flushes the block device before it commits¹⁶. Then the client gets its acknowledgement.

Two trips to the device, in series.

FlashStore sends both at once. The data goes to a fresh 4 KiB extent and the WAL unit goes to the log ring in parallel. The WAL record carries the crc32c of every new data block, so replay after a crash can tell whether the data made it⁷. The acknowledgement goes out, in log order, when the data, the WAL unit and every earlier transaction are durable⁷. The ADR’s own rationale for the design is “parallel commit gives one device latency per ack”⁷.

Two things keep this honest. On a busy OSD both stores batch commits, so the gap under load is smaller than one device latency per write. And a single device trip on NVMe is a few tens of microseconds. Whether you notice it depends on whether your clients are waiting on latency at low queue depth, which databases on RBD often are, or pushing throughput, where it washes out.

croit tried to go further, to one device write per transaction, ZFS-style. They built it, measured it, and threw it out: the gain only showed at queue depth 1 and the complexity was not worth it¹⁷. I rate that more than any benchmark chart. It is a decision record that says “we built this and it was not worth it”, in public.

The Write Under 4 KiB

The smaller write is where the two really diverge.

Say a client overwrites 512 bytes in the middle of a block that already holds data. Umbrella still runs its original write path by default; bluestore_write_v2, the rework, ships off¹⁸. On that path a write smaller than min_alloc_size into a block that is already allocated becomes a deferred write¹⁹. The data rides inside the RocksDB transaction as a key under the L prefix, and only goes to its real home after the client has been told it is safe²⁰²¹.

So the data reaches the drive twice. Umbrella’s documentation does not hide it: “writes smaller than min_alloc_size must first pass through the BlueStore WAL”⁴.

FlashStore never overwrites live data in place, full stop⁷. Objects of up to 2 KiB go further and live entirely in the tree, so one WAL write is the whole commit⁷.

That also means FlashStore has no deferred-write queue to drain, no deferred replay at mount, and no L keys to clean up. It pays somewhere else instead: freed blocks are held until the next checkpoint, so freed space lags by up to one checkpoint interval⁷.

Threads Against Cores

The top half of the OSD differs just as much.

Umbrella’s ceph-osd is a classic threaded server, and every number in the left half of the diagram below is one of its defaults on flash²². An op crosses several of those pools on its way through. They share state behind locks.

FlashStore keeps the whole journey on one pinned, polling core, from the network card to the drive and back²³. Each placement group belongs to exactly one core, and so does everything the PG touches: “no locks, no atomics, no shared mutable state”²⁴. Cross-core traffic goes over single-producer rings, and ownership of the message goes with it.

The store follows the same split. flashstore_store_shards defaults to one shard per core, each with its own WAL ring, tree and allocator, and a PG lives in exactly one shard²⁵⁷. BlueStore is one store per OSD with one RocksDB.

There is a price, and the ADR states it: “idle cores burn CPU (accepted; optional adaptive sleep later)”²³. Later work added adaptive idle and device-timed waits so a core does not spin for the whole device latency at low queue depth⁸, but the principle stands, and flashstore_cores defaults to 8 when you do not pin a range²⁵. You give FlashStore whole cores.

On a dedicated storage node that is fine. On a hyperconverged box where those cores also run VMs, it is the first sum to do.

The Drive Has To Keep Its Promise

BlueStore is written to cope with a drive that lies about when data is safe. FlashStore refuses to deal with that drive at all.

The BlueStore side is the kv sync thread already cited¹⁶. The FlashStore side is the durability ADR and the host check: PLP means the drive reports no volatile write cache, so a completion means the data is on stable media²⁴²⁶.

This is the right call for the target, but be clear about what it rules out, because almost all consumer NVMe has a volatile cache and no PLP, and so does a fair amount of “datacentre” kit sold on price. No PLP, no FlashStore. Whatever else you think of it.

What FlashStore Will Not Run On

Put the requirements side by side and the target market is obvious.

The BlueStore column is Umbrella’s own: ceph-volume encrypts logical volumes “using dmcrypt by specifying the --dmcrypt flag”²⁷, and ms_bind_msgr1 still defaults to true²⁸. The FlashStore column comes from its hardware reference and its scope decision²⁶²⁹.

Two of those deserve more than a table row.

Encryption at rest is out, with a reason: SPDK bypasses dm-crypt, and in-store encryption is not planned. The ADR’s answer is self-encrypting NVMe drives, or encrypting above the store at the client or RGW²⁹. If you have a compliance requirement that names dm-crypt on the OSD, that is a stopper today.

msgr1 is out as an accepted risk. Kernel clients older than 5.11 and pre-Nautilus librados cannot talk to a FlashStore OSD at all²⁹. Most estates are long past that. Some are not, and it is the old forgotten box in the corner that finds out.

It Tells The Monitor It Is BlueStore

This is the one that will catch people.

By default, a FlashStore OSD boots and tells the monitors osd_objectstore=bluestore³⁰. It is not hiding. It adds its own flashstore_version and flashstore_build keys next to it³⁰. The reason is compatibility: an Umbrella monitor only allows features such as EC overwrites on a pool when every OSD in it reports BlueStore, and manager modules like Prometheus and telemetry branch on the same value³⁰. The ADR is explicit that bluestore here “is a compat value for mon/mgr feature gates, not a product name”³⁰.

There is a switch, flashstore_report_objectstore=flashstore, which renames everything: the metadata, the bluestore_bdev_* keys, the perf logger, the tell commands and the BLUESTORE_* health codes. It needs patched monitors, and the default stays bluestore²⁵³⁰. croit’s own Ceph builds carry those patches, so on them FlashStore reports itself as flashstore and none of what follows applies. The bluestore name is there for the stock monitors and managers people run in production today.

I understand why. A monitor gate keyed on a string is Ceph’s design, not croit’s, and refusing to say bluestore would switch off features on an unpatched cluster. That is the trap on a stock cluster. Every dashboard, every inventory script and every ceph osd count-metadata osd_objectstore you run will count FlashStore OSDs as BlueStore. During a migration, when you most need to know which OSDs are which, the obvious field tells you nowt.

So check flashstore_version in ceph osd metadata, not osd_objectstore. Put that in your runbook before the first FlashStore OSD goes in.

Same Knobs, Different Behaviour

FlashStore reads a lot of Ceph’s existing options, and that is mostly good news. Not every option means what it meant on Umbrella, and the ones that do not are where a tuned cluster gets surprised.

Compression matches. BlueStore’s default mode is none and its default algorithm is snappy³²³³; FlashStore implements the same four modes, the same pool options and the same 64 KiB chunking, and the ADR notes that a 50% compressible 64 KiB chunk takes the same space on both³⁴. It was not always going to be there. The ADR file is still named no-store-compression, from the original decision, overturned when croit set a mandate of full functional parity with classic Ceph³⁴.

Memory is not parity. Umbrella autotunes BlueStore’s caches against osd_memory_target while it runs³⁵³⁶. FlashStore reads the target once at start, splits it over its shards, and a change made with ceph config set takes effect at the next restart³⁷. It also does not hold the process to the target: hugepages, network buffers and PG state sit outside it³⁷. So osd_memory_target sizes FlashStore’s caches. It does not cap the OSD’s memory the way you may be used to.

And mClock is ignored outright³⁷. Not tuned down. Ignored. There is no op scheduler to configure, because each core owns its PGs. Recovery and backfill are held back by osd_max_backfills, osd_recovery_max_active and the sleep options instead³⁷. If you tuned mClock profiles for a rebuild on Umbrella, those settings do nothing on a FlashStore OSD, and nothing will warn you.

There Is No Conversion, In Either Direction

You cannot convert a BlueStore OSD into a FlashStore one. croit’s migration guide is plain about it: “There is no in-place conversion of a BlueStore device”³⁸. It offers two orders instead, and both are backfill³⁸:

Nor does it get easier later. FlashStore’s on-disk format has feature bits, and a mount refuses an unknown bit: “Every format change means mkfs; unknown bits refuse mount (no downgrade)”⁷. So a format change in a later FlashStore release is the same drain-and-refill as the first migration.

One thing the documentation gets wrong. The migration guide’s checklist says to make sure “the pools do not use erasure coding on croit-flashstore-osd OSDs”³⁸. The erasure-coding guide in the same snapshot says FlashStore serves legacy EC pools by default, and the code’s own default for optimised EC is all³⁹⁴⁰. One of those is stale, and from the code it is the migration guide. I would still read the EC guide in full before mixing EC pools across both kinds of OSD, because the failure it describes is a PG that degrades and a backfill that stalls³⁹.

A Store Built For The Drives We Actually Buy

BlueStore was built to run on everything. Spinning disks, cheap SSDs, NVMe, partitions, LVM, encrypted volumes, a fast DB device in front of a slow data device. That breadth is why it became the default, and it is why it carries a deferred write path, a flush before every commit, a key/value database on a file system of its own, and a pool of threads to keep all that moving. Every one of those exists because some drive somewhere needed it.

FlashStore starts from the opposite end. It assumes the drive is NVMe, that it has power-loss protection, and that it is fast enough that the software is now the bottleneck. Then it throws out everything that only existed for the other drives.

Is that the right bet? For new all-flash clusters, built on proper enterprise NVMe, yes. That is where Ceph clusters are going, and an OSD that spends its cores on locks, handovers and a database compacting behind it is paying, every second of every day, for flexibility a cluster like that will never use.

But the bet has a cost, and I would rather you saw it before you buy into it. You give up HDD, dm-crypt, msgr1, live memory tuning and mClock. On a stock cluster you get an OSD that tells your monitors it is BlueStore, because today’s monitors and managers leave it no other way; croit’s own builds will not need that. You get a format you cannot convert into, and cannot convert between versions of either. And you get it in preview, from a single vendor, which happens to employ me.

So treat the source as the evidence, not this post and not croit’s benchmark report. The decision records are public and they say what was measured and what was thrown away. Read them, build a test cluster on the kit you would actually deploy, and measure your own workload. If it does what the ADRs say it does on your drives, you will know. If it does not, you will know that too, and you will have found out on a test cluster.