FlashStore Against Crimson: Two Rewrites Of The Same OSD

Disclosure up front: I work for croit, who wrote FlashStore. Everything below is read from Ceph’s source and documentation at v21.2.0 and from croit’s published ceph-rust snapshot, all linked at the foot of the page. Where croit’s own reasoning is on one side of an argument, I say whose reasoning it is.

Two Rewrites Of The Same Daemon

The classic Ceph OSD is a threaded C++ server that grew up on spinning disks. On fast NVMe and fast networks, it spends a lot of its CPU on things that are not storing your data: handing ops between thread pools, taking locks, waiting on other threads. Two projects have set out to fix that by rewriting the OSD outright.

One is Crimson, Ceph’s own. Umbrella’s documentation calls it “the next generation ceph-osd”, built “to deliver enhanced performance on fast network and storage devices”¹. The other is croit’s FlashStore OSD, from the ceph-rust project, in public preview since 9th November 2026².

Both say they are drop-in replacements. Both say they are for NVMe. Both are in preview, and neither should hold production data today. So if you run NVMe Ceph and you are wondering which one to watch, it is worth knowing where they actually differ, because it is not where the headlines would suggest.

This compares them as you would find them on Ceph 21.2.0, Umbrella: Crimson as that release documents and ships it, against FlashStore, which is built and tested against that same release². It follows on from FlashStore against Umbrella BlueStore, and anything already covered there about FlashStore is taken as read.

Where Each One Comes From

The first difference is who builds it and how.

Crimson lives in Ceph’s own repository and is built by Ceph’s own developers. It is written on Seastar, a C++ framework for thread-per-core servers that came out of ScyllaDB³. The Umbrella docs say plainly that it “is in a tech-preview stage and is not suitable for production use”¹.

FlashStore is written from scratch in Rust and links no Ceph code at all; Ceph’s C++ is read as a reference and re-implemented from croit’s own specifications⁴. Its README says it is for testing and evaluation and “not yet recommended for production data”, with general availability planned for the first quarter of 2027². The public repository is a read-only snapshot: “no issue tracker, no contribution workflow”².

That row is worth reading carefully. Crimson is developed in the open, and anyone can read the pull requests and argue on them. FlashStore, for now, arrives as finished snapshots from one company: you can read every line, but you cannot send it a patch yet. That is the preview stage, not the plan. The preview is there to gather community feedback while croit runs its production test cases internally, before it marks the OSD safe for use. Open development follows that first production release, the same route most corporate-sponsored open-source projects take. That comes from inside croit rather than from a published roadmap, so hold them to it.

croit did look at forking Crimson and wrote down why it did not. Its ADR says Crimson “tries to support everything (HDD, alienstore/BlueStore, all pool types)”, and adds “Complex code is never fast”⁴. That is croit’s opinion of Crimson, not a measurement, and I would treat it as such. The same ADR also rejected a fork partly over the LGPL⁴, and croit then licensed its own daemon LGPL-3.0 eight days later⁵. So the licence was never really the reason.

Memory Safety: Rust Against C++

An OSD is the code that holds your data. A use-after-free or a buffer overrun in it is not a crash you restart from; it is corrupted data on disk, copied to every replica. That is exactly the class of bug the Western cyber agencies have spent the last three years telling the industry to design out.

The joint guidance says it plainly: “Programming languages such as C and C++ are examples of memory unsafe programming languages”⁶. CISA and the FBI go further, and call “the development of new product lines for use in service of critical infrastructure or NCFs in a memory-unsafe language (e.g., C or C++)” dangerous when a memory-safe alternative could be used⁷. That is guidance, not law. But storage under critical services is precisely the code it is aimed at, and both of these are next-generation OSDs that will be judged against it.

FlashStore meets it. It is written in Rust, and croit’s ADR gives the reason in one line: “Storage = correctness-critical; memory-safety bugs = data loss”⁸. The compiler checks memory safety everywhere except code marked unsafe, and the same ADR confines that to the FFI, DMA, NVMe and network-driver modules, each with a SAFETY comment⁸. I counted it in the snapshot: 1,216 lines that mention unsafe out of 696,125 lines of Rust, about 0.17%, and they sit in the RDMA, runtime, NVMe and store crates where the code talks to hardware. The joint guidance names that escape hatch itself, noting that Rust “can contain the ‘unsafe’ keyword”⁶, so that is the honest measure: not zero, but small, fenced off, and in the places you would expect. The C it does call, SPDK and DPDK, sits behind those same boundaries⁸.

Crimson does not. It is a new OSD written in C++ on a C++ framework¹³, which is the language the agencies above name as memory unsafe. Every line of it is outside what a compiler can check for memory safety, and in its default configuration it hands the store to BlueStore, which is C++ too.

Ceph’s developers did not choose C++ carelessly; Crimson grew out of a C++ code base, and rewriting in place meant staying in it. That is a reasonable engineering path. It still leaves the next-generation Ceph OSD written in the one kind of language the agencies of five countries have asked new critical code to leave behind.

How Each Joins A Cluster

This is the biggest practical difference between the two, and it is not about performance at all.

A FlashStore OSD joins the pools you already have. It sits in the same placement groups beside classic BlueStore OSDs, and you swap one drive at a time².

A Crimson OSD does not. Not even close. Umbrella’s docs say: “Crimson OSD won’t instantiate PGs from non-Crimson pools”¹. The code does exactly that. When the monitor asks a Crimson OSD to create a PG in a pool without the crimson flag, it logs “pool lacks crimson flag” and cancels the creation⁹.

So a Crimson trial means new pools, created with the crimson flag, and your data moved into them. And those pools come with conditions the monitor enforces:

FlashStore serves classic pools, both replicated and erasure-coded, and declines the EC configurations it does not support instead of refusing them at the monitor¹⁴. Neither approach is wrong. But one lets you test on a copy of your real pool layout, and the other makes you build a separate one first.

No Way Across For The Pools You Already Have

So can you turn an existing pool into a Crimson pool, and move it over a drive at a time? No.

Umbrella’s docs say Crimson is “aiming to allow seamless migration from existing ceph-osd deployments”¹. Aiming. The release does not do it, and the source shows why it cannot. The crimson flag is an option on ceph osd pool create and nowhere else¹⁵. ceph osd pool set has no crimson variable to set on a pool that already exists¹⁶. And a Crimson OSD ignores every PG in a pool without the flag, as above⁹. I found no conversion tool in the source or the docs.

That makes a move to Crimson a data copy you run yourself: RBD images can be live-migrated “between different pools”¹⁷, and object or file data needs whatever your applications use to copy it. Every client then has to be pointed at the new pool.

FlashStore is compatible with the pools you run today. You prepare it on a drive, the cluster backfills the placement groups it is given, and your clients never notice the pool changed under them, because it did not¹⁸. That is the whole difference between a rewrite that joins your cluster and one that sits beside it.

What Each Tells The Monitor

Crimson announces itself. At boot it sends osd_type=crimson¹⁹, and the monitor refuses to let it in unless the allow_crimson OSDMap flag is set, with a log line that says crimson-osd “is considered unstable and may result in crashes or data loss”²⁰. Getting that far takes four commands from the docs, the first of which sets an option called enable_experimental_unrecoverable_data_corrupting_features¹.

FlashStore does the opposite. It sends no osd_type at all and reports osd_objectstore=bluestore, so the monitor sees a classic OSD and asks no questions²¹. The previous post covers why: Umbrella’s monitor gates features such as EC overwrites on every OSD in a pool reporting BlueStore²¹. That is for the stock monitors and managers in production today. On croit’s own Ceph builds, where they are patched to know it, FlashStore reports itself as flashstore²¹.

I would rather software told me what it was. Crimson does. Its way is the honest one, and the flag name leaves you in no doubt about what you are agreeing to. FlashStore’s way is what makes drop-in work on the stock monitors and managers people run now, and it is a compatibility measure, not the end state. Both are deliberate, and both are written down. Just know which one you are dealing with when you count your OSDs.

Cores: Reactors Against Pinned Polling Threads

Both projects threw out the thread pools. Good. Both run one thread per core and shard their work by core. The difference is in how much they ask of the operator and how hard they hold on to the core.

On Crimson, crimson_cpu_num “does not have a default value”, must be set before the OSDs are deployed, and “cannot be changed after deployment”¹. It does not pin threads unless you also set crimson_cpu_set, which the docs describe as “generally less recommended due to its complexity”¹. The reactor polls for 200 µs when idle before it goes back to sleep, and a full poll mode exists “at the expense of 100% cpu usage”, off by default²². SeaStore also refuses a change in reactor count after mkfs²³.

FlashStore pins by design, polls by design, and backs off when idle²⁴²⁵. Its store shard count is fixed when the store is prepared, the same constraint in a different place²⁶.

In practice, both want whole cores and both want you to size them before you deploy. Crimson makes the polling a choice you opt into. FlashStore makes it the default and lets the idle logic claw some of the CPU back.

Crimson’s Default Store Is BlueStore

This is the part that surprised me. It should not have.

Crimson has two kinds of object store. Native ones run inside the Seastar reactor; non-native ones run through “a thread pool proxy” to “alien threads: worker threads not managed by Seastar”¹. SeaStore is the native one. BlueStore is non-native. And BlueStore is the default.

The docs are blunt about it. SeaStore “is the primary native object store for Crimson OSD, though it is not the default as the support is in early stages”¹. Cephadm deploys Crimson with --objectstore bluestore unless told otherwise, and even then SeaStore cannot be deployed with --method raw; you are told to use LVM¹.

So a Crimson OSD deployed today with defaults runs a Seastar front end on top of the same BlueStore, RocksDB and BlueFS that the classic OSD uses, with crimson_bluestore_num_threads threads set aside for it²². On the store side, nowt has changed from the classic OSD. That is a sensible way to build a migration path. It is also why comparing “Crimson” against FlashStore on performance means very little unless you say which store Crimson was running.

The fair comparison for the store is SeaStore against FlashStore. That is the rest of this post.

SeaStore And FlashStore: Where The Metadata Lives

Both stores were written for NVMe from the start, both are sharded by core, and both keep their metadata in B-trees of their own instead of RocksDB. Their layouts are still very different.

SeaStore gives each reactor its own root²⁷. Under it sit an onode tree, an LBA tree and a backref tree. Each object’s onode then points at its own OMAP B-tree and its own XATTR B-tree, and at data extents with logical addresses²⁷. The LBA tree maps those logical addresses to physical ones, so a block can move on disk without the object’s own trees changing; the docs call that indirection the reason most extents “can be relocated freely”²⁷.

FlashStore puts everything in one copy-on-write B+tree per shard: onodes, xattrs, OMAP, extent maps with a crc32c per 4 KiB block, the PG log, and the allocator’s bitmap²⁸. Extent maps point straight at physical blocks. There is no logical layer in between.

Both choices have a reason. SeaStore’s indirection is what makes its cleaner cheap: it can move live data without rewriting every tree that points at it. FlashStore does not need that, because it does not have a cleaner, as below. Its ADR lists “Per-object omap subtree/root” among the alternatives it considered and rejected²⁸.

One difference worth knowing on checksums: SeaStore calculates its crc32c over a whole cached extent²⁹, where FlashStore keeps one per 4 KiB block²⁸.

The Order A Write Commits In

SeaStore’s default device type is SSD, which uses the segmented backend³⁰. On that backend, a transaction is a journal record carrying deltas, and two things can happen to the data.

Small, short-lived extents can ride inline in the journal record, and then the record is the whole commit²⁷. Larger or longer-lived extents go to separate out-of-line segments, and then the docs set the order: the “final record with the deltas should be written only once the other blocks are persisted”²⁷. That is two writes in series, the same shape as BlueStore’s data-then-RocksDB commit.

FlashStore writes the data extent and the WAL unit at the same time, puts the crc32c of the new data in the WAL record so replay can check it, and acknowledges when both are durable²⁸. One device trip, whatever the size.

How much that matters depends on your workload, exactly as in the BlueStore comparison. At low queue depth, where clients wait on every write, a second trip is real latency. Under load, batching hides most of it.

Getting Space Back

The other half of a log-structured design is cleaning up.

SeaStore on the segmented backend only ever writes to open segments. Before a segment can be reused, its live blocks have to be rewritten somewhere else, and the design doc says what that costs: “we must ‘clean’ one byte of an existing segment for every byte written at steady state”²⁷. It mixes some of that cleaning into ongoing writes, to avoid spikes, and does the rest in the background when the drive is quieter²⁷. The segment size defaults to 64 MiB³¹.

SeaStore does have a second backend for NVMe, the random block manager, which the docs say is “Preferred for fast NVMe devices where overwrites are efficient enough that log structured updates aren’t worth the overhead”²⁷. It is not the default. You choose it with seastore_main_device_type³⁰.

FlashStore has no segment cleaner. None. A replaced block goes back to the shard’s allocator once a checkpoint has passed it, which happens every five seconds while there is unpersisted work, or sooner when the WAL ring is half full²⁸. Its design document rejected a log-structured layout outright²⁸. That does not mean it has no background work: it still merges and compacts tree pages and defragments free space, paced under client I/O²⁸. But it never copies your live data to make room.

On flash, which already runs its own garbage collection inside the drive, I would rather the OSD did not run a second one on top. SeaStore’s own docs seem to agree, given what they say about the RBM backend. The difference is that FlashStore made that the only option and SeaStore made it the non-default one.

What Each Will Run On

Crimson keeps the breadth. The docs say SeaStore “may not be suitable for traditional HDDs”, but that “Crimson will continue to support BlueStore ensuring compatibility with HDDs and slower SSDs”¹, and SeaStore’s design doc still lists an interest in HDDs²⁷. Its messenger refuses v1 outright, “ProtocolV1 is no longer supported”³², the same choice FlashStore made³³. Its docs recommend “one of the latest available builds, as Crimson evolves rapidly”, which in practice means CI images of main¹.

FlashStore narrows everything to NVMe with power-loss protection, and gets its simplicity from that³⁴. Which is exactly the trade croit’s ADR describes, from the other side.

Two Answers To The Same Question

Both projects started from the same observation: the classic OSD wastes fast hardware. They gave very different answers.

Crimson’s answer is the community one. Rewrite the OSD in place, inside Ceph, keep every old store and every old drive working while the new store matures, and let everyone see the work as it happens. That is slow, and it shows. In Umbrella, the default Crimson OSD still runs BlueStore through a thread pool, its own pools are fenced off from yours, and turning it on means typing the words “unrecoverable data corrupting”. But it belongs to everyone who runs Ceph, and nobody can take it away from you.

FlashStore’s answer is the vendor one. Pick a narrower target, NVMe with power-loss protection, throw out everything that only existed for other hardware, and ship a store that drops into the pools you already have. That is faster, and it shows too. But it comes from one company, as read-only snapshots until its first production release, and on a stock cluster it gets in by telling the monitor it is BlueStore.

I work for that company, so weigh what follows accordingly. croit made the right technical bet for all-flash clusters. Crimson made the right bet for Ceph as a project. Those are not the same thing. A project that serves everyone cannot drop HDDs. A vendor can, and should, if that is what its customers buy.

What I would not do is pick between them on a benchmark chart from either side. Crimson’s numbers depend on which of its three stores was running; FlashStore’s come from croit’s own lab. Both projects publish their design documents. Read them, then put both on the drives you would actually deploy and run your own workload. The one that still does what its documents say when it is your data on the line is the one worth waiting for.