Sorting TSC When Your Firmware Is Broken

Update, 11th October 2026, 16:30: I’ve since flashed BIOS 1.30. It does not fix the offset. tscsync still lines the cores up, but on two sync boots the kernel dropped the TSC anyway, through its clocksource watchdog rather than the warp check: BIOS 1.30: The Sync Holds, The TSC Still Goes. With tsc=nowatchdog added, the TSC held, and five minutes measured against the real-time clock found no warps and no drift: tsc=nowatchdog: Five Minutes, No Warps. One boot is not a run of boots, as such I’m calling it holding, not fixed.

Every Boot, The Same Warning

This was sat in the kernel log of my ThinkPad E595 on every single boot:

TSC synchronization [CPU#0 -> CPU#2]:
Measured 6601281288 cycles TSC warp between CPUs, turning off TSC clock.
tsc: Marking TSC unstable due to check_tsc_sync_source failed
clocksource: Switched to clocksource hpet

Nothing crashed. Nothing popped up on the desktop. The machine quietly ran its whole clock on a slower timer chip from the moment it booted, and it would have gone on doing that forever if I had not been reading the journal for something else.

The Time Stamp Counter (TSC) is a counter inside every CPU core that ticks at a fixed rate. Reading it is a single instruction. Linux uses it as its main clock, but only if every core agrees on the count¹. If one core is behind, the kernel cannot trust it and falls back to the High Precision Event Timer (HPET), a separate timer device. tscsync’s author measured a clock read at about 1.2 µs on HPET against tens of nanoseconds on the TSC, and the kernel and desktop read the clock tens of thousands of times a second¹. That is latency and battery, paid on every boot, for nowt.

So how far behind was the core? Over three seconds. On a 2.1 GHz laptop that is 6.6 billion cycles.

What The Firmware Hands Over

The message names CPU 0 and CPU 2, but the fault is not CPU 2. Measured by tscsync from the boot loader, and by the kernel once it starts, the picture is the same every time:

Seven cores agree. One is three seconds late. The firmware brings the boot processor up, does its work on it, and hands over to the boot loader without ever bringing that one counter back into line with the rest.

Why can Linux not just fix it? On Intel CPUs there is a register, IA32_TSC_ADJUST, that lets the kernel nudge a core’s counter, and the kernel’s sync code uses it when it is there². AMD Zen CPUs do not have it. Without it, nothing in Linux can repair the offset, and the in-kernel attempt to write the counters directly (tsc=directsync) was rejected upstream as unreliable³.

So the kernel does the only honest thing left. It measures, it finds time going backwards between two cores, it prints the warning and it stops using the TSC².

Forty Boots, Forty Warps

One boot is an anecdote. The journal on this machine goes back to 30th May 2026, so I asked it.

Every boot. Cold boot, reboot, after a power-off long enough that you would expect anything stale to be gone. The amount moves about, which fits the tscsync author’s own finding on a Lenovo Legion: there, the warp scales almost perfectly with how long the machine spends in firmware before the boot loader starts³. A boot that sits on the logo longer comes out further behind.

The tempting fix is tsc=reliable on the kernel command line, which tells Linux to stop checking. Do not do it. The cores really are three seconds apart, and turning the check off does not change that; it just lets the kernel use a clock that jumps backwards when a task moves between cores. The tscsync README says the same, and names the audio and video stutter that follows¹.

Can I Ask Why This Still Ships?

I am not asking who. I am asking what in Lenovo’s process let a BIOS go out that leaves a core three seconds behind on every boot.

The answer is in the tscsync design notes, and it is not flattering. Fisher-Price OS (Windows) writes the TSCs itself at boot³. A laptop validated only on that passes, because the operating system quietly tidies up after the firmware. Linux keeps its safety check on and reports what it was given, which is exactly why the fault shows up there and nowhere else.

The awkward part is that this has reportedly been fixed before. Users report that Lenovo fixed the same class of bug by BIOS update on the ThinkPad A485, the T14s Gen 1 AMD and the E495⁴. The E495 is the 14-inch sibling of this machine, and they share one BIOS: Lenovo’s own readme for the package lists both⁵.

My E595 was on 1.28 and warped on every boot. Lenovo has since published 1.29 and 1.30, and neither change list says a word about the TSC; 1.30 fixes “system randomly freeze since BIOS V1.27” and some security issues⁵. So I put 1.30 on and measured again. The first boot on it, with tscsync switched off, logged a 6,126,446,316-cycle warp: 2.92 seconds, CPU#0 -> CPU#2, same as ever. 1.30 does not fix it. The two models share one BIOS, so if the E495 report were right, this machine would be fixed too. The report names no BIOS version and links no logs⁴.

Will Lenovo go back to a 2019 laptop for a fault the operating system they test on hides from them? No.

tscsync, Measured First

tscsync is a small UEFI program that runs from GRUB or systemd-boot just before Linux. It asks every core for its TSC through the firmware’s own multiprocessor services, works out each core’s offset from the boot processor, moves the lagging counters forward until they agree, then checks the result with a test modelled on the kernel’s own warp check¹. It never writes flash, NVRAM or firmware settings; the counters it changes reset at power-off¹.

That is a lot to hand to a boot loader, as such I read the code first, at commit 230eaa9, before building it. The install was then done the way the project asks:

Measure mode is the bit worth copying into any tool like this. It shows you the fault on your own hardware before it is allowed to change anything, and a hang guard turns it off if a boot never reaches Linux¹.

The first boot in sync mode, straight from tscsync-status:

survey (read-only):
  cpu  1 apic   1: offset 6815086011  rtt 126  BAD
  ...
  cpu  7 apic   7: offset 6815086010  rtt 378  BAD
  0 ok, 7 not ok
moving BSP forward by ~6815086010 cycles to match cpu 7:
  bsp vs cpu  7 apic   0: before 6815086001  after 3  rtt 357  iter 3  warps bsp/ap 0/0  OK
AP pass (corrects any AP still off):
  7 ok, 0 not ok
total 121 ms

And the kernel, a second later:

clocksource: Switched to clocksource tsc-early
tsc: Refined TSC clocksource calibration: 2096.066 MHz
clocksource: Switched to clocksource tsc

No warp line. No “unstable”. /sys/devices/system/clocksource/clocksource0/current_clocksource reads tsc. 121 ms of boot time to get the clock the hardware was meant to have all along. That was on BIOS 1.28. 1.30 has not been as tidy.

The 21 Nanoseconds Linux Overruled

There is one line in that report that looks worse than it is, and it is worth explaining rather than hiding. tscsync’s final verify pass flagged one core:

cpu  7 apic   7: offset 45  rtt 357  warps bsp/ap 0/0  BAD
6 ok, 1 not ok
RESULT: some APs out of sync (Linux will fall back to HPET)

45 cycles is about 21 nanoseconds. tscsync aims for agreement within roughly 10 cycles¹, so by its own target CPU 7 missed, and it said so. Then Linux booted and did not fall back to HPET at all.

The two checks are asking different questions:

The warps bsp/ap 0/0 on that same line is tscsync running the kernel-style test. It saw none. Neither did the kernel.

The project already knows its verdict can be stricter than the kernel’s. An IdeaPad in its hardware list had four cores flagged at 28 to 42 cycles that the kernel accepted, and since 2.1 a slow round trip is reported as UNSURE rather than BAD⁴. The README’s rule for that case is the right one: the kernel’s own check decides, and if the clocksource is tsc, it passed¹. A 357-cycle round trip is not slow on this machine, so it came out BAD instead. A tool that shows its working and then lets the kernel have the final word is the right way round.

BIOS 1.30: The Sync Holds, The TSC Still Goes

On 1.30 I put tscsync back in sync mode and rebooted twice. Both times it did its job. It moved the boot processor forward about 6.17 billion cycles, every core came out within a few dozen cycles, and the kernel logged no warp. Both times Linux still ended up on HPET a few seconds later:

tsc: Detected 2095.890 MHz processor
tsc: Refined TSC clocksource calibration: 2096.053 MHz
clocksource: Watchdog remote CPU 5 read timed out
clocksource: Marking clocksource tsc unstable due to frequency skew
clocksource: Watchdog                    hpet interval:        495136113ns
clocksource: Clocksource                  tsc interval:        496026671ns
tsc: Marking TSC unstable due to clocksource watchdog

This is a different check. The warp test runs as each core comes up and asks whether time ever goes backwards between cores. The clocksource watchdog keeps running after boot. Every half second or so it compares how far the TSC moved with how far HPET moved, and if they disagree by more than a small margin it marks the TSC unstable⁶. Once marked, it stays marked until the next boot.

So is the TSC running at the wrong rate? No. With the machine up, I timed the TSC against the kernel’s HPET-based raw clock for 30 seconds in 5-second windows. Every window came out at 2,096.062 MHz, about 4 parts per million from the kernel’s own calibration. A real rate fault does not swing from slow to fast between two boots and then vanish. Summat disturbs one half-second window early in boot, and the remote CPU 5 read timed out line just before it points at a core briefly not answering. Firmware stealing time from the CPUs would fit. I haven’t proved it.

Two boots is all the evidence there is. I can’t say yet whether it’s 1.30, chance, or something tscsync’s write sets off. The one sync boot on 1.28 kept the TSC and the two on 1.30 lost it, and that isn’t enough to blame the BIOS.

There is a narrower kernel option than tsc=reliable for this case. tsc=nowatchdog switches off the clocksource watchdog for the TSC and leaves the boot-time warp check alone⁷. If tscsync ever failed to sync, the kernel would still catch the warp and fall back to HPET. That was the next test. The numbers, and some suggested changes to tscsync’s own reporting, are on the project’s issue tracker⁸.

tsc=nowatchdog: Five Minutes, No Warps

I added tsc=nowatchdog and rebooted. tscsync moved the boot processor forward 6,207,104,421 cycles, the kernel logged no warp and no skew, and current_clocksource read tsc. First boot on 1.30 to keep it.

That only proves the watchdog was what dropped the TSC. It does not prove the TSC is safe to run on, and with the watchdog off, nothing in the kernel is watching any more. So I watched it myself.

My earlier rate check was no use here. It timed the TSC against CLOCK_MONOTONIC_RAW, which was fine while the kernel ran on HPET. On the TSC, that clock is built from the TSC, so the test would compare the counter with itself. HPET cannot be read from user space either, because Fedora’s kernel is built without CONFIG_HPET_MMAP. The real-time clock (RTC) can. It runs off its own crystal, and the kernel will wake a program on every tick of its seconds counter⁹.

So I wrote a small check, tsccheck. It pins itself to CPU 0 and runs four tests every 15 seconds:

The round trip is the core of it. CPU 0 reads its counter and passes it over, the other core reads its own and passes that back, and CPU 0 reads again. If the cores agree, the middle reading has to land between the other two:

The loop that does it:

for (int i = 0; i < ITERS; i++) {
	uint64_t t0 = rdtsc_o();                 /* CPU 0 */
	atomic_store(&pp.t0, t0);
	atomic_store(&pp.seq, 2 * i + 1);        /* the other core reads t1 now */
	while (atomic_load(&pp.seq) != 2 * i + 2)
		;
	uint64_t t2 = rdtsc_o();                 /* CPU 0 again */
	uint64_t t1 = atomic_load(&pp.t1);
	if (t1 < t0 || t2 < t1)
		r.warps++;                           /* time went backwards */
	if (i > 100 && t2 - t0 < r.rtt) {
		r.rtt = t2 - t0;
		r.offset = (int64_t)(t1 - (t0 + (t2 - t0) / 2));
	}
}

The whole program is plain C for x86-64 Linux, with nothing tied to tscsync or this laptop. It needs root for /dev/rtc0, and the third argument is your own kernel’s figure from journalctl -k -b | grep 'Refined TSC':

gcc -O2 -Wall -pthread -o tsccheck tsccheck.c
sudo ./tsccheck 15 20 2096.073

SHA-256 02eba1966a0097196acd5dfdcad9c4b315cafc1854c60f34c5b56080e6c38408.

Twenty samples over five minutes, straight after boot:

The windows look worse than they are. A late RTC tick makes one window long and the next one short, so the errors come in pairs that cancel: 61 parts per million (ppm) slow at 30 seconds, 15 ppm fast at 60. Drift doesn’t cancel itself out. The figure over the whole run barely moved.

The whole-run figure sits 22 ppm under the kernel’s 2,096.073 MHz for the full five minutes. That is the gap between two crystals, and it held still. chrony, steering the clock against internet time servers, had it 16 to 19 ppm slow¹⁰. Same direction, same size.

The offsets hop between the same few values, 10, 21 and 33 cycles either side, with no core walking away. That is noise from a round trip of about 550 cycles. tscsync’s verify would still call some of them BAD. The kernel would not.

Did the TSC hold? Yes, on every measure I had. Five minutes after one boot is still not a run of boots. Next is the same check after a long uptime, and again over a few more restarts.

The Fix Belongs In The Firmware

tscsync should not need to exist. Its own README says so: firmware vendors are welcome to the approach and the code, “which is where this should be fixed”¹. Lenovo has the source. The tscsync author had only what their laptop would tell them, and 80 boots of logs¹.

That is the bit that sticks. The fix took a stranger outside the company to measure, diagnose and work round, while the firm that sold the hardware validated it against the one operating system that hides the fault. A machine you cannot repair because its firmware is closed is not really yours, it is on loan with conditions. Open firmware with signed updates would have let this be fixed once, properly, by anybody who cared enough to look.

Meantime, good work from the tscsync author. Measure mode first, a hang guard, the numbers in the open and an honest verdict even when it disagrees with the kernel. That is how low-level tools should be built.

Every KVM Host With The Same Firmware

On a laptop this costs some latency and some battery. On a hypervisor every guest pays it as well, and none of what follows is down to Proxmox. It all lives in the Linux kernel’s KVM code, so it is the same on any host built on KVM, whatever sits on top: Proxmox VE, oVirt, OpenStack, or plain QEMU and libvirt¹¹.

What KVM Does When The Host Loses Its TSC

KVM only hands its guests a stable clock when the host itself runs on the TSC. In the kernel source the master clock, one shared time reference for every vCPU in a VM, is switched on only if host_tsc_clocksource is true¹². Drop to HPET and it switches off, and several things change at once:

The last row is the one that scales badly. Without the stable bit, the guest kernel cannot trust each vCPU’s clock to agree with the others, so every read is compared with “last_value, a global accumulator” and swapped in with an atomic compare-and-exchange if it is newer¹⁴. Every vCPU in the guest writes to that one value. The more vCPUs read the clock, the more they queue on it.

None of it shows up as an error. The guest’s clocksource still reads kvm-clock, and nothing in the guest will tell you why it got slower.

What It Costs, Measured

I measured it on this laptop rather than leave it as theory. The host’s clocksource can be switched while it runs, which turns KVM’s master clock off just as the boot-time fallback does. A throwaway guest booted the host’s own kernel with 4 vCPUs and a tiny initramfs whose only program times clock_gettime(). Two runs each way, host first, then guest:

The guest does not pay HPET’s price directly, because kvm-clock still reads the TSC underneath. What it loses is the user-space read and the independence between vCPUs, and that alone is seven times slower. With all four vCPUs reading at once the cost goes up again, which is that shared global value at work. A guest with 32 or 64 vCPUs queues harder on it. I haven’t measured that.

Put a number on it. A guest that reads the clock a million times a second on one vCPU spends 2.8% of that vCPU doing it on a healthy host, and 19% on this one. Databases, the Java Virtual Machine (JVM) and anything else that timestamps its work read the clock a lot.

This run also understates it. Switching clocksource by hand turns the master clock off but does not mark the TSC unstable, so the offset rewrites in the table above never ran. The boot-time fallback on this E595 marks it unstable, and a host in that state pays both.

Checking A Host, And The Limits

I covered why guest clocks are fragile in the first place in the ptp_kvm post; this is the same problem arriving from underneath. Check the host, not the guest:

journalctl -k -b | grep -iE 'TSC warp|TSC unstable'
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

tscsync runs from GRUB or systemd-boot¹, which covers most KVM hosts. Proxmox VE uses GRUB, or systemd-boot on a UEFI install with ZFS as root and Secure Boot off¹⁵. Be honest about the limits before it goes near a production node, though.

A firmware fault that costs a laptop some battery costs a hypervisor a slice of every guest it runs, every second it is up, and the guest owners have no way to see it. That is the case for checking current_clocksource on every host you run, whoever made it.