Webex Is Looking For Your Certificates On A Cisco Build Server

Webex puts a yellow banner across the top of the window. Offline - No internet connection. Phone services show as disconnected, nothing syncs, and you cannot join the meeting that started ninety seconds ago.

That is not an inconvenience when the thing is your work phone. Webex is where my calls come in, and my VoIP line runs through it, so a client that will not authenticate is a desk phone that will not ring, a meeting I am not in, and a colleague who gets voicemail. It stopped me working. Not degraded, not slow. Stopped.

And none of it was mine to cause or to prevent. A working day went sideways because of quality control that was not applied, at a supplier that gets paid, on a product sold with a support contract against a platform its own requirements page says is supported. I did not misconfigure anything. I installed the vendor’s package, from the vendor’s repository, on a platform the vendor lists, and it could not open a TLS connection. Then I spent an evening of my own time finding out why, which is time the people who signed the package did not spend.

The machine is not offline. The browser next to it is loading pages. Your mail is arriving, your terminal is pulling from a remote, and if you ask the operating system whether it can reach the internet it says yes. Webex itself agrees, in its own log, eleven seconds before it tells you otherwise.

What has actually happened is that the copy of OpenSSL Cisco ship inside Webex cannot find a single certificate authority, because it was compiled to look for them in /workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl. That is a directory on a Cisco build container. It has never existed on your computer and it never will. Every TLS connection the application makes fails at certificate verification, the login token cannot be refreshed, a fifteen-second timer runs out, and the interface reaches for the only explanation it has a string for.

So the banner is wrong in a specific and unhelpful way. It names your network. The fault is a path inside their build.

This is version 46.8.0.35631, on Fedora 44, kernel 7.2.4. It is a supported platform. Cisco publish Linux system requirements and ship a signed .rpm, webex-46.8.0.35631-1.x86_64¹. What follows is how to prove it in about ten minutes, why none of the obvious fixes work, the one that does, and then the part that matters more than any of it: this is not a subtle bug. It is a build that nobody ever ran on a machine that had not built it.

What The Banner Is Actually Telling You

Webex runs its connectivity display off a composite state machine, seven sub-machines each with its own timers. At startup they initialise like this:

ConnectivityStateMachine::ConnectivityBanner - Initializing with state: Connected
ConnectivityStateMachine::Network        - Initializing with state: NoNetwork
ConnectivityStateMachine::Services       - Initializing with state: Connected
ConnectivityStateMachine::Mercury        - Initializing with state: Disconnected
ConnectivityStateMachine::Authentication - Initializing with state: UserNotAuthenticated
ConnectivityStateMachine::Syncing        - Initializing with state: Synced
ConnectivityStateMachine::Survivability  - Initializing with state: SurvivabilityHide

Forty milliseconds later the operating system reports back, and the application writes it down:

NetworkManagerPowerNetworkWatcher.cpp:93 onConnectivityCheckSuccess::
  The host is connected to a network, that appears to be able to reach the full Internet.

That line is in the same file, in the same session, as the banner claiming there is no internet connection. The application knew. It had the answer in hand at 08:25:12.131 and showed the opposite at 08:25:27.132.

Between those two moments, this:

The Authentication sub-machine never leaves UserNotAuthenticated, so Services derives itself as disconnected, so the banner fires. Every stage of that is correct behaviour given the input. The input is wrong, and the input is a number: 167772294, on every failed request, from the first one to the last.

That number is the whole post. Hold on to it.

Three Paths, None Of Which Exist

Webex does not use the system OpenSSL. It brings its own, along with its own libcurl, and the libcurl links against the bundled one rather than yours:

$ ldd /opt/Webex/bin/libcurl.so | grep -E 'ssl|crypto'
    libssl.so.3    => /opt/Webex/bin/../lib/libssl.so.3
    libcrypto.so.3 => /opt/Webex/bin/../lib/libcrypto.so.3

Fine so far. Bundling a TLS library is a defensible choice and plenty of vendors do it. What matters is what got baked into it, and OpenSSL will tell you if you ask:

$ strings /opt/Webex/lib/libcrypto.so.3 | grep -E 'OPENSSLDIR|ENGINESDIR|MODULESDIR'
OPENSSLDIR: "/workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl"
ENGINESDIR: "/workspace/.conan2/p/b/cisco8ee8b59cf93de/p/lib/engines-3"
MODULESDIR: "/workspace/.conan2/p/b/cisco8ee8b59cf93de/p/lib/ossl-modules"

Three of them. Not one slipped setting. The whole install prefix, carried through from the machine that compiled it, in a library that reaches the customer as a signed package on a supported platform.

OPENSSLDIR is set once, at configure time, with --openssldir, and OpenSSL’s own build documentation is blunt about what it is for: “Directory for OpenSSL configuration files, and also the default certificate and key store.”² Everything downstream of trust hangs off it. The default CA file is cert.pem inside it and the default CA directory is certs inside it³. /workspace is a Conan build cache. Conan is the C++ package manager Cisco build with, and it stores each package under a hash of its build inputs⁴. The hash cisco8ee8b59cf93de is a fact about a container that was probably deleted minutes after the build finished.

Ask the library what it is, and it is not even stock OpenSSL:

VERSION              CiscoSSL 3.5.5.8.5.4 27 Jan 2026
BUILT_ON             built on: Wed Feb 25 06:29:18 2026 UTC
PLATFORM             platform: conan-Release-Linux-x86_64-gcc-13
DIR                  OPENSSLDIR: "/workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl"

A maintained in-house fork, with a version scheme of its own, built in February, shipped in August, and still carrying the working directory it was made in.

Do Not Take strings For An Answer. Ask The Library

strings finds text in a file. It does not prove the library uses it. So load the shipped library and ask it directly, which takes about twelve lines of Python and no root:

import ctypes
c = ctypes.CDLL("/opt/Webex/lib/libcrypto.so.3")
for f in ("X509_get_default_cert_file", "X509_get_default_cert_dir",
          "X509_get_default_cert_file_env", "X509_get_default_cert_dir_env"):
    getattr(c, f).restype = ctypes.c_char_p
    print(f"{f:34} {getattr(c, f)().decode()}")

X509_get_default_cert_file         /workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl/cert.pem
X509_get_default_cert_dir          /workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl/certs
X509_get_default_cert_file_env     SSL_CERT_FILE
X509_get_default_cert_dir_env      SSL_CERT_DIR

There it is from the library’s own mouth. When anything in Webex asks this copy of OpenSSL for the default trust store, it is handed a file and a directory that do not exist. That is what SSL_CTX_set_default_verify_paths does, and it is what almost every client does unless it has been told otherwise.

The last two lines are worth noting, because they are the escape hatch: the library will let SSL_CERT_FILE and SSL_CERT_DIR override both⁵. Remember that as well. It becomes important, and not in the way you would expect.

Reproducing The Exact Error Code

Proof by inspection is not proof. Take the shipped libssl.so.3 and libcrypto.so.3, make a real handshake to a real host with nothing but the library’s own defaults, and watch what comes back.

The interesting run is the one where the defaults point at nothing. SSL_CERT_FILE and SSL_CERT_DIR override exactly the two values a missing OPENSSLDIR leaves dangling, so pointing them at a path that does not exist reproduces the shipped condition precisely:

$ SSL_CERT_FILE=/nonexistent/cert.pem SSL_CERT_DIR=/nonexistent/certs python3 tls.py
set_default_verify_paths -> 1
set_fd -> 1
SNI -> 1
SSL_connect -> -1 SSL_get_error -> 1
verify result 20: unable to get local issuer certificate
  err: 0xa000086 error:0A000086:SSL routines::certificate verify failed

0x0A000086 in decimal is 167772294.

That is the number in every failed line of the Webex log, and it did not come from Webex. It came from Cisco’s own TLS library, running outside their application, failing for exactly one reason: it had no trust anchors to check the chain against. Same library, same error, no application in the way.

Point the same two variables at the real Fedora bundle and the same code path completes:

$ SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem python3 tls.py
SSL_connect -> 1 SSL_get_error -> 0
verify result 0: ok

Nothing about the network changed between those two runs. One file path did.

Why Nothing Warned You

Two things conspire to make this silent, and only one of them is Cisco’s fault.

The first is reasonable. A program that ships its own anchors separately should not be forced to care, so the call succeeds, the store is empty, and nothing anywhere in the stack says I have loaded zero certificate authorities. The first thing that notices is a verification failure half a second later. I ran that call against the shipped library with the paths pointed at /nonexistent and it returned 1. It is in the output above.

The second is a policy that comes down from the service, and the log records it:

Wdm.cpp:1162 parseDeviceJson: Adding policy << allowSelfSignedCertificate with value: false
NetworkManager.cpp:1738 onConfigReady: ...httpRequestSSLRetryEnabled: 0
HttpRequestManager.cpp:1883 rawHttpRequest: {"validateCertificates":"true","useClientCertificate":"false"}

Those three flags are the only reason this fault is an outage instead of something far worse, and that is worth sitting with rather than hurrying past.

Work it through. The bundled library loads zero trust anchors, and nothing beneath the policy layer was ever going to notice, because the failure is silent by design all the way down. What turned it into a banner was three flags. Flip any one of them the way plenty of clients ship them and the same build does not fail at all. It connects, to anything holding any certificate, because it has nothing to check one against.

What it surfaces as is the problem. The user gets “Offline - No internet connection”. The log gets “Error in SSL handshake” and a decimal integer. The word certificate does not appear anywhere the user or a first-line support engineer will ever look, and the one diagnostic breadcrumb is a number you have to convert to hex before it means anything.

The Fixes That Did Not Work

Both of the obvious ones fail, and the reasons are different and both worth knowing.

Editing the shipped openssl.cnf

Webex ships a configuration file at /opt/Webex/lib/openssl.cnf, and as shipped it activates one provider:

[provider_sect]
fips = fips_sect

Only FIPS. Not the default provider, which is where the ordinary TLS algorithms live⁶. Adding it back is a one-line edit and it changes nothing at all, because the bundled OpenSSL never reads that file. It looks for openssl.cnf inside OPENSSLDIR⁷, and OPENSSLDIR is the path that does not exist. The file sits in the installation directory looking authoritative. Nothing reads it.

It is a second defect hiding behind the first. Even if the path problem were fixed tomorrow by pointing OPENSSLDIR at /opt/Webex/lib, this configuration would then load and would activate only FIPS. And the FIPS module itself is loaded from MODULESDIR, the third dead path from the same tree, so fips.so shipping in /opt/Webex/lib cannot be found either. Three paths, one wrong prefix, and every one of them broken in a way that hides the next.

Setting the environment variables

The library honours SSL_CERT_FILE and SSL_CERT_DIR. I proved that above. The successful run is right there. So the obvious move is to set them in the desktop entry, which is what was tried:

Exec=env OPENSSL_CONF=/opt/Webex/lib/openssl.cnf \
  SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
  SSL_CERT_DIR=/etc/pki/tls/certs/ /opt/Webex/bin/CiscoCollabHost %U

No change. And the reason is not that Webex ignores the variables. The reason is that the process never received them:

$ tr '\0' '\n' < /proc/20293/environ | grep -E 'SSL|OPENSSL|CURL'
$ tr '\0' '\n' < /proc/20293/environ | wc -l
99

Ninety-nine variables in the running Webex process, not one of them the three that were set. Because there are two desktop entries with the same name on this machine:

And the specification is not ambiguous about which wins: “The base directory defined by $XDG_DATA_HOME is considered more important than any of the base directories defined by $XDG_DATA_DIRS.”⁸ $XDG_DATA_HOME is ~/.local/share. The user copy shadows the packaged one, every time, on every desktop that follows the spec⁹.

So Webex installs a second copy of its own launcher into your home directory, and that copy is the one your desktop runs. Change the packaged file all you like. You are editing a document nothing reads.

The Fix That Works

If the library insists on a path, give it the path. Create the directory it was compiled to want and fill it with symlinks to the real thing:

#!/bin/bash
# Point the bundled CiscoSSL at the system trust store by building the
# directory it was compiled to look for. Tested: Fedora 44, Webex 46.8.0.35631.
set -euo pipefail

OPENSSLDIR=$(strings /opt/Webex/lib/libcrypto.so.3 \
  | grep -oP '(?<=OPENSSLDIR: ")[^"]+')
[ -n "$OPENSSLDIR" ] || { echo "no OPENSSLDIR found in the shipped library"; exit 1; }

for p in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
         /etc/ssl/certs/ca-certificates.crt \
         /etc/pki/tls/certs/ca-bundle.crt \
         /etc/ssl/cert.pem; do
  [ -f "$p" ] && { CA_BUNDLE="$p"; break; }
done
[ -n "${CA_BUNDLE:-}" ] || { echo "no system CA bundle found"; exit 1; }

echo "OPENSSLDIR: $OPENSSLDIR"
echo "CA bundle:  $CA_BUNDLE"

sudo mkdir -p "$OPENSSLDIR"
sudo ln -sf "$CA_BUNDLE" "$OPENSSLDIR/cert.pem"
sudo ln -sf "$(dirname "$CA_BUNDLE")" "$OPENSSLDIR/certs"

sudo tee "$OPENSSLDIR/openssl.cnf" > /dev/null <<'CONF'
openssl_conf = openssl_init
[openssl_init]
providers = provider_sect
[provider_sect]
default = default_sect
fips = fips_sect
[default_sect]
activate = 1
CONF

echo "done. now restart Webex"

Restart it, and the same startup sequence produces the opposite outcome. Same binary. Same log lines. Token refresh, which failed within sixty milliseconds before, now finishes in three hundred and thirty:

AuthTokenRequester.cpp:579  Managed to fetch a new Kms access token.
AuthTokenRequester.cpp:579  Managed to fetch a new CloudApps access token.
AuthTokenSupervisor.cpp:310 Auth tokens refreshed. Expires in [64799 secs].
AuthenticationManager.cpp:1860 onUserAuthenticated: User authenticated.
ConnectivityStateMachine::Authentication - UserNotAuthenticated -> UserAuthenticated

Authenticated in about 750 ms, so the fifteen-second timer never fires and the banner never appears. Phone services go from Disconnected to Connecting, a state the broken session never reached in a minute of trying.

Note what the fix does to your filesystem, because it should bother you. It creates a top-level directory called /workspace on your machine, a name the Filesystem Hierarchy Standard has no place for¹⁰, holding a Conan cache path and a build hash belonging to a company you bought software from. That is the shape of the remedy Cisco have left you: mount somebody else’s build environment at the root of your own.

It will also break. Three ways:

None of that is maintenance. It is you standing in for a step in someone else’s build pipeline, indefinitely, unpaid. Patching a defect at runtime, on every machine you own, because the vendor would not patch it once at build time.

They Knew The Rule And Applied It To Half The Build

Here is what turns this from a bug report into an argument.

Read the dynamic section of the shipped binaries:

$ readelf -d /opt/Webex/bin/libcurl.so | grep RUNPATH
 0x1d (RUNPATH)  Library runpath: [$ORIGIN:$ORIGIN/../lib]
$ readelf -d /opt/Webex/bin/CiscoCollabHost | grep RUNPATH
 0x1d (RUNPATH)  Library runpath: [$ORIGIN/../lib]

$ORIGIN expands at load time to the directory holding the object itself¹². It is the correct tool for a relocatable bundle and they used it properly. They had to: Webex ships the same tree twice, once to /opt/Webex and once to ~/.local/share/WebexLauncher/46.8.0.35631_9e6196c9-…/, and a launcher picks between them at startup. Two prefixes, one build, and the linker finds its libraries in both.

So the people who made this package understood the problem exactly. Absolute paths do not survive being shipped. They solved it for the code.

Then they left the data paths as absolute strings naming the container that compiled them. Same build. Same afternoon.

So this cannot be filed under they did not know. $ORIGIN is not something you stumble into. You reach for it because you have understood that an absolute path baked into a shipped artefact is a defect, and understood it well enough to go and fix it in the linker. Then the same build writes three absolute paths into the same libraries, and ships them.

Knowing the rule and applying it to half the build is worse than not knowing it. Not knowing is a training problem and training has a fix. This is a package that had the right idea in it, in writing, in the ELF header where anyone could read it, and it went out of the door broken anyway. Which tells you that nothing downstream of the compiler was looking at the result. Nowt was.

The Container Was Already There

Where /workspace came from is not a mystery, and you do not have to guess. It is in the package header:

$ rpm -qi webex | grep -E 'Build Host|Build Date|Vendor'
Build Date  : Sat 08 Aug 2026 20:47:43 BST
Build Host  : c964ea9239ae
Vendor      : Cisco

c964ea9239ae is not a hostname anybody typed. It is twelve hexadecimal characters, which is what a container reports as its hostname when nothing sets one. So the package was built inside a container, by a company that plainly has the images, the registry and the orchestration to do it, and three separate artefacts on this machine say so independently:

They then signed it. The build finished at 20:47:43 and the signature is dated 21:02:07 the same evening, key ID 9995e5bbb5ccde3c. Fifteen minutes. So there is a release gate, somebody or something operates it, and what it attests is who made the package, not whether the package works. A signature is a statement about provenance. It has never been a statement about fitness, and a process that has one and not the other has its priorities in the wrong order.

Because the missing step is the cheap one. The container is already in the pipeline. Take the artefact that just came out of it, start a clean image of each distribution you claim to support, install it, launch it, and read the first hundred lines of the log:

docker run --rm fedora:44 sh -c '
  dnf -y install ./webex-46.8.0.35631-1.x86_64.rpm &&
  timeout 25 /opt/Webex/bin/CiscoCollabHost &
  sleep 20
  grep -c "Error in SSL handshake" ~/.local/share/Webex/current_log.txt'

Non-zero, every time, on this build. The install is 1.1 GB, so call it a minute per target on a warm cache. Six distributions is six minutes of a machine that is already running, on hardware Cisco already own, in a pipeline that already exists. It was not run. Not once.

And that is the answer to the thing people still say about Linux, that supporting several distributions is hard. It stopped being hard the day this tooling arrived, and the tooling is the same tooling they compile with. One base image per target. Same artefact into each. The matrix is a loop.

A pipeline that builds in a container and never runs the result in one is not a pipeline. It is a compiler with a cron job in front of it and a signing key behind it, and whatever it produces is a guess.

Why Is A 2026 Release Built Against A 2018 Libc?

The package declares what it needs, and the interesting line is the first one:

$ rpm -q --requires webex | grep glibc
glibc >= 2.28

glibc 2.28 was released on 1 August 2018¹³. It is the version in Red Hat Enterprise Linux 8¹⁴, a release whose full support ended in 2024. This is a 2026 product, compiled in 2026, targeting the C library of 2018. Then shipping its own 2.5 MB copy of libstdc++.so.6 into the user’s home directory, because the C++ runtime that goes with a base that old cannot carry the code.

Why would anybody still do that? Because they will not link statically.

That is the whole of it. The moment you dynamically link against the host’s C library, the oldest distribution you are willing to support becomes a constraint on the machine you compile on. You cannot use a symbol that the oldest target has never heard of, so you pin the build to an ancient base image and you stay there. Every year the gap widens. Every new language or library feature arrives with an argument about whether the floor can move. Ship your own libstdc++ to paper over the worst of it, and you are now maintaining a private runtime as well.

That trade made sense when a build host was a physical machine in a rack that somebody had to reinstall. It has not made sense for a decade. If you must link against a host libc, a container per target gives you a real build on a real version of each platform, and none of them constrain the others.

But look at what the old-target discipline actually bought here. It is ABI conservatism at considerable cost: an eight-year-old floor, a bundled C++ runtime, a support matrix frozen around it. And the client still will not start. Because what broke was a file path, and no amount of care about symbol versions protects a file path. They paid the compatibility tax and the bundling tax, and skipped the one check that costs nothing. Both bills. No product.

They Paid For A Self-Contained Bundle And Did Not Get One

The long-standing way to ship commercial software on Linux is to depend on as little of the host as possible. Static where you can, a self-contained tree where you cannot, and no assumptions about the distribution underneath. It is not elegant and nobody pretends it is. It exists because of the alternative. A binary that needs a particular version of a particular library in a particular place turns every customer’s machine into a support case.

Cisco went the second route and bundled. Here is the bill:

Two gigabytes of dependencies. That is the full price of bundling: the download, the disc, the duplication, the security burden of being the only party who can patch any of it, the whole lot. Pay that and what you buy is a program that does not care what the host has installed, that behaves the same on Fedora and Debian and Arch and whatever a customer has standardised on this year, and that cannot be broken by a distribution upgrade it never heard about.

Except it does. It cares very much about one directory, and it is a directory on a build server.

They shipped their own Kerberos and their own ICU and their own spell checker, and they could not ship a working path to the certificates. The bundle’s entire purpose is to be self-contained, and it is not self-contained in the one respect that stops it working. Every one of those 1.1 GB gets found correctly through $ORIGIN. The forty-odd bytes that matter most do not.

And this is the bit that gets me. Bundling is the expensive option. They took the expense, they did the hard engineering, they got the relocation right for a hundred and fifty libraries across two install prefixes. Then pointed the one part that decides whether any of it works at a machine no customer has ever owned.

Nobody Documented It Either

A second failure sits alongside the first, and it is the one that would have caught the first.

Ask the package what documentation it ships:

$ rpm -qd webex | wc -l
0
$ rpm -qc webex | wc -l
0

No documentation files. And no configuration files. Zero entries marked %config in a package that ships an openssl.cnf. That is not cosmetic. An RPM marks a file %config so that the package manager preserves what the administrator changed, saving a .rpmsave rather than clobbering it¹⁵¹⁶. /opt/Webex/lib/openssl.cnf is shipped as an ordinary file, so the next update overwrites any edit you made to it and tells you nothing. The one file a customer might legitimately need to adjust is the one the packaging treats as disposable.

Nothing published says which trust store the client uses, which environment variables it honours, or where it reads its TLS configuration from. There is no page to check. None was written. The only way I established any of it was strings, readelf, ldd and ctypes against the shipped binaries. Reverse-engineering a supported product to answer a question its documentation should have answered in a sentence.

And here is why that matters more than it sounds. Writing that documentation is itself a test. Put anyone at the vendor in front of a blank page headed where Webex for Linux reads its CA certificates from, and the first thing they have to do is go and look. The moment they look, they find /workspace/.conan2/p/b/cisco8ee8b59cf93de/p/ssl, and the next thing out of them is a question. Documented configuration paths are not paperwork for the customer’s benefit. They are the cheapest audit a vendor can run on its own build, and skipping them is how a path like that survives to a release.

None of that is a big ask. Say where your configuration lives. Say which environment variables you honour. Mark your configuration files as configuration so an update does not eat them. Then the customer who hits a fault has somewhere to look that is not a hex editor.

Nobody Ran It

The missing step has been named enough times above. What is worth asking is why it went missing on this platform and not the others.

Not on Fedora, anyway. On macOS and the Fisher-Price OS (Windows) this class of fault cannot show up the same way, because those platforms have a system TLS stack with a trust store the operating system manages. Linux has no such thing. OpenSSL is the trust store, and as such whoever ships it owns where it looks. So the one platform where the bundled library is load-bearing is the platform that got shipped untested, which is a decision about which customers are worth a smoke test.

The whole investigation took an evening: reading the log, spotting that network detection passed before the banner claimed otherwise, pulling the compiled path out of the binary, reproducing the exact error code against the shipped library. All of it done with an AI agent doing the log correlation and the ctypes harness while I worked out what to ask it. I mention that for one reason: the diagnosis a vendor never made before signing this package and putting it in their own repository is now inside an evening’s reach of any customer with the patience to look. Cisco’s engineers have Claude available to them the same as anyone else, and it would have built this properly. You cannot ask it to bake /workspace/.conan2 into a shipping artefact without it telling you what will happen when the artefact leaves the workspace. The tooling to catch this is no longer scarce, and neither is the knowledge. What is missing is anyone at the vendor whose job it was to look.

Meanwhile the support path for the person hitting this is a banner that says their internet is down. They will restart the router. They will ring their provider. They will raise a ticket that goes nowhere, because the symptom Cisco chose to display points away from Cisco.

That is the complete list of what went wrong. It is worth saying what right would have looked like, because every item on it has a settled answer that predates this product.

How It Should Have Been Built

Enough of what went wrong. Here is the standard, and none of it is novel. It is what shipping a binary to somebody else’s computer has asked of you for twenty years.

Start with the decision Cisco got right in principle and wrong in execution: how much of the host you are willing to depend on. Linking statically is the strongest answer, and it is worth being concrete about it, because “just link it statically” gets waved about by people who have never had to and dismissed by people who have never tried.

And one line in the left column that the others are only supporting: the artefact your pipeline produced is the artefact the customer runs, byte for byte. Test it and you have tested the thing you shipped. That is the property this whole post is about the absence of.

The right column is worth being honest about. Genuine costs, and the reason people reach for musl or accept a hybrid. But look at the third row. Owning every patch applies identically to what Cisco already did: a bundled tree of 150 libraries is the same commitment with none of the load-time guarantees. They signed up to own every patch either way and got nothing back for it.

So the rule is simple, and it is the rule they broke: whatever you cannot link in, you must find by a path relative to the binary. $ORIGIN for the code, and the same discipline, deliberately, for every data path the library will go looking for. In this case there are four of them and every one has a documented handle on it:

Four paths, four environment variables, all of them in one manual page their own library ships with. If any string in your artefact starts with a / and was decided at build time, it is a defect waiting for a customer to find. There is no third option where an absolute build path is fine.

The Fixes For This One, And The Check That Catches It

Down from the principle to this specific defect. Six changes, not one of them research:

The first is the one-flag fix and it would have shipped the product working. It is a single value in a build script, set once, that they got wrong because nothing downstream ever checked it.

The check is smaller than the fix:

# in CI, on the packaged artefact, in a clean container
test -d "$(strings lib/libcrypto.so.3 | grep -oP '(?<=OPENSSLDIR: ")[^"]+')" \
  || { echo "shipping a trust store path that does not exist"; exit 1; }

One line. It would have failed this release loudly, in February, on the machine that made it, in the container that made it, before the package was signed. The reason it is not there is not difficulty and not cost. It is that nobody was asked to write it, and so whether the packaged artefact behaved like software was not anyone’s job.

What A Sloppy Build Process Costs Everybody Else

Everything above is one product’s build process seen from the inside, and I will not walk you back through it. The question worth asking is whether that standard of care is likely to stop at one team.

So put the outside record next to it. CISA maintain a catalogue of vulnerabilities known to be exploited in the wild. Not theoretical, not scored. Observed being used against people. As of 14 September 2026 it holds 1,710 entries¹⁸:

Second, behind an operating system monopoly and ahead of everybody else. The most recent Cisco addition went in on 14 September 2026, the day before this was written.

I counted the same file three weeks earlier for /en/random/is-your-msp-lying-to-you-part2/: version 2026.08.27, 1,685 entries, Cisco on 96. Two more since, in twenty-one days.

Be fair about what that table does and does not prove on its own. A large installed base in high-value places attracts attention, and attention finds bugs, so any vendor at that scale will carry a long list. The number is a prior, not a verdict.

The composition is harder to wave away than the total. Twenty-five of Cisco’s ninety-eight sit on the security lines: the firewalls, the appliances, the VPN concentrators, the mail and web gateways, the identity services. And the four most recent entries added against them, in a row, are Secure Firewall Management Center twice, Secure Firewall ASA, and Secure Email Gateway, the last of those on 14 September 2026¹⁸. Not the switches. Not the collaboration kit. The products sold specifically to be the thing that keeps everybody else safe.

Which is the sentence this whole post has been walking towards, so here it is plainly. This is a company whose business is selling security appliances, and it cannot get a desktop client to check a certificate. Not a hard case. Not a novel attack. The single most routine security operation in computing, performed by every browser on every page load, in a library they forked themselves, and they shipped it pointed at a directory that has never existed on a customer machine. Then signed it.

What this post adds is a sample of the process behind it. Not a vulnerability. A plain packaging defect, the least subtle class of fault there is, on a supported platform, caught by any smoke test anyone cared to run, and shipped anyway. If a build pipeline will put that in front of a paying customer, it is not obvious what it would stop. Nothing did.

Can I Ask Why This Was Signed?

Can I ask why a package that cannot complete a TLS handshake was signed and published against a platform your own requirements page lists as supported? Not who signed it. I have no interest in a name and it is not about one. What permitted it, because something did, fifteen minutes after the build finished, and whatever it is it is still running today.

Now the blunt version. This is not a project I pulled off a forge and took my chances with. It is paid for. There is a contract behind it, a support line, a requirements page making a claim about Linux, and a signing key asserting the package came from Cisco and is fit to install. Each of those is a statement to a customer, and on this release each was worth nothing, because the result was never opened.

And the standard being missed here is not mine. It is theirs. The four variables that would have carried those paths are documented in a manual page you ship inside the bundle. Their own packaging format has a %config marker they did not use and a documentation section they left empty. Their own build ran in a container they never used to run the output. I am not asking a networking and collaboration company to invent anything. I am asking why it did not read its own manuals.

As such the excuse I will not accept is that this is hard. It is not hard for anybody, and it is certainly not hard for a company that does this daily, at this scale, for this money. Professionals doing this every day have no excuse, and being large is not one.

And can I ask one more, because it is the one that actually matters. You sell firewalls. You sell an email gateway, a VPN concentrator, an identity service, a management centre for all of it, and the pitch on all of them is that you understand this better than your customer does. So how does a company that positions itself as an authority on network security ship a client that cannot validate a certificate? Not fails to validate it cleverly. Cannot look one up, because the directory was never there.

There is no version of that answer I want to hear that starts with the desktop team being separate from the appliance team. They are the same signature, the same pipeline, the same published claim about a supported platform, and the same missing step at the end, which is opening the box. If certificate validation is not checked before shipping in the product where breakage is loud and harmless, I have no reason to believe it is checked in the product where breakage is quiet and expensive.

And the honest part, said plainly. None of this surprised me. I have come to expect this standard from this vendor, which is why, where the choice is mine, I do not buy their kit and I do not build on it. That is not a preference about badges. It is the same judgement I would make about any supplier: I have measured their work, more than once, and it keeps coming back the same. The catalogue above is one measurement. What it took to write the first half of this post is another.

The reason I was on Webex at all is that the choice was not mine, and that is worth naming, because it is the position most people reading this are in. You rarely get to avoid a vendor whose quality you have already taken the measure of. Somebody else signs the contract, the kit arrives, and the first person to find out what was skipped in the build is you, at your desk, with a meeting starting.

Shipping Something You Never Ran

There will be an internal explanation. A busy sprint, a pipeline that changed hands, a platform with no owner. None of it is worth hearing, because each describes the same thing: the work was not done and nothing in the process required it to be.

There is an old standard in trades work that has never made it into software: you do not leave the job until you have run the thing. You fill the system and check every joint. You energise the board and test every circuit. Not because you doubt your work. Because the customer is going to use it, and finding out in front of them is not a professional outcome. Nobody watches you do it. You do it anyway. That is the entire meaning of the word.

Software has spent thirty years arguing that it is different, that the build is the deliverable and the install is somebody else’s problem, that a green pipeline is the same thing as a working product, and it is not. A build that has never been executed outside the container that produced it has not been finished, it has been abandoned at the point where finishing gets boring. Everything downstream of it is a claim about work that was not done.

The fix is one configure flag. The check is one line of shell. The cost of neither is what is interesting; the interesting number is how many people typed their credentials into a Webex window, watched it say their internet was down, and believed it, because Cisco told them so and Cisco are a networking company. That is what the missing test actually bought: not a bug, a lie that the product tells confidently, every launch, to people who have no way to know better.

And that is the line worth drawing before you close the tab, because the two halves of this post are not two subjects. A trust path that points at a build container and an authentication bypass on an edge appliance are the same failure at different stakes. Both are a value nobody checked, in an artefact nobody ran, signed by a process that attests provenance and not fitness. The one in front of me was the harmless kind. It broke loudly, on my own desk, and I was the first to know. The other kind does not do that for you.

Nobody at Cisco decided to ship an exploitable firewall, any more than they decided to ship a client that cannot reach the internet. That is not a defence. It is the charge. Neither needs deciding, and that is precisely the problem: both are what comes out the far end when a pipeline compiles, signs and publishes without anyone being made responsible for opening the result. A process that will not catch a directory that does not exist was never going to catch a length that is not checked, and a vendor that sells you the appliance guarding your perimeter is not entitled to that process.

So when a vendor’s name keeps appearing on that list, resist the comfortable explanation that they are simply big and heavily targeted. Scale explains the volume. It does not explain the kind. Look instead at what their build process does with the boring things, because the boring things are measurable from the outside, by you, today, on kit you already own.

Check your own bundles. strings and readelf and twenty minutes will tell you which of your vendors ship a path to a machine you will never see. What you are really measuring is not the path. It is whether anybody there was looking, and if the answer is no on something this cheap to catch, you already know what it is on the things that are not.