Mutual authentication between nodes and researcher¶
Fed-BioMed encrypts all gRPC traffic between nodes and the researcher with TLS. By default only the researcher side is authenticated: each node trusts whichever certificate the endpoint presents, and the researcher accepts any node that connects. Mutual authentication makes the verification go both ways — the researcher requires and verifies each node's client certificate, and each node pins the researcher's certificate — so every channel is bound to a registered Fed-BioMed party.
The protocol carrying it is mTLS (mutual TLS), the variant of TLS where both sides present a certificate. That is why gRPC failures around it are reported as TLS handshake errors.
It is what establishes who the parties are, which is why a deployment cannot go without it. Two situations make the point:
- nothing else authenticates the parties — Fed-BioMed running on plain Docker images, or embedded in an existing system that provides no identity of its own;
- the surrounding infrastructure authenticates machines rather than parties, so whoever is admitted to the network can still spoof another Fed-BioMed party at the gRPC level. Mutual authentication binds each channel to a registered component identity.
Either way it covers the "mutual verification of other party identity during gRPC setup" item of the security model (mitigating a malicious insider man-in-the-middle that spoofs a party inside the network).
Mandatory for a deployment
[authentication] mutual_authentication = True must be present in the configuration of the researcher and of every node in a deployment. The setting is off by default, and an absent [authentication] section reads the same as mutual_authentication = False — a component left that way accepts an unverified peer, so it is not deployable.
Leave it off only for local experimentation, where the channel falls back to the server-authenticated TLS described above and nothing verifies who the parties are.
What mutual authentication changes¶
| Without it (server-authenticated TLS) | Mutual authentication enabled | |
|---|---|---|
| Node → researcher | Node fetches the server cert at connect and trusts it | Node pins a pre-registered researcher cert |
| Researcher → node | Node identity not checked | Node must present a registered client cert |
| Node identity | Declared in the message only | Every message's declared id must match the component id the presented certificate is registered under |
| New certificates | — | On the researcher: picked up on the next handshake, no restart (hot-add). On a node: read once at startup, so a restart applies them |
Component certificates¶
Each component already owns a self-signed certificate, generated automatically when the component is created (recorded in the [certificate] section of its config, under etc/). Its subject carries two fields:
CN=(Common Name) holds the component id, which is what the certificate identifies — an id, not a host;O=(Organization) isFed-BioMed, marking a certificate Fed-BioMed itself issued, so itsCN=is read as a component id rather than as free text.
Mutual authentication reuses these certificates — you exchange and register them, then flip a switch.
O=Fed-BioMed records which tooling issued a certificate, not that anything verified it: a self-signed certificate asserts its own subject, so the field is descriptive and nothing treats it as a security check. A certificate issued elsewhere — by your own CA — carries whatever CN= that issuer chose, often a hostname, and its component id is supplied when you register it.
A generated certificate declares its role in an Extended Key Usage: clientAuth on a node certificate, serverAuth on a researcher one. The field describes the credential rather than restricting it — gRPC applies no purpose check of its own, and a certificate is registered whatever role it declares, including none, as certificates issued elsewhere often do.
Names on the researcher certificate¶
A node verifies the researcher by name, so the researcher certificate is issued for the hosts nodes reach it at: the [server] host of its configuration, and the names given with --san, and nothing else. For a researcher reachable under a name its configuration does not hold — a public DNS name, a second interface — issue it for those names too:
fedbiomed researcher certificate generate \
--san fbm.example.org --san 10.0.0.9 --force
The component was issued a certificate when it was created, so this requires --force — without it the command refuses to overwrite. The previous private key is lost, so every party holding the old certificate has to register the new one.
Each name is written as what it is: an address goes in as an iPAddress entry, a host name as a dNSName one, and TLS never matches one against the other. Naming any loopback form — localhost, 127.0.0.1, ::1 — issues the certificate for all three, since a node on the researcher's own machine dials whichever of them its configuration holds. Only these two kinds of entry state a host: a Subject Alternative Name carrying just an e-mail address or a URI states none.
A node refuses to register a researcher certificate that states no host, and under mutual authentication builds no channel on the one it pins. gRPC verifies such a certificate against its Common Name, which here holds the component id, and which any other issuer fills as it likes — a host among the rest: the researcher would be authenticated on free text.
Without mutual authentication the certificate is the one the endpoint served, so its name vouches for nothing the fetch did not already grant. It is used as it is, gRPC matching the host dialled against its Common Name — which is how a researcher whose certificate states no host is reached.
A node certificate states no host and is registered on the researcher as it is: a node is authenticated by the certificate registered for it, never by name.
When the address dialled is not a name the certificate carries
A researcher certificate is issued for the hosts known when it was generated, which need not include the [researcher] ip a node dials. The node then verifies the connection under the first name the certificate does carry and connects, warning once at startup — unless that name is a loopback form of the machine it dials, the same machine under another spelling. This is a design choice and may change — what authenticates the researcher is the certificate the node holds, not the address.
Enabling mutual authentication¶
For an existing installation, first follow the upgrade checklist. It covers older configuration files, researcher certificates without host SANs, certificate redistribution and restarts, and migration from the removed bundled VPN deployment.
Every component must already be created, since that is when its certificate is generated. The setup is then symmetric: every party exports its certificate, shares it through a trusted channel, and registers the certificates it receives. Then both sides turn mutual authentication on.
1. Export and share each certificate¶
On every component (each node and the researcher), produce its registration bundle:
fedbiomed node certificate registration-instructions # on a node
fedbiomed researcher certificate registration-instructions # on the researcher
Send the printed certificate to the other parties over a trusted channel (e-mail, secure messaging, …). Fed-BioMed does not exchange certificates for you.
Who registers whose certificate
- The researcher must register every node's certificate (the trust bundle it verifies clients against).
- Each node must register the researcher's certificate (the one it pins).
Node-to-node registration is not needed for mutual authentication.
2. Register the received certificates¶
Save each received certificate to a file and register it. The component id is read from the CN= field of a certificate carrying O=Fed-BioMed, so --component-id is optional — supply it for a certificate issued elsewhere, whose CN= states no component id; given alongside an embedded identity, it must match it:
# On the researcher: register each node certificate
fedbiomed researcher certificate register -pk /path/to/node1.pem
fedbiomed researcher certificate register -pk /path/to/node2.pem
# On each node: register the researcher certificate
fedbiomed node certificate register -pk /path/to/researcher.pem
Check what is registered at any time:
fedbiomed [node|researcher] certificate list # shows component id and expiry
On a node, the registry is read once — at startup
The researcher re-reads its trust bundle on every handshake, so a certificate registered or deleted there applies to the connections that follow. A node reads the researcher certificate it pins when it starts and keeps it for the whole run: registering, replacing (--upsert) or deleting a certificate on a node changes nothing for the running process — restart the node to apply it.
An expired certificate is refused, on a node and on the researcher alike: it completes no handshake, so registering it would only move the failure to the connection. A certificate within 30 days of expiry registers, with a warning naming the date.
A node registers at most one certificate — its researcher's. Registering a second component is rejected; re-registering the same component goes through --upsert. Those three — an unexpired certificate, one certificate on a node, and a certificate stating a host — are the only registration refuses on the registering component's own account.
certificate list reports a node holding more than one certificate, and a node refuses to start in that state, since it cannot tell which to pin — delete the extras with fedbiomed node certificate delete.
3. Turn mutual authentication on¶
The config file (etc/config.ini) of both the researcher and every node must carry, adding the section if it is not already there:
[authentication]
mutual_authentication = True
It is mandatory in a deployment: the entry has to be present and set to True on every component, the researcher included.
Each component reads this setting when it starts, so change it while the component is stopped, or restart it afterwards — on the researcher, by relaunching it or restarting the Jupyter Notebook. Only the researcher re-reads its registered certificates while running — a node keeps the certificate it read at startup — and a component left on its previous setting behaves exactly as the state combinations below describe.
4. Start the components¶
Start the researcher first, then the nodes. The researcher starts even with no node certificate registered yet — it binds the port and rejects every node handshake until a certificate is registered, which it then picks up without a restart. On success you will see security log lines such as:
- Researcher:
Node <NODE_id> identity verified by mutual authentication. - Node:
Mutually authenticated communication established with researcher …; node identity verified by the researcher.
The researcher retires connections once they reach a maximum age, so a node reconnects regularly even while nothing goes wrong. The node announces its first connection and any that follows an interruption; a connection merely replacing a retired one is logged at debug level. The security audit log records every one of them.
From the Node GUI¶
A node does the same from the Configuration tab of the Node Management page of its GUI, which is restricted to administrators. Its Connection & certificates group holds the researcher endpoint, this node's own certificate and the [authentication] setting, each with the window it opens:
- Node certificate, under this node's own pair: read what the node presents, download it to send to the researcher, regenerate it, or replace it with a certificate and private key issued elsewhere. The displaced pair is kept as the single
.bakbackup beside it, asfedbiomed node certificate replacedoes; - Researcher certificate, under the researcher: read the registered researcher certificate, register one — pasted or from a file — and delete it. A registered certificate is deleted before another is registered;
--upsertreplaces it in one step from the command line. Turningmutual_authenticationon opens this window, since that is what makes a registered certificate necessary.
It reports what the node itself checks as it starts, so the same findings appear here and from fedbiomed node certificate check. It reports as a problem what stops the node: mutual authentication enabled with no researcher certificate registered, several registered, one that states no host, one that has expired, or the node's own key or certificate missing, unreadable or expired. It reports as a warning what the node tolerates but you should fix: a registered certificate naming a host other than the one configured, or either certificate within 30 days of expiry. With mutual authentication off the node pins nothing and presents nothing, so what is registered is still reported, as a warning, to be fixed before you turn it on.
Certificates are read when the node starts, so registering one or changing the setting takes effect when the node is restarted.
The Connection & Diagnostics tab reads the state of the connection to the researcher as the node last recorded it: whether the channel is up, whether it is mutually authenticated, why it last failed, and what to do about it.
The researcher has no GUI; it is configured with the commands above.
Operating a running federation¶
Adding a node to a running instance (hot-add)¶
The researcher re-reads its trust bundle on every handshake, so a new node can join without restarting the researcher:
- On the new node: register the researcher certificate and set
[authentication] mutual_authentication = True. - Send the new node's certificate to the researcher.
- On the researcher:
fedbiomed researcher certificate register -pk /path/to/newnode.pem. - Start the new node — it is trusted on its first connection.
Renewing / rotating a certificate¶
Certificates are valid for 5 years. When one is renewed, its identity is unchanged but the key and expiry differ, so every party that pinned or trusted it must register the new one:
- Regenerate the certificate on the component. Replacing an existing one requires
--force, and the previous private key is lost:A component reads its own certificate and private key when it starts, so restart it to present the new ones — a researcher included, whose own certificate is not part of what it re-reads while running.fedbiomed <component> certificate generate --force
A certificate issued elsewhere — by a certificate authority rather than by Fed-BioMed — is installed rather than generated, together with the private key that matches it. The pair is checked together before either file is written, so one the component could not serve is refused while the pair it serves still stands:
fedbiomed <component> certificate replace \
--public-key /path/to/certificate.pem --private-key /path/to/private.key
.bak, which the next replacement overwrites: certificate replace refuses to do so unless --force is given. 2. Re-share it and re-register it on the other parties with --upsert to overwrite: fedbiomed researcher certificate register -pk /path/to/renewed.pem --upsert
Which expiries are watched
Registration refuses an expired certificate and warns about one within 30 days of expiry, on both components — that is where a node hears about the researcher certificate it pins. Afterwards, only the researcher keeps watching: it logs a warning when a registered node certificate is within 30 days of expiry (reported once, when it enters that window). A running component is warned about neither its own certificate nor, on a node, the researcher certificate it pins — check those with fedbiomed node certificate check, which reports both, or with fedbiomed [node|researcher] certificate list, which prints every expiry date.
Development and testing shortcut: certificate-dev-setup¶
When several components sit side by side in one directory, one command replaces steps 1 and 2: it registers in each of them the certificates it must trust — the node certificates on the researcher, the researcher's on each node. It expects one researcher and at least two nodes:
fedbiomed certificate-dev-setup
Components are read from the directory the command is run in, and only from its first level — a component nested deeper is not part of the federation being set up. Give --path to point at another directory:
fedbiomed certificate-dev-setup --path /path/to/components
Certificates already registered are reported and left as they are, so the command can be re-run to complete a federation a node was added to. A registration that no longer matches the certificate the component serves — after regenerating one, for instance — fails the run instead, since it would leave a federation that cannot handshake.
--prune clears every registration in each component before writing the ones found under the path, so the trust stores end up describing the components currently there and nothing else. Use it after replacing or removing a component: without it the departed component stays trusted, and a node left holding two researcher certificates cannot tell which one to pin.
fedbiomed certificate-dev-setup --prune
This registers certificates only. Setting [authentication] mutual_authentication = True in each config remains mandatory for any deployment, and --enable-mutual-authentication does it for every component at once, after their certificates are registered:
fedbiomed certificate-dev-setup --enable-mutual-authentication
Components already running have to be restarted to pick the setting up, and a running node equally to pick up the certificates the command registered or pruned in it. The command reads components that are already on the machine, so it is for development and testing — it replaces no step of a real deployment.
Outcome by state¶
What each party ends up doing depends on both sides at once: whether mutual authentication is enabled there, and what its certificate registry holds. The two tables below answer what happens in this state, for the combinations that occur in practice. The message a failing state produces, and its fix, are indexed the other way round — by what is logged — in verifying and troubleshooting.
By the mutual authentication setting¶
Node [authentication] | Researcher [authentication] | On the node | On the researcher |
|---|---|---|---|
| off | off | Fetches the certificate the endpoint presents and trusts it: Communication established … over server-authenticated TLS (node identity not verified) | Accepts the node without checking its identity |
| on | on | Pins the registered researcher certificate and presents its own: Mutually authenticated communication established …; node identity verified by the researcher | Node <NODE_id> identity verified by mutual authentication on the node's first request |
| on | off | Warns, refuses the task and retries. The researcher names the node it authenticated in every task response; naming none is what tells the node. Enabling it on the researcher connects the node with no restart | Answers the request, then sees the node close the channel and ask again; it would have accepted it like any other, ignoring the client certificate presented |
| off | on | Stops. No configuration the node can reach by retrying would connect | Rejects the node inside the TLS handshake; the node never appears, and the rejection is visible only with GRPC_VERBOSITY=INFO |
By certificate state (both sides enabled)¶
| Node state | Researcher state | On the node | On the researcher |
|---|---|---|---|
| Researcher certificate registered, own certificate valid | Node certificate registered, own certificate valid | Connects; node identity verified | Authenticates the node |
| Starts before its certificate is registered | Certificate registered while the researcher runs (hot-add) | Connects on a later retry, no restart | Picks the new certificate up at the next handshake, no restart |
| No researcher certificate registered | any | Refuses to start | The node never connects |
| Several researcher certificates registered | any | Refuses to start: the one to pin is ambiguous | The node never connects |
| any | No node certificate registered | Warns once, then retries at debug level. Registering its certificate connects the node with no restart | Starts and binds the port, warns that mutual authentication is on with no node certificate registered, and rejects every node handshake until one is — picked up with no restart |
No valid certificate of this node on the researcher: never registered, expired, or regenerated with --force and not re-shared | Trust bundle holding no current certificate for that node | Warns once, then retries at debug level. Registering the current certificate connects the node with no restart | Rejects it inside the handshake, no per-node log. An expiry had been announced by certificate_expiring warnings from 30 days before; a regenerated certificate is registered with --upsert |
| Certificate within 30 days of expiry | Holds that certificate | Connects normally | Warns once |
| Pins an outdated or wrong researcher certificate | Serves its current certificate | Warns once, then retries at debug level — treat as possible MITM | Handshake aborted by the node, nothing logged |
| Pins the researcher's current certificate | Own certificate expired | Same handshake failure as an outdated pin | No node can establish a channel |
| Declares a node id different from the component id its certificate is registered under | Certificate registered under that other component id | Warns once, then retries at debug level | Aborts the request UNAUTHENTICATED |
| Running with an established connection | Its certificate is deleted while the researcher runs | Rejected on the next request, then retries; reconnection attempts are refused during the handshake until it is registered again | Refuses the node until it is registered again |
| Running with an established connection | Restarted | Retries until the researcher answers again, at debug level; catching the restart window warns once | Authenticates the node again on its next request |
| Running while its own registry changes: the researcher certificate is registered, replaced or deleted on the node | any | Unaffected — it keeps pinning the certificate it read at startup. The change applies when the node is restarted | Unaffected |
Verifying and troubleshooting¶
Both sides log the security state of the channel and record structured security audit events, identifying the peer certificate by subject, issuer, serial and expiry — never by its contents. The researcher additionally records the node's source address; it re-records a node whose certificate or address changes, so reconnections and certificate rotations are visible. A failure the node retries is logged once as a warning, then repeated at debug level only until the connection recovers; only a state that stops the node is an error. If a connection does not establish, match the symptom below — diagnosis is mostly node-side: the researcher rejects untrusted nodes inside the TLS handshake and logs nothing per rejected node (it says so once at startup).
Before starting a node, ask it what it checks about its certificates. The command reports the same findings the GUI shows, and exits non-zero when one of them would stop the node, so it also serves as a deployment check:
fedbiomed node certificate check
fedbiomed node status reads the node's last connection state back as a table, without searching the logs, and --history adds the changes it recorded recently:
fedbiomed node status --history
A state recorded while the node was running is reported as it was when the node stopped, since a stopped node records nothing. The fields and states of the table are described in Checking the connection to the researcher.
Seeing failed handshakes as gRPC reports them
Each end reports its own side of a failed handshake through gRPC itself, at INFO, naming the OpenSSL reason: the researcher a node reached without a certificate logs PEER_DID_NOT_RETURN_A_CERTIFICATE, the node that cannot verify the researcher certificate it pinned logs CERTIFICATE_VERIFY_FAILED. Fed-BioMed lowers gRPC to ERROR on both components, because a node retrying its connection would otherwise add one line to each output every couple of seconds. A value set in the environment takes precedence, so raise it on the side being diagnosed:
GRPC_VERBOSITY=INFO fedbiomed researcher start
GRPC_VERBOSITY=INFO fedbiomed node start
| Log / error | Cause | Fix |
|---|---|---|
FB619 … no researcher certificate is registered (node won't start) | Mutual authentication on, researcher cert missing on node | Register the researcher certificate on the node |
FB619 … certificates are registered (node won't start) | More than one certificate is registered on the node, so the one to pin is ambiguous | Delete the extras with fedbiomed node certificate delete, keeping its researcher's |
FB619 … states no host (registration refused on the node) | The researcher certificate names no host, so gRPC would verify it against its Common Name. Its Subject Alternative Name is absent, or carries only entries that name no server — an e-mail address, a URI | Request the researcher to reissue its certificate for the hosts nodes reach it at, then register that one |
FB628 … states no host (node stops) | The same certificate reaching the connection under mutual authentication: one registered before this rule. Without mutual authentication the certificate is fetched, not pinned, and is used as it is | Request the researcher to reissue its certificate for the hosts nodes reach it at, register the new one on the node and restart it |
FB628 … does not carry the name this node verifies it under (node stops) | gRPC accepted the researcher's certificate but refused its name. The name comes from the certificate the node holds, so this needs one that vouches for a differently named certificate — a certificate authority's, held in place of the researcher's own. A researcher simply serving another certificate is refused earlier, as the mTLS handshake failure below. Retrying cannot change either | Hold the certificate the researcher serves, not one that vouches for it: register it on the node and restart it; without mutual authentication, restart the node, which fetches it |
Debug Researcher server is not available (node retries) | The endpoint is not up | Check the researcher started |
Warning Mutual authentication (mTLS) handshake with researcher failed (node retries) | The certificates the two sides hold for each other do not match — most often a pinned researcher certificate that is wrong or outdated — or a possible MITM | Re-register the current researcher certificate on the node and restart it — a running node keeps the certificate it read at startup — and check this node's certificate is the one registered on the researcher |
Warning … reachable but closes the connection during the TLS handshake (node retries) | Node cert not registered on the researcher, or expired since it was registered — rejected inside the handshake. A researcher that is restarting closes connections the same way, and clears on its own | Register the node's certificate on the researcher; if it is registered, check its expiry, then whether the researcher was restarting |
FB628 … researcher requires mutual authentication but it is disabled on this node (node stops) | Researcher has mutual authentication on, node has it off | Enable [authentication] on the node and register the researcher certificate there, then request the researcher to register this node's certificate |
Warning Researcher rejected this node's identity (node retries) | Its certificate is not registered on the researcher, or the node id it declares is not the one that certificate is registered under | Register the node's certificate on the researcher — the node connects with no restart — and ensure the node id matches how it is registered |
Warning … the researcher does not verify node identities: NO node in the federation is authenticated (node retries) | Node has mutual authentication on, researcher has it off, so the researcher names no node in its task responses. The node refuses every task until that changes | Request the researcher to enable [authentication] and register this node's certificate — the node connects with no restart. Disabling it on the node instead also clears the warning, but leaves the federation unauthenticated and is not deployable |
FB628 … verified this connection as <other id> (node stops) | The researcher answered naming a component that is not this node. A researcher rejects a mismatched id before answering, so this is not a configuration case — treat it as a researcher that is not behaving as one | Investigate the endpoint the node is pinned to; a certificate registered under the wrong id surfaces as Researcher rejected this node's identity instead |
Warning Mutual authentication is enabled but no node certificate is registered (researcher starts) | Mutual authentication on, but no node certificate registered yet | Register at least one node certificate — the researcher picks it up with no restart. Until then it binds the port but rejects every node handshake |
Warning Certificate <component_id> expires on <date> (researcher) | A registered node certificate is within 30 days of expiry; reported once, when it enters that window | Renew it on that node, share it again, and re-register it with --upsert before that date |
FB628 … Declared node id … does not match the identity … its certificate is registered under (researcher error) | A node declared an id different from the one its certificate is registered under | Investigate; the node id and its registered component id must be the same component |
FB628 … Refusing the node declaring id … its certificate is not registered (researcher error) | The node completed the handshake but its certificate is absent from the registry — typically deleted while the researcher was running | Re-register the node's certificate, or leave it rejected if the removal was deliberate |
A handshake failure may be an attack
Under mutual authentication a failed handshake is not silently retried as "server unavailable" — it is logged as a security event. If certificates are correct and a node still cannot connect, treat it as a possible man-in-the-middle and verify the endpoint.