Skip to content

Federated Preprocessing

Federated pre-processing

Overview

Federated pre-processing applies data transformations across nodes without moving raw data. The result is a new federated dataset, updated node by node.

At the moment, Fed-BioMed provides one federated pre-processing method: FedComBat.

FedComBat in short

FedComBat harmonizes tabular data to reduce site effects. In this context, one site corresponds to one Fed-BioMed node.

  • Phenotypes: modified variables to harmonize.
  • Covariates: predictive variables whose effects are modeled and preserved.

Reference paper: Federated ComBat.

Requirements

  • Datasets must be in CSV format.
  • At least 2 nodes must participate.
  • Node-side federated pre-processing must be enabled.
  • Node-side federated analytics must be enabled (FedComBat uses federated statistics).

See node configuration details in Configuring Nodes and Federated Analytics - Nodes.

Configure FedComBat in an experiment

Attach pre-processing to an Experiment with set_preprocessing:

from fedbiomed.common.constants import PreprocType

fedcombat_args = {
    "covariates": ["SEX", "AGE", "PTEDUCAT"],
    "phenotypes": ["CDRSB.bl", "ADAS11.bl"],
}

exp.set_preprocessing(PreprocType.FEDCOMBAT, fedcombat_args)

Arguments

Mandatory:

  • covariates: list of column names or list of column indices.
  • phenotypes: list of column names or list of column indices.

Constraints:

  • covariates and phenotypes must be lists.
  • They must use the same type (str or int).
  • They must be disjoint (no duplicate across both lists).

Optional:

  • rounds: FL rounds for the internal FedComBat model training.
  • training_args: training arguments for internal FedComBat model training.
  • model_args: additional model arguments for internal FedComBat model training.
  • standardize_result (False by default): if False restore initial value scale for harmonized output. If True, keep harmonized output in standardized scale.

Execution behavior

When you run training (exp.run() or exp.run_once()), pre-processing is executed automatically if needed.

So after attaching a pre-processing to an experiment, next exp.run() ensures the experiment dataset is harmonized across the nodes before the experiment model is trained. This means the experiment trains on the (automatically) harmonized dataset.

You can also trigger harmonization manually:

exp.preprocessing.execute()  # executes only if needed
exp.preprocessing.execute(force=True)  # force re-run even if not needed

FedComBat is re-executed only when context changes, for example:

  • selected nodes changed,
  • federated dataset changed,
  • pre-processing configuration changed.

After harmonization, the experiment training data is updated in place:

exp.training_data().data()

Important

If pre-processing is re-executed, it is re-executed depending on the content of the federated dataset. For example, if one node was removed from the federated dataset, it can lead to re-harmonize already harmonized dataset. This may not be the desired behaviour. In this case, one may want to re-execute harmonization on the original datasets of the remaining nodes.

Training plan approval (secure setups)

In deployments requiring training plan approval, FedComBat's internal training plan must be approved before execution.

Submit FedComBat's internal training plan to the nodes for approval, then wait for the node to approve it before running the pre-processing:

exp.preprocessing.approve("FedComBat harmonization models")

See Training with approved training plans.

Node permissions

FedComBat requires pre-processing and federated analytics to be allowed on every node of the experiment. Request the nodes to enable allow_preproc and allow_federated_analytics (see Changing a setting).

  • In a deployment, each node enables them only if its data governance policy accepts it.
  • In a local federation for testing, they can be relaxed: docker/local-federation.yml enables both.