Skip to content

NodeStateManager

Attributes

NODE_STATE_TABLE_NAME module-attribute

NODE_STATE_TABLE_NAME = 'Node_states'

Classes

NodeStateFileName

Bases: _BaseEnum

Collection of File names that composes a Node state. File names should contains 2 %s: one for round number, the second for state_id Example: VALUE = "some_values_%s_%s"

Attributes

MODEL_WEIGHTS class-attribute instance-attribute
MODEL_WEIGHTS = 'persistent_model_weights_%s_%s'
OPTIMIZER class-attribute instance-attribute
OPTIMIZER = 'optim_state_%s_%s'
PREPROC class-attribute instance-attribute
PREPROC = 'preproc_%s_%s'

NodeStateManager

NodeStateManager(dir, node_id, db_path)

Node state saving facility: Handles saving and loading Node states from previous Rounds, given a state_id. NodeStateManager ensures that states are not reset from one Round to another.

Interfaces with database use to save and load Node State entries.

Database Table that handles Node states is built with the following entries
  • state_id
  • experiment_id
  • version_node_id: state id version, that checks if states can be used regarding FedBioMed versions.
  • state_entries: content of each entry that cmposes a Node State

Parameters:

Name Type Description Default
dir str

Root directory of the component

required
node_id str

Unique ID of the node

required
db_path str

path to the node state database

required

Raises:

Type Description
FedbiomedNodeStateManagerError

failed to access the database

Source code in fedbiomed/node/node_state_manager.py
def __init__(self, dir: str, node_id: str, db_path: str):
    """Constructor of the class.

    Args:
        dir: Root directory of the component
        node_id: Unique ID of the node
        db_path: path to the node state database

    Raises:
        FedbiomedNodeStateManagerError: failed to access the database

    """
    # NOTA: constructor has been designed wrt other object handling DataBase
    self._dir = dir
    self._node_id = node_id
    self._query: Query = Query()
    # node state base directory, where all node state related files are saved
    self._node_state_base_dir: Optional[str] = None
    self._state_id: Optional[str] = None
    self._previous_state_id: Optional[str] = None
    try:
        self._connection = TinyDB(db_path)
        self._connection.table_class = DBTable
        self._db: Table = self._connection.table(
            name=NODE_STATE_TABLE_NAME, cache_size=0
        )
    except Exception as e:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: Error found when loading database"
        ) from e

Attributes

previous_state_id property
previous_state_id

Getter for previous state ID

Returns:

Type Description
Optional[str]

previous state ID, or None if does not exist

state_id property
state_id

Getter for state ID

Returns:

Type Description
Optional[str]

state ID, or None if not defined yet

Methods:

add
add(experiment_id, state)

Adds new Node State into Database.

Parameters:

Name Type Description Default
experiment_id str

experiment_id used to save entry in database

required
state Dict[str, Dict[str, str]]

state to be saved in database.

required

Returns:

Type Description
str

state_id

Raises:

Type Description
FedbiomedNodeStateManagerError

state manager is not initialized

Source code in fedbiomed/node/node_state_manager.py
def add(self, experiment_id: str, state: Dict[str, Dict[str, str]]) -> str:
    """Adds new `Node` State into Database.

    Args:
        experiment_id: experiment_id used to save entry in database
        state: state to be saved in database.

    Returns:
        state_id

    Raises:
        FedbiomedNodeStateManagerError: state manager is not initialized
    """

    if self._state_id is None:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: Node state manager is not initialized"
        )
    header = {
        "version_node_id": str(__node_state_version__),
        "state_id": self._state_id,
        "experiment_id": experiment_id,
    }

    state.update(header)
    self._save_state(self._state_id, state)
    return self._state_id
generate_folder_and_create_file_name
generate_folder_and_create_file_name(
    experiment_id, sequence_nb, element
)

Creates folders and file name for each content (Optimizer, model, ...) that composes a Node State.

Parameters:

Name Type Description Default
experiment_id str

experiment_id

required
sequence_nb int

current Round number ou Step Number

required
element NodeStateFileName

a NodeStateFileName object used to create specific file names. For instance, could be a NodeStateFileName.OPTIMIZER

required

Raises:

Type Description
FedbiomedNodeStateManagerError

raised if node state manager is not initialized

FedbiomedNodeStateManagerError

raised if folder can not be created.

Returns:

Type Description
str

path to the file that corresponds to the object that needs to be saved.

Source code in fedbiomed/node/node_state_manager.py
def generate_folder_and_create_file_name(
    self, experiment_id: str, sequence_nb: int, element: NodeStateFileName
) -> str:
    """Creates folders and file name for each content (Optimizer, model, ...) that composes
    a Node State.

    Args:
        experiment_id: experiment_id
        sequence_nb: current Round number ou Step Number
        element: a NodeStateFileName object used to create specific file names. For instance,
            could be a NodeStateFileName.OPTIMIZER

    Raises:
        FedbiomedNodeStateManagerError: raised if node state manager is not initialized
        FedbiomedNodeStateManagerError: raised if folder can not be created.

    Returns:
        path to the file that corresponds to the object that needs to be saved.
    """

    node_state_base_dir = self.get_node_state_base_dir()
    if node_state_base_dir is None:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: working directory has not been "
            "initialized, have you run `initialize` method beforehand ?"
        )

    if self._state_id is None:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: Node state manager is not initialized"
        )
    base_dir = os.path.join(node_state_base_dir, "experiment_id_%s" % experiment_id)
    try:
        os.makedirs(base_dir, exist_ok=True)

    except Exception as e:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: Failing to create directories {base_dir}"
        ) from e
    # TODO catch exception here
    file_name = element.value % (sequence_nb, self._state_id)
    path = os.path.join(base_dir, file_name)
    logger.debug(
        f"Generated node state file path: experiment_id={experiment_id} "
        f"round={sequence_nb} element={getattr(element, 'name', repr(element))} path={path}"
    )
    return path
get
get(experiment_id, state_id)

Returns a state of an experiment on the Node.

Parameters:

Name Type Description Default
experiment_id str

the experiment for which a state is requested

required
state_id str

the unique identifier of the experiment

required

Returns:

Type Description
Dict

dict containing the experiment state

Raises:

Type Description
FedbiomedNodeStateManagerError

no matching state in the database

Source code in fedbiomed/node/node_state_manager.py
def get(self, experiment_id: str, state_id: str) -> Dict:
    """Returns a state of an experiment on the `Node`.

    Args:
        experiment_id: the experiment for which a state is requested
        state_id: the unique identifier of the experiment

    Returns:
        dict containing the experiment state

    Raises:
        FedbiomedNodeStateManagerError: no matching state in the database
    """
    logger.debug(
        f"Loading node state: experiment_id={experiment_id} state_id={state_id}"
    )
    state = self._load_state(experiment_id, state_id)

    if state is None:
        raise FedbiomedNodeStateManagerError(
            f"{ErrorNumbers.FB323.value}: no entries matching"
            f" experiment_id {experiment_id} and "
            f"state_id {state_id} found in the DataBase"
        )

    # from this point, state should be a dictionary
    self._check_version(state.get("version_node_id"))
    return state
get_node_state_base_dir
get_node_state_base_dir()

Returns Node State base directory path, in which are saved Node state files and other contents

Returns:

Type Description
Optional[str]

path to Node state base directory, or None if not defined

Source code in fedbiomed/node/node_state_manager.py
def get_node_state_base_dir(self) -> Optional[str]:
    """Returns `Node` State base directory path, in which are saved Node state files and other contents

    Returns:
        path to `Node` state base directory, or None if not defined
    """
    return self._node_state_base_dir
initialize
initialize(previous_state_id=None, testing=False)

Initializes NodeStateManager, by creating folder that will contains Node state folders.

Parameters:

Name Type Description Default
previous_state_id Optional[str]

state_id from previous Round, from which to reload a Node state

None
testing Optional[bool]

only doing testing, not training

False

Raises:

Type Description
FedbiomedNodeStateManagerError

raised if needed directories cannot be created

Source code in fedbiomed/node/node_state_manager.py
def initialize(
    self, previous_state_id: Optional[str] = None, testing: Optional[bool] = False
) -> None:
    """Initializes NodeStateManager, by creating folder that will contains Node state folders.

    Args:
        previous_state_id: state_id from previous Round, from which to reload a Node state
        testing: only doing testing, not training

    Raises:
        FedbiomedNodeStateManagerError: raised if needed directories cannot be created
    """

    self._previous_state_id = previous_state_id
    logger.debug(
        f"Initializing node state manager: node_id={self._node_id} "
        f"previous_state_id={previous_state_id} testing={testing}"
    )
    if not testing:
        self._generate_new_state_id()

        self._node_state_base_dir = os.path.join(
            self._dir, VAR_FOLDER_NAME, "node_state_%s" % self._node_id
        )
        # Always create the base folder for saving states for this node
        try:
            os.makedirs(self._node_state_base_dir, exist_ok=True)
        except Exception as e:
            raise FedbiomedNodeStateManagerError(
                f"{ErrorNumbers.FB323.value}: Failing to create"
                f" directories {self._node_state_base_dir}"
            ) from e
list_states
list_states(experiment_id)
Source code in fedbiomed/node/node_state_manager.py
def list_states(self, experiment_id: str):
    raise NotImplementedError
remove
remove(experiment_id, state_id)
Source code in fedbiomed/node/node_state_manager.py
def remove(self, experiment_id: Optional[str], state_id: Optional[str]):
    raise NotImplementedError

Functions: