Attributes¶
Classes¶
NodeStateFileName ¶
Bases: _BaseEnum
Collection of File names that composes a Node state. File names should contains 2 %s: one for round number, the second for state_id Example: VALUE = "some_values_%s_%s"
NodeStateManager ¶
NodeStateManager(dir, node_id, db_path)
Node state saving facility: Handles saving and loading Node states from previous Rounds, given a state_id. NodeStateManager ensures that states are not reset from one Round to another.
Interfaces with database use to save and load Node State entries.
Database Table that handles Node states is built with the following entries
- state_id
- experiment_id
- version_node_id: state id version, that checks if states can be used regarding FedBioMed versions.
- state_entries: content of each entry that cmposes a Node State
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dir | str | Root directory of the component | required |
node_id | str | Unique ID of the node | required |
db_path | str | path to the node state database | required |
Raises:
| Type | Description |
|---|---|
FedbiomedNodeStateManagerError | failed to access the database |
Source code in fedbiomed/node/node_state_manager.py
def __init__(self, dir: str, node_id: str, db_path: str):
"""Constructor of the class.
Args:
dir: Root directory of the component
node_id: Unique ID of the node
db_path: path to the node state database
Raises:
FedbiomedNodeStateManagerError: failed to access the database
"""
# NOTA: constructor has been designed wrt other object handling DataBase
self._dir = dir
self._node_id = node_id
self._query: Query = Query()
# node state base directory, where all node state related files are saved
self._node_state_base_dir: Optional[str] = None
self._state_id: Optional[str] = None
self._previous_state_id: Optional[str] = None
try:
self._connection = TinyDB(db_path)
self._connection.table_class = DBTable
self._db: Table = self._connection.table(
name=NODE_STATE_TABLE_NAME, cache_size=0
)
except Exception as e:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: Error found when loading database"
) from e
Attributes¶
previous_state_id property ¶
previous_state_id
state_id property ¶
state_id
Methods:¶
add ¶
add(experiment_id, state)
Adds new Node State into Database.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
experiment_id | str | experiment_id used to save entry in database | required |
state | Dict[str, Dict[str, str]] | state to be saved in database. | required |
Returns:
| Type | Description |
|---|---|
str | state_id |
Raises:
| Type | Description |
|---|---|
FedbiomedNodeStateManagerError | state manager is not initialized |
Source code in fedbiomed/node/node_state_manager.py
def add(self, experiment_id: str, state: Dict[str, Dict[str, str]]) -> str:
"""Adds new `Node` State into Database.
Args:
experiment_id: experiment_id used to save entry in database
state: state to be saved in database.
Returns:
state_id
Raises:
FedbiomedNodeStateManagerError: state manager is not initialized
"""
if self._state_id is None:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: Node state manager is not initialized"
)
header = {
"version_node_id": str(__node_state_version__),
"state_id": self._state_id,
"experiment_id": experiment_id,
}
state.update(header)
self._save_state(self._state_id, state)
return self._state_id
generate_folder_and_create_file_name ¶
generate_folder_and_create_file_name(
experiment_id, sequence_nb, element
)
Creates folders and file name for each content (Optimizer, model, ...) that composes a Node State.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
experiment_id | str | experiment_id | required |
sequence_nb | int | current Round number ou Step Number | required |
element | NodeStateFileName | a NodeStateFileName object used to create specific file names. For instance, could be a NodeStateFileName.OPTIMIZER | required |
Raises:
| Type | Description |
|---|---|
FedbiomedNodeStateManagerError | raised if node state manager is not initialized |
FedbiomedNodeStateManagerError | raised if folder can not be created. |
Returns:
| Type | Description |
|---|---|
str | path to the file that corresponds to the object that needs to be saved. |
Source code in fedbiomed/node/node_state_manager.py
def generate_folder_and_create_file_name(
self, experiment_id: str, sequence_nb: int, element: NodeStateFileName
) -> str:
"""Creates folders and file name for each content (Optimizer, model, ...) that composes
a Node State.
Args:
experiment_id: experiment_id
sequence_nb: current Round number ou Step Number
element: a NodeStateFileName object used to create specific file names. For instance,
could be a NodeStateFileName.OPTIMIZER
Raises:
FedbiomedNodeStateManagerError: raised if node state manager is not initialized
FedbiomedNodeStateManagerError: raised if folder can not be created.
Returns:
path to the file that corresponds to the object that needs to be saved.
"""
node_state_base_dir = self.get_node_state_base_dir()
if node_state_base_dir is None:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: working directory has not been "
"initialized, have you run `initialize` method beforehand ?"
)
if self._state_id is None:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: Node state manager is not initialized"
)
base_dir = os.path.join(node_state_base_dir, "experiment_id_%s" % experiment_id)
try:
os.makedirs(base_dir, exist_ok=True)
except Exception as e:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: Failing to create directories {base_dir}"
) from e
# TODO catch exception here
file_name = element.value % (sequence_nb, self._state_id)
path = os.path.join(base_dir, file_name)
logger.debug(
f"Generated node state file path: experiment_id={experiment_id} "
f"round={sequence_nb} element={getattr(element, 'name', repr(element))} path={path}"
)
return path
get ¶
get(experiment_id, state_id)
Returns a state of an experiment on the Node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
experiment_id | str | the experiment for which a state is requested | required |
state_id | str | the unique identifier of the experiment | required |
Returns:
| Type | Description |
|---|---|
Dict | dict containing the experiment state |
Raises:
| Type | Description |
|---|---|
FedbiomedNodeStateManagerError | no matching state in the database |
Source code in fedbiomed/node/node_state_manager.py
def get(self, experiment_id: str, state_id: str) -> Dict:
"""Returns a state of an experiment on the `Node`.
Args:
experiment_id: the experiment for which a state is requested
state_id: the unique identifier of the experiment
Returns:
dict containing the experiment state
Raises:
FedbiomedNodeStateManagerError: no matching state in the database
"""
logger.debug(
f"Loading node state: experiment_id={experiment_id} state_id={state_id}"
)
state = self._load_state(experiment_id, state_id)
if state is None:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: no entries matching"
f" experiment_id {experiment_id} and "
f"state_id {state_id} found in the DataBase"
)
# from this point, state should be a dictionary
self._check_version(state.get("version_node_id"))
return state
get_node_state_base_dir ¶
get_node_state_base_dir()
Returns Node State base directory path, in which are saved Node state files and other contents
Returns:
| Type | Description |
|---|---|
Optional[str] | path to |
Source code in fedbiomed/node/node_state_manager.py
def get_node_state_base_dir(self) -> Optional[str]:
"""Returns `Node` State base directory path, in which are saved Node state files and other contents
Returns:
path to `Node` state base directory, or None if not defined
"""
return self._node_state_base_dir
initialize ¶
initialize(previous_state_id=None, testing=False)
Initializes NodeStateManager, by creating folder that will contains Node state folders.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
previous_state_id | Optional[str] | state_id from previous Round, from which to reload a Node state | None |
testing | Optional[bool] | only doing testing, not training | False |
Raises:
| Type | Description |
|---|---|
FedbiomedNodeStateManagerError | raised if needed directories cannot be created |
Source code in fedbiomed/node/node_state_manager.py
def initialize(
self, previous_state_id: Optional[str] = None, testing: Optional[bool] = False
) -> None:
"""Initializes NodeStateManager, by creating folder that will contains Node state folders.
Args:
previous_state_id: state_id from previous Round, from which to reload a Node state
testing: only doing testing, not training
Raises:
FedbiomedNodeStateManagerError: raised if needed directories cannot be created
"""
self._previous_state_id = previous_state_id
logger.debug(
f"Initializing node state manager: node_id={self._node_id} "
f"previous_state_id={previous_state_id} testing={testing}"
)
if not testing:
self._generate_new_state_id()
self._node_state_base_dir = os.path.join(
self._dir, VAR_FOLDER_NAME, "node_state_%s" % self._node_id
)
# Always create the base folder for saving states for this node
try:
os.makedirs(self._node_state_base_dir, exist_ok=True)
except Exception as e:
raise FedbiomedNodeStateManagerError(
f"{ErrorNumbers.FB323.value}: Failing to create"
f" directories {self._node_state_base_dir}"
) from e
list_states ¶
list_states(experiment_id)
Source code in fedbiomed/node/node_state_manager.py
def list_states(self, experiment_id: str):
raise NotImplementedError
remove ¶
remove(experiment_id, state_id)
Source code in fedbiomed/node/node_state_manager.py
def remove(self, experiment_id: Optional[str], state_id: Optional[str]):
raise NotImplementedError