pyf.componentized.core.manager – PyF Componentized Manager | PyF, flow-based python programming pyf.componentized.core.manager – PyF Componentized Manager

pyf.componentized.core.manager – PyF Componentized Manager

pyf.componentized.Manager

class pyf.componentized.Manager(config_tree)

The main dataflow manager.

It will instanciate ProcessPipe objects for each of the process defined in the configuration tree.

Parameters:
  • config_tree (ElementTree object) – Configuration Tree
get_process_config(name)

Returns the particular named process config in the configuration

process(name=None, params=None, progression_callback=None, message_callback=None, finalize=False, process_config=None, options=None)

Initializes ProcessPipes, and launches processing. Launches the process with the given name. If no name is given, it will run all the available processes in the configuration.

Parameters:
  • name (string or None) – If given, will only launch process with the given name, if not, will launch all the processes
  • params (dict) – Parameters to be passed to the process (available to the producers)
  • progression_callback (function) – A function that will be called every time there is information on the process progression (can receive integers or decimal.Decimal)
  • message_callback (function) – A function that will be called with information on the process.
  • finalize (bool) – Should the process finalize (copy output files to asked folders) or not. Default: False.
  • options (dict or None) – Options to be applied on the process tree. It will replace specified entries in the main config node with values from the passed dictionnary (if an option hasn’t been specified in a component config, it will default to the root config, and that’s what you can modify at run time here).
  • process_config (ElementTree object) – if specified, ignore main config tree, and process with this tree instead.

pyf.componentized.core.manager.ProcessPipe

class pyf.componentized.core.manager.ProcessPipe(config_tree, data_manager, process_name, options=None)

A single process manager, instanciating all plugins needed, parsing the configuration tree and giving a simple way to launch it.

Parameters:
  • config_tree – The process configuration tree
  • data_manager (pyf.manager.DataManager instance) – The manager that will handle the network and data
initialize(progression_callback=None, message_callback=None, params=None)

Initializes the process (compile embeded python source code, load plugins, link everything and transform it in a low-level pyf.manager network) and adds it to the data manager as a network.

Parameters:
  • progression_callback (function) – A function that will be called every time there is information on the process progression (can receive integers or decimal.Decimal)
  • message_callback (function) – A function that will be called with information on the process.
  • finalize – Should the process finalize (copy output files to asked folders) or not. Default: False.
  • params (dict) – Parameters to be passed to the process (available to the producers)
parse_node(config_node, before=None)

Parses a specific node from ET format

process(progression_callback=None, message_callback=None, params=None, finalize=False)

Launches process. (no need to call initialize before, as this function calls it too)

Parameters:
  • progression_callback (function) – A function that will be called every time there is information on the process progression (can receive integers or decimal.Decimal)
  • message_callback (function) – A function that will be called with information on the process.
  • params (dict) – Parameters to be passed to the process (available to the producers)
  • finalize (bool) – Should the process finalize (copy output files to asked folders) or not. Default: False.
splitter(source, out, size=3)

Splits a data source in n sources (size kwarg). This is the default splitter if no other is specified.