A network-style manager. The Network object sets up nodes and relation between them from a dictionnary representation. The nodes object do prelink and link from the bottom up (consumers to producers) The Network can then set up a status handler and furnish a final iterator over statuses.
- class pyf.manager.network.DataJunction(joiner=None, strip_bypass=True)¶
A DataJunction is an object specialised in taking multiple sources, setting up a joiner runner and permitting to get a generator output.
It handles BYPASS_VALS constants to pass them to the output (only one if there are many).
- add_input_source(source, name)¶
Appends a named input source
Removes the continuation values (values defined in BYPASS_VALS) and puts them in the queue. Returns an iterator over non bypassed values
This is done so the joiner runner doesn’t have to handle the bypassing values.
Sets up all the sources in the runner and returns an iterator over the output port of the junction runner.
Warning, this function shouldn’t be called twice !
Outputs the values passed in, and if there is something in the queue, pass it, but discard all other objects in the queue until next input data. (If there is a continuation, just pass one, and not the others... It’s usefull if there is a split then a merge with continuation vals, to avoid duplication of sync packets)
- class pyf.manager.network.Network(nodes=None)¶
Initialises a network of nodes from a dictionary structure. Once the nodes have been set up, you can either get the consumer iterators, or get a status handler checking that all the outputs resolve to True.
Adds a node to the main node dictionnary
Returns the consumers Node objects
Return the outputs from the consumer Node objects
Returns a status handler that returns True each time it receives a True from parents.
Experimental: if “advanced” is set to True, uses an intelligent status handler that looks for parent nodes until it finds a splitting node and consumes all the buffer on each branch to avoid high memory usage
Creates the Node objects from the dictionnary data structure, set up the links between them and prelink_up then link_up the consumers.
Return the producers Node objects
- class pyf.manager.network.Node(node_id, orunner, in_port=None, out_port=None, joiner=None, splitter=None, joiner_strip_bypass=True, deffer_to_process=False)¶
Object that is here to be linked to other nodes. Requires a node_id and a runner object (orunner).
First step is to prelink, then to link. Order is from consumers up to producers :
- Consumers take output from up nodes than themselves take sources up...
- If there is multiple source, the joiner argument is used (or a default) and encapsulated in a DataJunction
- If there is multiple output, the splitter argument is used (or a default)
Adds a node as an input
Adds a node as an output
Returns a default splitter (basically the same than an itertools tee: pass the same data in all outputs)
Returns an output generator. If there is one output, return the main runner out port, else, return a free port from the splitter
Connect data inputs in the main runner or in a junction connected to the main runner. If needed, connect output in the splitter.
Returns true if there are no output ports
Returns true if there are no input ports
Link to the input nodes. (ask up to do the same) If not inited yet, init the current node
Returns the name (id) of the current node
If there is a splitter needed, initialize it. If there is a junction needed, initialize it.
Prelinks the input nodes... (ask up to do the same)