Base classes for pipelines and pipeline blocks.
Base class for all blocks in copper.
Blocks should take their parameters in
__init__and provide at least the
processmethod for taking in data and returning some result.
Process input data and produce a result.
Subclasses must implement this method, otherwise it shouldn’t really be a
Clear the state of the block.
Some blocks don’t keep stateful attributes, so
cleardoes nothing by default.
Feedforward arrangement of blocks for processing data.
To create a pipeline, the following two rules are needed: blocks in a list processed in series, and blocks in a tuple are processed in parallel.
Blocks that are arranged to take multiple inputs should expect to take the corresponding number of inputs in the order they are given. It is up to the user constructing the pipeline to make sure that the arrangement of blocks makes sense.
Parameters: blocks (container) – The blocks in the pipline, with lists processed in series and tuples processed in parallel.
dict – Dictionary of blocks in the pipeline. Keys are the names given to the blocks in the pipeline and values are the block objects.
processmethod of each block in the pipeline, passing the outputs around as specified in the block structure.
Parameters: data (object) – The input to the first block(s) in the pipeline. The type/format doesn’t matter to copper, as long as the blocks you define accept it. Returns: out – The data output by the
processmethod of the last block(s) in the pipeline.
Return type: object
clearmethod on each block in the pipeline. The effect depends on the blocks themselves.
PassthroughPipeline(blocks, expand_output=True, name=None)¶
Convenience block for passing input along to output.
A passthrough pipeline block is useful when you want to process some data then provide both the processed output as well as the original input to another block downstream.
CallablePipelineBlock(func, func_args=None, func_kwargs=None, name=None, hooks=None)¶
A PipelineBlock that does not require persistent attributes.
Many PipelineBlock implementations don’t require attributes to update on successive calls to the process method, but instead are essentially a function that can be called repeatedly. This class is for conveniently creating such a block.
If the function you want to use takes additional arguments, such as a keyword argument that
Note: if you use an anonymous function as the func argument, (e.g.
lambda x: 2*x), it is recommended to explicitly give the block a meaningful name.
- func (callable(data)) – Function that gets called when the block’s process method is called. Should take a single input and return output which is compatible with whatever is connected to the block.
- func_args (list, optional) – List (or tuple) of additional arguments to pass to func when calling it for processing. If None (default), no arguments are used.
- func_kwargs (dict) – Keyword argument name/value pairs to pass to func when calling it for processing. If None (default), no keyword arguments are used.
- name (str, optional, default=None) – Name of the block. By default, the name of the processor function is used.
- hooks (list, optional, default=None) – List of callables (callbacks) to run when after the block’s process method is called.