core

Base classes for pipelines and pipeline blocks.

class copper.core.PipelineBlock(name=None, hooks=None)

Base class for all blocks in copper.

Notes

Blocks should take their parameters in __init__ and provide at least the process method for taking in data and returning some result.

process(data)

Process input data and produce a result.

Subclasses must implement this method, otherwise it shouldn’t really be a PipelineBlock.

clear()

Clear the state of the block.

Some blocks don’t keep stateful attributes, so clear does nothing by default.

class copper.core.Pipeline(blocks, name=None)

Feedforward arrangement of blocks for processing data.

A Pipeline contains a set of :class:`PipelineBlock`s which operate on data to produce a final output.

To create a pipeline, the following two rules are needed: blocks in a list processed in series, and blocks in a tuple are processed in parallel.

Blocks that are arranged to take multiple inputs should expect to take the corresponding number of inputs in the order they are given. It is up to the user constructing the pipeline to make sure that the arrangement of blocks makes sense.

Parameters:blocks (container) – The blocks in the pipline, with lists processed in series and tuples processed in parallel.
named_blocks

dict – Dictionary of blocks in the pipeline. Keys are the names given to the blocks in the pipeline and values are the block objects.

process(data)

Calls the process method of each block in the pipeline, passing the outputs around as specified in the block structure.

Parameters:data (object) – The input to the first block(s) in the pipeline. The type/format doesn’t matter to copper, as long as the blocks you define accept it.
Returns:out – The data output by the process method of the last block(s) in the pipeline.
Return type:object
clear()

Calls the clear method on each block in the pipeline. The effect depends on the blocks themselves.

class copper.core.PassthroughPipeline(blocks, expand_output=True, name=None)

Convenience block for passing input along to output.

A passthrough pipeline block is useful when you want to process some data then provide both the processed output as well as the original input to another block downstream.

class copper.core.CallablePipelineBlock(func, func_args=None, func_kwargs=None, name=None, hooks=None)

A PipelineBlock that does not require persistent attributes.

Many PipelineBlock implementations don’t require attributes to update on successive calls to the process method, but instead are essentially a function that can be called repeatedly. This class is for conveniently creating such a block.

If the function you want to use takes additional arguments, such as a keyword argument that

Note: if you use an anonymous function as the func argument, (e.g. lambda x: 2*x), it is recommended to explicitly give the block a meaningful name.

Parameters:
  • func (callable(data)) – Function that gets called when the block’s process method is called. Should take a single input and return output which is compatible with whatever is connected to the block.
  • func_args (list, optional) – List (or tuple) of additional arguments to pass to func when calling it for processing. If None (default), no arguments are used.
  • func_kwargs (dict) – Keyword argument name/value pairs to pass to func when calling it for processing. If None (default), no keyword arguments are used.
  • name (str, optional, default=None) – Name of the block. By default, the name of the processor function is used.
  • hooks (list, optional, default=None) – List of callables (callbacks) to run when after the block’s process method is called.