dl_data_pipeline.deferred package

Submodules

dl_data_pipeline.deferred.deferred_wrapper module

dl_data_pipeline.deferred.deferred_wrapper._create_pipeline_node(func, *args, **kwargs) → PipelineNode

dl_data_pipeline.deferred.deferred_wrapper.deferred_execution(func)

A decorator that defers the execution of a function until the actual data is provided.

The first argument of the function must be either DATA_PLACEHOLDER or None, indicating that the function execution should be deferred. When the function is eventually called with the actual data, the deferred execution is triggered.

Parameters:

func (callable) – The function to be deferred.

Returns:

A lambda function that takes the data as its first argument and executes

the original function with the deferred arguments.

Return type:

callable

dl_data_pipeline.deferred.deferred_wrapper.instant_excecution() → None

Context manager to temporarily disable deferred execution and execute functions immediately.

This context manager allows the user to execute functions immediately even when they are decorated with @deferred_function, bypassing the default deferred execution mode.

Upon entering the context (with instant_execution():), deferred execution is disabled. Once the block is exited, deferred execution is restored to its previous state, even if an exception occurs within the block.

Note

This context manager ensures that deferred execution is always restored after the with block, even if an exception is raised.

Example usage:

>>> @deferred_function
>>> def add(a, b):
>>>     return a + b

>>> # Deferred execution mode (default)
>>> result_node = add(PipelineNode(), 5)

>>> # Instant execution mode (within context)
>>> with instant_execution():
>>>     result = add(3, 5)
>>>     # Executes immediately and returns 8

Module contents

This module provides the deferred_execution decorator, which is used to defer the execution of functions until the necessary data is provided.

The deferred_execution decorator is particularly useful in scenarios where function execution needs to be delayed and performed as part of a data processing pipeline or computation graph. It wraps a function so that its execution is postponed until it receives data via PipeNode objects, enabling flexible and modular data processing workflows.

Functions:

deferred_execution(func): A decorator that defers the execution of a function until the

actual data is provided.

Usage Example:

>>> from dl_data_pipeline.deferred_execution import deferred_execution
>>> from dl_data_pipeline import PipeNode

>>> @deferred_execution
>>> def add(x, y):
>>>     return x + y

>>> input_node1 = PipeNode()
>>> input_node2 = PipeNode()

>>> add_node = add(input_node1, input_node2)

>>> input_node1._set_value(5)
>>> input_node2._set_value(3)

>>> add_node.execute()
>>> result = add_node.value  # result will be 8