Plugin extensions beyond pipeline components

[dlvenable]

Problem

The term pipeline component refers to a source, buffer, processor, or sink. Today, Data Prepper plugins only support creating pipeline components. (Or in some cases, sub-configurations within a component.) The opensearch sink is an example of component plugin. Data Prepper creates an instance of this class for each sink declared in a pipeline configuration.

However, Data Prepper has no solution for extending existing functionality beyond pipeline components. Nor does it have a solution to add new functionality to the overall system.

One use-case for this is described in #2570. This proposes:

• Extending the data-prepper-config.yaml to support AWS credential configurations.
• Adding new functionality for other plugins to use which allow them to get credentials from a common mechanism.

Another use-case is supporting variable injection (through environment variables or AWS Secrets). For example, see #2780.

Another problem extensions could help with is sharing resources. For example, we could possibly share data across multiple opensearch sinks through the use of extensions. See #2589 for more details.

Solution

Provide a mechanism for extending Data Prepper functionality. Extensions will provide this mechanism.

An extension is a global component that can be used across all of Data Prepper. Other pipeline components (sources, processors, sinks) can use extensions. And even Data Prepper core functionality can be modified by using extensions.

These extensions should allow:

• Adding new configurations in data-prepper-config.yaml
• Creating new types that can be shared within the plugin framework. These types can themselves be registered as plugins.
• Add new configurations within a pipeline YAML file

Tasks

• Create starting point interfaces for extensions #2636
• Create support for extension-related objects which can be loaded by other Data Prepper plugins #2637
• Allow injecting configurations into data-prepper-config.yaml #2638
• Allow extracting configurations which were injected into data-prepper-config.yaml into plugins via the DataPrepperPluginConstructor. #2824
• Allow injecting configurations into the pipeline_configurations #2826
• Allow extensions to depend upon other extensions #2825