What is a pattern?
The term design pattern is a recurring concept in system architecture, as well as in data management and data warehousing.
The following explains what the term means in the context of MetaKraftwerk.
Generally, the definition of the term is rather abstract and technology-independent. In the field of data management, the term is even less precisely defined than, for example, in software development.
There are examples of patterns in the field of data integration for synchronization, aggregation, or migration of data, as well as for loading data into a data lakehouse (full vs. incremental).
A basic distinction can be made between patterns for low-code (ETL) and code platforms (Snowflake, PySpark notebooks, etc.).
In MetaKraftwerk, a pattern is always a specifically implemented data integration process for a particular target platform, whether it is a pipeline tool like Informatica or a database like Snowflake.
However, this process is designed to be instantiable, meaning it can be automatically created for different data entities. This is comparable to a higher-order function, which only becomes a concrete function when additional parameters are inserted.
In the case of MetaKraftwerk, these parameters are the Instance Metadata.
Depending on the target platform, a pattern consists of templates for files, as well as more complex constructs, such as a pipeline template of an ETL tool. The templates contain rules that define how metadata should be used in various places to create executable assets, such as a database load procedure, a DDL file, or an Informatica mapping. The process of creating an executable process instance of pattern is called a Build.
How it works
Develop the process on a target platform.
The process pattern is developed in a target platform such Informatica Data Management Cloud (IDMC), Azure Synapse or Informatica PowerCenter. The process itself is not executable as a generic pattern — it contains placeholders and parameterised parts where instance-specific values belong.Export the pattern metadata.
The metadata that describes the process (templates, file templates, pipeline definitions, mapping/configuration rules) is exported from the target platform and imported into MetaKraftwerk.Define instance metadata.
Instance metadata (see Instance Metadata) provides the concrete values for placeholders in the pattern: field names and datatypes, table names, column mappings, schedules, connection references, ownership/roles, etc.Instantiate the pattern.
MetaKraftwerk merges the process pattern and one instance’s metadata to create an executable process instance — filling placeholders in data structures and configuration properties to generate the actual artefacts (DDL, load procedures, mappings, pipeline definitions).Deploy to the target platform.
The generated, directly deployable processes can then be imported or deployed into the target platform.
Pattern components (examples)
To develop a pattern in MetaKraftwerk, there are a set of different editors, for example to develop:
- file templates (DDL, scripts) with substitution rules
- pipeline/mapping templates (ETL tool definitions) with parameterised parts
- configuration rules that dictate how metadata maps into templates
Patterns allow automation and standardisation: once a pattern is defined for a target platform, the same pattern can be reused to create many concrete process instances by providing different instance metadata — similar to a higher-order function that becomes a concrete function when parameters are supplied. All you have to do now is provide metadata in the form of instances and build these instances using the pattern.