Databricks

Important

This guide is only relevant if you are using MetaKraftwerk's Databricks integration. If you are only developing ETL patterns, you can skip this page.

Databricks is a cloud-based platform that combines data engineering, data science, and machine learning.
MetaKraftwerk supports creating reusable ELT patterns that can be instantiated and deployed as Databricks jobs.

Uploading Templates from Databricks

Templates must first be developed in Databricks and then uploaded to MetaKraftwerk.
In MetaKraftwerk, these templates are enriched with dynamic rules to make them deployable as parameterized Databricks jobs.

A template can contain:

• Notebooks (Python, SQL, or Scala)
• SQL scripts
• Job JSON definitions

Export Notebook from Databricks

• In the Databricks workspace, open the notebook you want to export.
• Click File → Export → Source File (.dbc) or Download as .ipynb.
• Export additional job definitions as JSON if needed.

Import Notebook into MetaKraftwerk

• Upload the .dbc or .ipynb files as template assets to the pattern.
• Define placeholders (e.g. ${TABLE}, ${DATABASE}, ${INPUT_PATH}, ${OUTPUT_PATH}) where instance metadata will be inserted.