Azure Synapse Analytics

Introduction

This guide uses screenshots and descriptions to walk you through creating a Simple Landing-to-Bronze data integration pattern with Azure Synapse Analytics and MetaKraftwerk. The pattern will be able to generate Synapse notebooks that read CSV data from a landing zone, transform it and finally load it into a bronze layer in Delta format. The example assumes that you have a Medallion architecture. If you have a different architecture or layer structure, that's fine; the example is for illustrative purposes only.

Goal

By the end of this tutorial, you will:

• Create a Pattern in MetaKraftwerk (MKW) for Landing to Bronze.
• Configure the Instances in MKW.
• Run a Synapse notebook to:
  • Read CSV from Landing.
  • Compute HASH_KEY from key columns.
  • Compute HASH_VALUE from descriptive columns.
  • Write data as Delta format to Bronze.

Prerequisites

• Access to MetaKraftwerk and a Project workspace.

• Azure Synapse workspace and a Spark pool.

• Access to ADLS Gen2 ("Storage Blob Data Contributor" role).

• Sample landing data:

  Download csv_landing_data_for_synapse_tutorial.zip

• Sample Instances for Metakraftwerk:

  Download instances_for_synapse_tutorial.zip

• Sample Pattern file for Metakraftwerk:

  Download bronze_dv_prep_support_live.zip

• Sample Notebook for Synapse:

  landing_2_bronze.py

• Sample Notebooks and Pipelines for Synapse:

  SYNAPSE_DEMO.zip

• Sample Pipeline (bronze_dv_prep) for Synapse:

  Download bronze_dv_prep.json

• Bronze Master Pipeline with For Each activity (PL_LDG_2_BRONZE) for Synapse:

  Download PL_LDG_2_BRONZE.json

Tutorial Parts

Part 1 — Create a Pattern in MetaKraftwerk
Learn how to define and configure a reusable pattern in MetaKraftwerk as the foundation for your Synapse project.

Part 2 — Create Instances of the Pattern in MKW
Learn how to create specific instances in MetaKraftwerk, including the configuration required to generate operational code.

Part 3 — Synapse: Prepare and Create a Notebook
Set up your Azure Synapse workspace and create a notebook for running your data processing logic.

Part 4 — Synapse: Create a Pipeline and Download Support Files
Build and configure a pipeline in Synapse to orchestrate data movement and processing tasks. Download the support files of the pipeline, which will be uploaded to MKW as a pattern template.

Part 5 — Upload & Develop Pattern in MKW
Upload your pattern to MetaKraftwerk and extend your pattern functionality.

Part 6 — Build the Instances and Deploy to Synapse
Build the Instances, download build results and Import the Noteboooks into Synapse.

Part 7 (optional) — Add a Dynamic Markdown Header
Add a dynamic markdown header to improve readability and documentation within your Notebooks.

Part 8 (optional) — Create a Master Pipeline in Synapse
Create a master pipeline that orchestrates the execution of multiple Bronze notebooks in parallel.

---

Each part builds on the previous one — so start from Part 1 and work your way down.
When you complete all steps, you’ll have a fully operational, scalable Synapse setup powered by MetaKraftwerk.