What is Medallion?
Three quality layers — Bronze, Silver, Gold. Raw data arrives in Bronze, gets cleaned in Silver, and is modeled for BI in Gold. Principle: separate ingestion, quality, and consumption concerns completely.
Bronze — zero transformation, full fidelity
# Bronze ingestion — Delta + metadata columns
df_raw = spark.read.option("header", "true").csv("abfss://[email protected]/pos/")
df_raw \
.withColumn("_ingestion_ts", F.current_timestamp()) \
.withColumn("_source_file", F.input_file_name()) \
.write.format("delta").mode("append") \
.partitionBy("source_id", "load_date") \
.save("abfss://[email protected]/pos_data/")
Silver — SCD Type 2 merge
DeltaTable.forPath(spark, silver_path).alias("target") \
.merge(df_updates.alias("source"), "target.id = source.id AND target.is_current = true") \
.whenMatchedUpdate(
condition="source.updated_at > target.updated_at",
set={"is_current": "false", "end_date": "source.updated_at"}
).whenNotMatchedInsertAll().execute()
Gold — DBT Star Schema
SELECT s.sale_date, p.product_key, d.distributor_key,
SUM(s.quantity) AS total_units, SUM(s.revenue) AS total_revenue
FROM {{ ref('silver_pos_cleaned') }} s
JOIN {{ ref('dim_product') }} p ON s.product_id = p.product_id
GROUP BY 1,2,3