From Warehouse-First to Lakehouse-First: What Changed for Me

Inside our company, we’ve been trying to cut through the noise of all the “choose‑your‑own‑adventure” data storage patterns we’ve seen over the years. Every client has its own habits, every project has its own backstory, and every architect has their own comfort zone.

For a long time, my comfort zone was the Warehouse. Coming from a SQL Server background, it was the world I knew. The tooling made sense, the behavior was predictable, and it matched how I had built systems in the past. The more time I spent in Fabric working with Lakehouses though, the more my thinking shifted.

The more projects I worked on in Fabric the more I noticed that the warehouse was more rigid than I wanted. If I needed to bring unstructured data into Fabric, I had to ingest with a Lakehouse anyway and then move data to a warehouse. If I wanted to take advantage of shortcuts to avoid storage duplication, I had to use a Lakehouse. Lakehouses are also compatible with more tech than Warehouses. A notebook can be attached to a Lakehouse. That notebook can be used to interact with a Lakehouse using, Pyspark, Spark, Python. I can still use T-SQL when I want to through the Lakehouse SQL Endpoint or Notebooks. To take advantage of the most Fabric data unity benefits, Lakehouse is the way to go.

This post is really about my “why” behind my Lakehouse-first mindset. I still use warehouses. They have their place and they work well in Fabric. In my next entry, I’ll discuss use cases where I would use a warehouse instead.

‍

COMPETITIVE EDGE

Please contact us to explore how SDP can create a competitive edge for your business.

ABOUT THE AUTHOR

Valerie Gutraj is a Senior Data and Analytics Engineer at Software Design Partners.

‍

‍

‍