7:50 · Music video
The Iceberg Open Lakehouse
A visual tour of the open data architecture beneath modern analytics and AI.
Open videoDataEngr.com
A working reference for data engineers. 200 terms, each with a definition you can use in a meeting and an explanation you can build from.
Pipelines, storage, modeling, orchestration, governance, and the lakehouse and AI architectures they are all converging on. Written and maintained byAlex Merced, co-author of Apache Iceberg: The Definitive Guide.
Featured films
Two anthems for an open data future
Music-driven explainers for the open lakehouse and the open agentic stack.
7:37 · Music video
Open the Stack
A manifesto for composable agents, open standards, model choice, and portable execution.
Open videoStart Here
The terms most people look up first. Each one links out to the concepts around it, so you can follow a thread as far as you need to.
Common Questions
What does a data engineer actually do?
A data engineer builds and operates the systems that move data from the places it is created into the places it gets analyzed. That covers ingestion, storage, transformation and serving, plus the orchestration, testing and monitoring that keep those pipelines producing correct results on every run, not just the first one.
Read the full entryWhat is the difference between ETL and ELT?
ETL transforms data on a separate tier before loading it, so the destination only ever stores modeled tables. ELT loads raw data into the warehouse or lakehouse first and transforms it there using the platform's own compute. ELT keeps a replayable raw copy; ETL discards it at transform time.
Read the full entryWhat is a data lakehouse?
A data lakehouse is open files on object storage with a table format layered over them. The table format, usually Apache Iceberg, tracks every data file in metadata rather than inferring a table from a directory, which restores ACID transactions, schema evolution and time travel to data that any engine can still read.
Read the full entryDo I need streaming, or is batch enough?
Batch is enough whenever the business can act on data that is minutes or hours old, and it is cheaper and simpler to operate. Streaming earns its complexity when a decision has to happen within seconds of the event. Micro-batching sits between the two and covers a large share of real requirements.
Read the full entryWhere should I start if I am new to data engineering?
Start with how analytical data is shaped and where it lives: data modeling, the difference between a warehouse, a lake and a lakehouse, and how a pipeline is scheduled. Those three ideas explain most architecture decisions you will meet, and every other term connects back to them.
Read the full entryWorth Reading
Semantic Layer: The Definitive Guide
Why a shared metric layer sits above the tables rather than inside each BI tool.
Read ArticleApache Polaris: The Catalog Standard
How an open catalog keeps the lakehouse readable by more than one engine.
Read ArticleTable Formats Explained
What a table format does, and the problems it was invented to fix.
Read ArticleWhat Apache Iceberg Native Actually Means
Separating real Iceberg support from a connector with an Iceberg label on it.
Read ArticleWhat is Agentic Analytics?
What changes when an agent, not a person, is the one asking the questions.
Read ArticleCommunities & Events
Join the Conversation
The Data Lakehouse Hub Slack
Community Slack for data lakehouse practitioners and enthusiasts.
Data Events Organizing Slack
Slack community for organizing and coordinating data events.
Dremio Developer Community
Join the Dremio developer community for support and collaboration.
Dremio Dev Slack
Direct Slack workspace for Dremio developers.
Apache Iceberg Community
Official Apache Iceberg open source community.
Apache Polaris Community
Official Apache Polaris open source community.
Apache Arrow Community
Official Apache Arrow open source community.
Upcoming Events
Dremio Events & Workshops
Register for upcoming Dremio workshops, webinars, and hands-on sessions.
International Data Lakehouse Meetups
Luma calendar for worldwide lakehouse events and meetups.
Open Data Lakehouse Meetups
Global meetup group for open data lakehouse enthusiasts.
North American Open Data Lakehouse Linkups
Connect with lakehouse practitioners across North America.
NA Community Run Apache Iceberg Meetups
Community-organized Apache Iceberg meetups across North America.
Lakehouse Linkups
Lakehouse linkup events calendar on Luma.
NYC Data Lakehouse Events
Data lakehouse events in the New York City area.
Orlando Data Events
Data community events in the Orlando, Florida area.
West Coast US Data Lakehouse Events
Events in SF, Seattle, Denver, and more.
East Coast US Data Lakehouse Events
Events in NYC, Boston, Atlanta, Austin, and more.