#Data Engineering
4 posts tagged Data Engineering. All posts
Idempotent ETL jobs: watermarks, upserts and safe rerunsBuild an incremental extract-and-load job that can run twice without harm: high-watermarks, an overlap window for late rows, SQLite upserts and run ids.
dbt Core with DuckDB: models, tests and incremental tablesBuild a small dbt project on DuckDB: the dbt-duckdb profile, CSV sources, staging and mart models with ref, generic data tests, an incremental model and dbt build.
How Parquet files work: row groups, column chunks and statisticsWhat is inside a Parquet file: row groups, column chunks, pages, min/max statistics and predicate pushdown, codecs and encodings, inspected with pyarrow.parquet.
Polars for pandas users: expressions, lazy frames and group_byThe same DataFrame transformations in pandas and Polars side by side: pl.col expressions, with_columns, group_by().agg, window functions, lazy scan_csv and no index.
© 2026 David Li · Home · Blog · RSS ·Classic site · Built with Astro, Starlight and three.js · Logos:Font Awesome Free (CC BY 4.0)