Streaming data processing has come a long way, so why stick to old methods and not use modern practices. Let me share my fresh perspective that can help you solve your problem. Inspiration from Batch Processing Batch Processing shines with below (tho...
Every data team knows the drill: a PM needs to “just take a quick look” at some Parquet data. That usually means asking an engineer to write SQL or spin up a tool to pull a few rows. It’s a small ask, but one that happens often enough to slow everyon...
Ever been bogged down by data pipelines crashing due to memory issues? It's a frustratingly common problem in data engineering projects. This post chronicles my experience of identifying and resolving memory bottlenecks in our data processing using t...
In my last blog, I showed how to use Polars in a Fabric Python notebook to read, write and transform data. What I did not cover was how to write a Delta Lake table in a Python notebook that’s optimal for DirectLake performance. All engines in Fabric ...
1. Convert Column Names to Snake Case (pyjanitor.clean_names) import pandas as pd import janitor # Sample DataFrame df = pd.DataFrame({"Column Name 1": [1, 2], "AnotherColumn": [3, 4]}) # Convert column names to snake_case df = janitor.clean_names(...
The much-anticipated Python notebook in Fabric is finally available and the Fabric users have already developed cool libraries and blogged about the usefulness of these notebooks. Duckdb is everyone’s favorite, but I am a Python guy 😁 so here is qui...
Apache Iceberg Crash Course: What is a Data Lakehouse and a Table Format? Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse Engineering Video Playlist Data engineers and scientists often work wit...
There is an interesting trend in the data community involving the rebuilding of popular products using Rust. Polars is one example of this, and it has really caught my interest. YADL (Yet Another Dataframe Library)? I've seen many articles claiming t...
1. Creating DataFrames (pl.DataFrame) 🏗️ Boilerplate Code: import polars as pl Use Case: Create a DataFrame to hold your data, similar to pandas. 🏗️ Goal: Store and manipulate data in a high-performance DataFrame structure. 🎯 Sample Code: # Creat...
In Part 1 of this series, we explored the characteristics of NS&I Premium Bonds and created a function to simulate the results of a single monthly prize draw for a given bond holding. In Part 2, we built a Monte Carlo experiment, simulated 6 million ...
