Ever been bogged down by data pipelines crashing due to memory issues? It's a frustratingly common problem in data engineering projects. This post chronicles my experience of identifying and resolving memory bottlenecks in our data processing using t...
In my last blog, I showed how to use Polars in a Fabric Python notebook to read, write and transform data. What I did not cover was how to write a Delta Lake table in a Python notebook that’s optimal for DirectLake performance. All engines in Fabric ...
1. Convert Column Names to Snake Case (pyjanitor.clean_names) import pandas as pd import janitor # Sample DataFrame df = pd.DataFrame({"Column Name 1": [1, 2], "AnotherColumn": [3, 4]}) # Convert column names to snake_case df = janitor.clean_names(...
The much-anticipated Python notebook in Fabric is finally available and the Fabric users have already developed cool libraries and blogged about the usefulness of these notebooks. Duckdb is everyone’s favorite, but I am a Python guy 😁 so here is qui...
Apache Iceberg Crash Course: What is a Data Lakehouse and a Table Format? Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse Engineering Video Playlist Data engineers and scientists often work wit...
There is an interesting trend in the data community involving the rebuilding of popular products using Rust. Polars is one example of this, and it has really caught my interest. YADL (Yet Another Dataframe Library)? I've seen many articles claiming t...
1. Creating DataFrames (pl.DataFrame) 🏗️ Boilerplate Code: import polars as pl Use Case: Create a DataFrame to hold your data, similar to pandas. 🏗️ Goal: Store and manipulate data in a high-performance DataFrame structure. 🎯 Sample Code: # Creat...
In Part 1 of this series, we explored the characteristics of NS&I Premium Bonds and created a function to simulate the results of a single monthly prize draw for a given bond holding. In Part 2, we built a Monte Carlo experiment, simulated 6 million ...
💡 This is not a benchmark. This is one specific instance of a test, on a specific dataset using specific transformations. My goal was just to see how to set up daft + ray in Fabric and compare for the sake of learning. Daft is a distributed query ...
Abstract As the field of data processing and analysis continues to advance, it is becoming increasingly crucial to select the appropriate tools for the task at hand. The purpose of this white paper is to present an extensive comparison of three well-...
