Feed
Discussions

#apache-spark

#apache-spark

#apache-spark

46 followers·85 articles

#apache-spark

#apache-spark·46 followers·85 articles

#apache-spark

Changelog

New steps component and improved accessibility on Hashnode's blog and docs product.

Nov 05, 2024·

new

Trending Articles

Understanding Oone Gas and Fees: The Fuel Behind Every Blockchain Transaction.

Patience Uyire·55 reads

SolvBTC Rolls Out a Next-Gen Platform for Secure, Scalable, and Transparent Bitcoin DeFi

Wissy Ak·45 reads

SolvBTC, 안전하고 확장 가능하며 투명한 비트코인 디파이 플랫폼 출시

Wissy Ak·36 reads

Top commenters this week

Writing Challenges

#2Articles1Week Challenge

Become better at technical writing; accept Hashnode's writing challenge for four weeks.

#2Articles1Week Challenge

#WomenWhoTech

Share your story, achievements, or experiences as a woman, non-binary folk in tech or as a #WomenWhoTech ally!

#WomenWhoTech

Self Starter

Publish your first article on Hashnode and become a self starter!

Self Starter

Serial Blogger

Publish an article every day for 7 days and earn a cool serial blogger badge!

Serial Blogger

Talk of the town

Write a story that drives amazing engagement on Hashnode and become the talk of the town!

Talk of the town

Word Warrior

Write an in-depth article on your Hashnode blog that's more than 2000 words and become a word warrior!

Word Warrior

Lyn Alden

Kora Benson

Manjiri C

Ritesh

alejandro saenz

alejandro saenz

Islam Elbanna

handling-data-skew-in-spark-with-salting.hashnode.dev·May 10, 2025

May 10, 2025

Apache Spark: Fix data skew issue using salting technique

When working with large datasets in Apache Spark, a common performance issue is data skew. This occurs when a few keys dominate the data distribution, leading to uneven partitions and slow queries. It mainly happens during operations that require shu...

Apache Spark: Fix data skew issue using salting technique

Discuss·10 likes·25 reads

Yatin B.

blogs.stackedmind.com·May 08, 2025

May 08, 2025

Spark/PySpark

Date: 2024-01-22 Apache Spark is a powerful, open-source big data processing system offering speed and scalability. Its comprehensive ecosystem includes various components extending its capabilities across diverse domains. Spark boasts significant ...

spark ecosystem

Yatin B.

blogs.stackedmind.com·May 08, 2025

May 08, 2025

Create Your First Dataframe In Pyspark

Date: 2025-01-15 This article introduces PySpark, Python's interface for Apache Spark, a powerful distributed computing system for big data processing. It focuses on creating PySpark DataFrames, a distributed, tabular data structure analogous to pan...

Venkatesh Marella

Venkatesh Marella

data-engineer-solutions.hashnode.dev·Apr 16, 2025

Apr 16, 2025

Build your first ETL pipeline using PySpark

Overview In this session, we’ll walk through building a simple ETL (Extract, Transform, Load) pipeline using PySpark Steps Involved Setup Spark Environment Read Data from CSV Cleanse and Transform the CSV data Write final results to target locati...

Discuss·3 reads

Abhay Ahirkar

abhay-ahirkar.hashnode.dev·Apr 15, 2025

Apr 15, 2025

Building an End-to-End Lending Data Analytics Platform: My DE Zoomcamp 2025 Final Project

Introduction As part of my DE Zoomcamp 2025 journey, I developed a comprehensive data engineering solution for the lending industry. The project addresses a critical business need: helping financial institutions make data-driven lending decisions by ...

Building an End-to-End Lending Data Analytics Platform: My DE Zoomcamp 2025 Final Project

Discuss·1 read

Nextwebb

nextwebb.hashnode.dev·Jan 10, 2025

Jan 10, 2025

Data Engineering Foundations: A Hands-On Guide

Data Engineering Foundations: A Hands-On Guide Hey there! If you’ve been curious about data engineering, this guide will help you understand the basics and walk you through practical examples. Whether it’s setting up storage, processing data, automat...

Data Engineering Foundations: A Hands-On Guide

Discuss·77 reads

Shrinivas Vishnupurikar

Shrinivas Vishnupurikar

data-engineering-with-shrinivasv73.hashnode.dev·Dec 11, 2024

Dec 11, 2024

The Spark Eco-System with not very often elaborated Components

The article’s title is justified because when I was studying the Spark’s Ecosystem, I had a lot questions with respect to which components are siblings to each other or which are nested one another, and also had “Does such a component actually exists...

The Spark Eco-System with not very often elaborated Components

Discuss·15 reads

Jitender Kaushik

Jitender Kaushik

jitenderkaushik.com·Nov 08, 2024

Nov 08, 2024

Exploring Microsoft Fabric: Notebooks vs. Spark Jobs and How Java Fits In

Microsoft Fabric offers a versatile platform for data processing, blending interactive notebooks with powerful Spark jobs. While both tools serve different purposes, understanding their distinctions can optimize your workflows, especially with Java c...

Exploring Microsoft Fabric: Notebooks vs. Spark Jobs and How Java Fits In

Discuss·37 reads

microsft fabric notebook

Alex Merced

alexmerced.hashnode.dev·Oct 19, 2024

Oct 19, 2024

Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake

Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse Engineering Video Playlist Maintaining a persistent Airflow deployment can often add significant overhead to data engineering teams, especially when ...

Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake

Discuss·2 reads

Sharath Kumar Thungathurthi

Sharath Kumar Thungathurthi

sharaththungathurthi.hashnode.dev·Oct 19, 2024

Oct 19, 2024

Unlock PySpark’s Power: Techniques for Parallelizing

Conceptual Questions What is parallelize in PySpark? parallelize is a method in PySpark used to convert a Python collection (like a list or a tuple) into an RDD (Resilient Distributed Dataset). This allows you to perform parallel processing on the ...

Discuss·1 like·29 reads