Feed
Discussions

#apache-spark

#apache-spark

#apache-spark

46 followers·82 articles

#apache-spark

#apache-spark·46 followers·82 articles

#apache-spark

Changelog

New steps component and improved accessibility on Hashnode's blog and docs product.

Nov 05, 2024·

new

Trending Articles

Unlocking the Future of Web3 Communication: Your Guide to the Dmail Network

Nsi Berry·28 reads

Beyond the Hype: Why Real Web3 Projects Like Dmail Are Here to Stay

Nsi Berry·23 reads

Smarter Messaging in Web3: Why $DMAIL Is the Key to a Cleaner, Safer Inbox

Nsi Berry·27 reads

Top commenters this week

Shreyash Shukla

Shreyash Shukla

Julius Job

GreenFlux

Aakashi Jaiswal

Aakashi Jaiswal

Ruturaj Bayad

Writing Challenges

#2Articles1Week Challenge

Become better at technical writing; accept Hashnode's writing challenge for four weeks.

#2Articles1Week Challenge

#WomenWhoTech

Share your story, achievements, or experiences as a woman, non-binary folk in tech or as a #WomenWhoTech ally!

#WomenWhoTech

Self Starter

Publish your first article on Hashnode and become a self starter!

Self Starter

Serial Blogger

Publish an article every day for 7 days and earn a cool serial blogger badge!

Serial Blogger

Talk of the town

Write a story that drives amazing engagement on Hashnode and become the talk of the town!

Talk of the town

Word Warrior

Write an in-depth article on your Hashnode blog that's more than 2000 words and become a word warrior!

Word Warrior

Venkatesh Marella

Venkatesh Marella

data-engineer-solutions.hashnode.dev·Apr 16, 2025

Apr 16, 2025

Build your first ETL pipeline using PySpark

Overview In this session, we’ll walk through building a simple ETL (Extract, Transform, Load) pipeline using PySpark Steps Involved Setup Spark Environment Read Data from CSV Cleanse and Transform the CSV data Write final results to target locati...

Discuss·1 read

Abhay Ahirkar

abhay-ahirkar.hashnode.dev·Apr 15, 2025

Apr 15, 2025

Building an End-to-End Lending Data Analytics Platform: My DE Zoomcamp 2025 Final Project

Introduction As part of my DE Zoomcamp 2025 journey, I developed a comprehensive data engineering solution for the lending industry. The project addresses a critical business need: helping financial institutions make data-driven lending decisions by ...

Building an End-to-End Lending Data Analytics Platform: My DE Zoomcamp 2025 Final Project

Discuss·1 read

Nextwebb

nextwebb.hashnode.dev·Jan 10, 2025

Jan 10, 2025

Data Engineering Foundations: A Hands-On Guide

Data Engineering Foundations: A Hands-On Guide Hey there! If you’ve been curious about data engineering, this guide will help you understand the basics and walk you through practical examples. Whether it’s setting up storage, processing data, automat...

Data Engineering Foundations: A Hands-On Guide

Discuss·76 reads

Shrinivas Vishnupurikar

Shrinivas Vishnupurikar

data-engineering-with-shrinivasv73.hashnode.dev·Dec 11, 2024

Dec 11, 2024

The Spark Eco-System with not very often elaborated Components

The article’s title is justified because when I was studying the Spark’s Ecosystem, I had a lot questions with respect to which components are siblings to each other or which are nested one another, and also had “Does such a component actually exists...

The Spark Eco-System with not very often elaborated Components

Discuss·15 reads

Jitender Kaushik

Jitender Kaushik

jitenderkaushik.com·Nov 08, 2024

Nov 08, 2024

Exploring Microsoft Fabric: Notebooks vs. Spark Jobs and How Java Fits In

Microsoft Fabric offers a versatile platform for data processing, blending interactive notebooks with powerful Spark jobs. While both tools serve different purposes, understanding their distinctions can optimize your workflows, especially with Java c...

Exploring Microsoft Fabric: Notebooks vs. Spark Jobs and How Java Fits In

Discuss·37 reads

microsft fabric notebook

Alex Merced

alexmerced.hashnode.dev·Oct 19, 2024

Oct 19, 2024

Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake

Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse Engineering Video Playlist Maintaining a persistent Airflow deployment can often add significant overhead to data engineering teams, especially when ...

Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake

Discuss·2 reads

Sharath Kumar Thungathurthi

Sharath Kumar Thungathurthi

sharaththungathurthi.hashnode.dev·Oct 19, 2024

Oct 19, 2024

Unlock PySpark’s Power: Techniques for Parallelizing

Conceptual Questions What is parallelize in PySpark? parallelize is a method in PySpark used to convert a Python collection (like a list or a tuple) into an RDD (Resilient Distributed Dataset). This allows you to perform parallel processing on the ...

Discuss·1 like·29 reads

Gyuhang Shim

plto001.hashnode.dev·Oct 14, 2024

Oct 14, 2024

Lambda vs Kappa Architecture in Data Pipeline (Korean)

Lambda Architecture 구성 요소 Batch Layer 정기적으로 대량의 Historical Data 를 처리합니다. (예: Daily 또는 Hourly) 이를 통해 높은 정확도와 데이터 Completeness (완전성) 을 보장하며, 복잡한 Data 변환을 처리합니다. Speed Layer 실시간 Data 를 처리하여 low-latency 시간의 결과를 제공합니다. Batch Layer 가 동일한 Data 를 처리할...

Discuss·14 reads

kappa architecture

Gyuhang Shim

plto001.hashnode.dev·Oct 10, 2024

Oct 10, 2024

Trino (TSF) Installation and Configuration

Understanding the Clear Reasons for Using Trino Since the installation and configuration methods of Trino vary depending on how you use it, it's essential to first clearly understand the reasons for using it. Determine if you need to fetch data from...

Trino (TSF) Installation and Configuration

Discuss·61 reads

Gyuhang Shim

plto001.hashnode.dev·Sep 27, 2024

Sep 27, 2024

Trino (TSF) Installation and Configuration (Korean)

Trino 사용 이유를 명확하게 파악하기 Trino 는 사용하는 방법에 따라서 설치, 구성 방법이 다르기 때문에 사용하는 이유를 명확하게 파악하는 것을 먼저 수행되어야 합니다. 다양한 Data Source (MySQL, PostgreSQL, HDFS, Amazon S3, Cassandra, Kafka 등)로부터 데이터를 가져와 하나의 Query 로 통합하여 분석할 필요가 있는 지 파악 Business Intelligence Platform...

Trino (TSF) Installation and Configuration (Korean)

Discuss·52 reads