San Francisco, California – June 26, 2025 – RisingWave, the event stream data processing and management platform, today announced a new integration with Databricks, the Data and AI company. Building on its role as a key launch partner for Databricks’...
👋 Hi, I'm Rushank Patil I'm a Data Analyst & Engineer with a passion for building scalable, insightful analytics solutions that drive business value. I specialize in working across the full data lifecycle—from engineering robust pipelines to deliver...
Databricks has become a foundational platform for modern data engineering and AI. And with Unity Catalog, it adds a much-needed layer of data governance, security, and manageability. In this article, we’ll walk you through everything you need to know...
Introduction Delta Lake, a powerful storage layer built on top of Apache Spark, brings ACID transactions and scalable metadata handling to big data. But like any persistent storage system, performance can degrade over time due to file fragmentation a...
In a recent Big Data Analytics lab assignment, we were tasked with ingesting (consuming) data into Databricks from an external system. To tackle this challenge, I explored setting up data producers on a cloud server—leveraging the fact that these ser...
Imagine you walk into a messy library where books are scattered, nobody knows where anything is, and five people are trying to read the same book at once.That’s what traditional big data felt like — until Databricks walked in with a flashlight, a lab...
Every engineer has two journeys:One is the work we ship.The other is the growth we quietly build behind the scenes. The Engineer’s Logs is my attempt to capture that second journey. These notes and logs are rough and candid, meant for me to revisit ...
Overview In recent months, there's been a surge in frameworks promoting "agentic" architectures for solving information retrieval and decision-making tasks. These include MCP, A2A, AutoGen, LangGraph, and OpenAI’s agents-python-sdk. While these model...
If you’ve ever worked with a data lake, you know how quickly it can turn into a “data swamp.”Messy, Unstructured, Hard to trust, Harder to scale. That’s where Medallion Architecture comes in — and when combined with Databricks, it becomes an absolu...
Introduction Modern data platforms demand real-time capabilities — from ingestion to transformation to serving data for BI and ML use cases. Azure Databricks offers three powerful tools to help with this: Auto Loader: For scalable, file-based ingest...
