How to reduce infrastructure costs by up to 80% with Azure Databricks and Delta Lake

Figure 1. Modern data architecture with Delta Lake and Azure Databricks
Delta Lake and Azure Databricks enable the modern data architecture to simplify and accelerate data and AI solutions at any scale. The implementation of the modern data architecture allowed Relogix to scale back costs on wasted compute resources by 80% while further empowering their data team.
The medallion architecture (as noted in the following diagram) allows for flexible access and extendable data processing. The Bronze tables are for data ingestion and enable quick access (without the need for data modeling) to a single source of truth for incoming IoT and transactional events. As data flows to Silver tables, it becomes more refined and optimized for business intelligence and data science use cases through data transformations and feature engineering. The Bronze and Silver tables also act as Operational Data Store (ODS) style tables allowing for agile modifications and reproducibility of downstream tables. Deeper analysis is done on Gold tables where analysts are empowered to use their method of choice (PySpark, Koalas, SQL, BI, and Excel all enable business analytics at Relogix) to derive new insights and formulate queries.
Figure 2. Architecting your Delta Lake with the medallion data quality data flow
Read more about how Relogix reduced infrastructure costs with Delta Lake and Azure Databricks then register to see how you can turbocharge your business with machine learning using Azure Databricks.
Subscribe to my newsletter
Read articles from Ian Santillan directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by

Ian Santillan
Ian Santillan
Data Architect ACE - Analytics | Leading Data Consultant for North America 2022 | Global Power Platform Bootcamp 2023 Speaker | Toronto CDAO Inner Circle 2023