들어가며 배치성 작업으로 보이는 업무가 있을 때, 이것이 정말 배치 처리로만 해결 가능한 문제인지 고민하게 됩니다. 애플리케이션에서 직접 처리하거나 이벤트 스트리밍 같은 방법으로 문제를 해결한다면 실시간성이 확보되고 자원도 더 효율적으로 사용할 수 있기 때문입니다. 하지만 이번에 소개할 사례는 다양한 대안을 검토했음에도 결국 배치 처리를 선택할 수밖에 없었던 케이스입니다. 1. 문제 상황 1.1 기존 구조 및 접근 방식 저는 이커머스 플랫폼을 ...
No matter the complexity of Dynamics 365 FO implementation, developing custom batch jobs is always necessary to accommodate tailored business processes. As a Dynamics 365 Finance & Operations (D365 FO) developer, working with custom batch jobs is a r...
I have been working on the batch processing aspects lately, and I found many topics on this idea. One of the ways was to use Luigi, a Spotify maintained project. The main problem is that there aren't many good blogs about how to implement batch proce...
Hey there, developer! You might need some tools to boost your skills, and here are a few to check out: (https://jsr.io/@panth977/tools) Rate limit function invocation Sometimes you just want to make sure you don't go over the max number of calls to a...
High-Performance Computing (HPC) in the cloud just got smarter, simpler, and more scalable with Azure Batch. Let’s learn how it works, its unique features, and how to leverage it for your next big project. What is Azure Batch? Azure Batch is a plat...
Introduction In modern distributed systems, the way applications communicate with each other can significantly impact their performance, reliability, and scalability. While webhooks are often praised for their efficiency in real-time communication, A...
In Salesforce development, efficiently managing large volumes of data is a common challenge. Apex Batch is an essential feature that allows developers to process records in the background, enhancing performance and user experience. In this guide, we’...
Introduction to Data Pipeline 데이터 파이프라인은 원시 데이터 (Raw Data) 를 수집하여 유용한 정보로 변환하는 일련의 프로세스를 말합니다. 데이터의 수집, 저장, 처리, 분석, 그리고 시각화까지의 전 과정을 포함하며, Big Data 환경에서는 대용량 데이터의 효율적인 처리를 위해 필수적인 요소입니다. 데이터 파이프라인의 주요 구성 요소 데이터 수집 (Data Ingestion) 데이터 저장 (Data ...
Introduction Recommender systems have become essential to e-commerce, helping customers find products they’re interested in, ultimately improving the customer experience and driving sales. In this blog, we’ll guide you through implementing your recom...
To guarantee the scalability and availability of our own service, it's a must to protect downstream dependencies from being flooded with requests within a very shot period of time. One popular technique is to call a Batch API if one is available down...
