Introduction and Need of Rollup

In response to the challenges of managing extensive time-series data, PhonePe’s Metrics platform has introduced “Rollup”

This feature aggregates time series data, simplifying historical data analysis by reducing data resolution. When dealing with a massive volume of data points, Rollup’s value becomes evident. For instance, querying a year’s worth of data recorded every 60 seconds results in an overwhelming number of data points. Rollup addresses this by allowing users to work with lower-resolution data, such as hourly data.

Benefits of Metrics Rollup

The metrics platform handles a vast amount of data, i.e., ~200 Billion metrics, and ~1.6 TB data every day. Rollup enhances query performance(~5 times faster than raw data query) and reduces storage requirements(25GB of rolled-up data corresponds to ~1.6 TB). The technical implementation involves Spark jobs that aggregate weekly data, storing it efficiently and enabling faster data retrieval. The long-term plan is to maintain rolled-up data indefinitely and retain raw data for the last six months to manage space and reduce query time.

Conclusion

Metrics Rollup is a significant advancement in managing time-series data efficiently, providing faster queries and enhanced performance while addressing storage challenges. It ensures that historical data remains accessible for analysis and decision-making, reinforcing PhonePe’s commitment to delivering efficient financial services.