Live Streaming IPL to 30 Million+ Devices: JioHotstar Engineering Debunked

PS: I am looking for a job for the role of “Devops Engineer“ or “Fullstack Developer“, If you know someone who could refer or help, A referral would be appreciated ❤️

So Let’s first understand how video engineering works in the backend

Let’s assume you are in a 5g zone and travelling in your car and watching IPL match on your mobile device.

Suddenly you arrive in a 3g zone where the internet speed is not feasible and very less, So we can’t drop the quality suddenly so what JioHotstar does is it maintains a HLS Stream of 10+ seconds where even if the User’s internet connection drops suddenly ‘out of the blue’ we still need to adjust and give the minimum bit rate streaming possible so that the Viewing Experience is not compromised.

It also has to do a lot about Network Latency, where it maintains a Buffer for the video to be processed with Ads, JioHotstar Logos, Banner etc. Let me show you with a diagram.

The Job of Cameraman is just to record videos, clips and what’s going on in the match, He then send this RTMP Steam to the server which mainly consists of video files. Now JioHotstar as a Business POV also needs to serve Advertisements, Banners, Future Match Promos in this feed to keep the user entertained and to also fill its pockets from advertisers money.

So the people / server takes all the video recordings from the camera, Processes it makes it HLS Streamable (480p, 720p, 1080p, 4k) and then also maintains a buffer where all the content related to Ads, Match Promos are added and then this processed Stream is then forwarded to all the devices.

So technically, we are always served 15+ seconds late from the actual match that’s happening at the venue.

Geo location based streaming.

Have you ever wondered if the match is been played in Australia or Dubai, How are we being served the same match with 0 delay just 10-15 seconds behind of the actual match ? Here’s where CDN’s come into the picture.

Here, packets are sent to CDN Servers on specific Geo-locations where users from Mumbai are not served the match directly from Dubai, but from a near by CDN Network which is present in Mumbai, THis CDN is obviously owned by JioHotstar. Other geo locations CDN works in the same way where the packets of the match are sent to CDN and then the CDN serves the packets to the consumers / viewers.

JioHotstar and other Streaming Platforms also support CDN’s (Content Delivery Networks) where if the match is being recorded in Dubai and the fans or match viewers are present in Mumbai or California or Australia then Delivering Streams becomes an issue. Here’s where CDN come into play.

How Livestream is processed from Cameraman to your mobile device ?

When the camera man is recording the match through his expensive camera his RTMP Stream (Real Time Message Protocol) Stream is being sent through a Fibre Optic Cable to the JioHotstar Server (Cloud Server). This Server gets all the streams and dumps them into a Database. Now the server needs to handle HLS Streaming, where the quality of match is served based upon the user’s device internet speed and the video is served in those bit rates then.

Server is Responsible for serving higher bit rates to users with higher internet speed and preferrably watchable bitrates to users with lower internet speeds so that the customer / viewer gets atleast something to watch even after low internet speeds

---

This was an overall technical overview from scratch on IPL and how Streaming Platform works.

Let’s see how things happen in a hardcore mode in the backend engineering side of IPL.

Kafka is mainly used for Streaming purposes (also used by JioHotstar) where it has a Consumer and a Producer and serves streaming through bit rates, Kafka is used when we need to serve to a lot of users and the content is on a large scale. If you have read my previous blogs on Kafka you would know it acts like a message broker and sits in the middle allowing applications to publish and subscribe to streams of data, effectively acting as a hub for data ingestion and distribution.

JioHotstar also has a lot of backend architecture that works on Kubernetes Clusters and pods where auto scaling and autohealing is supported, but autoscaling is not used.

There are some moments in the match where the viewership of the match just skyrockets to a different level, be it MS Dhoni coming down to bat at no.7 in the end of the match or Virat Kohli playing against Pakistan or Rohit Sharma hitting everything out of the park. The number of concurrent users just goes on increasing. This is where the Devops Architecture also plays a very significant role where the System (JioHotstar) is capable of handling traffic which is real time (hard) without crashing even once.

Here are some top concerns made by the JioHotstar Team on a regular IPL Day Match:

1. Careful planning and back of the envelop calculation for all system components that are likely to get impacted by traffic surges. Like DBs, Kafka, CDN etc.

2. Run time auto scaling of DB capacity doesn't necessarily work for use cases like live streaming. You need to scale up before traffic surges.

3. Failures happen in the System, Just don’t make it visible to the customer on the app

4. Databases are the hardest part that needs to be scaled.

5. 75% of India’s internet bandwidth available is used by JioHotstar itself from the Cloud Provider

6. Calculate the Permutations and Combinations prematch to ensure the working of the match.

What happens after the match has been completed or finished:

1. The JioHotstar Team does a deep analysis on the CPU usage for the Databases.

2. The team scales down the system based upon the number of consumeres still present on the platform. For example: If there were 30 million viewers on the app then, after the match the concurrent viewers drops down and the system needs to scales down, But the team scales down in a ladder pattern where in some intervals 2 million , or 5 million is being scaled down so that other viewers who are using the app do not face any issue.

3. Mobile Device caches stores a lot of images / Video posters of the app so that the recursive things are not fetched from the main server everytime.

Thanks for reading this dev blog ❤️, More soon.

Here are my socials:

LinkedIn

Twitter