Play Local MP4 Video Files in Your Browser: Pure JavaScript Video Play

The source code for this article: https://github.com/epanikas/pure-javascript-video-player-for-local-video-files

The demo page: pure-javascript-video-player-for-lo.vercel.app

In this article, we will be discussing a problem that I have encountered during the development of one of my projects:

given a video file, located on a local device file system
play it in the browser, without uploading it to any remote server

There is no lack of JavaScript video players out there: ShakaPlayer, video.js, hls.js, and dash.js, to name a few.

However, all these solutions would require a remote server URL to provide a video file.

In our case, however, the video file is located locally and should also be consumed locally.

I did my preliminary research, and it turns out that the way to go is to use the so-called Media Source Extension - a relatively new (and sometimes considered experimental) standard that allows the browser <video /> tag to play video files.

Set up the project

In this tutorial, we will be using NextJS + React as our JS platform, but the presented steps can easily be adapted to another JS platform of your choice.

We will be using a standard HTML file input element:

<input type="file" onChange={onFileChange} multiple={false} className={"my-3"}/>

Let’s add a video player tag that would eventually play our video file

<video id={"my-video"} controls autoPlay/>

The rest of the setup steps are skipped, as they are quite standard and straightforward. The full source code for this article can be found at:

https://github.com/epanikas/pure-javascript-video-player-for-local-video-file

The sample page is available here: https://pure-javascript-video-player-for-lo.vercel.app

To run the project, use:

npm run dev

Here is the result of the initial setup of the application:

Using Media Source Extension to play a video file

According to the documentation, the Media Source Extension is to be used as follows:

first, create a MediaSource object using the statement

      const myMediaSource = new MediaSource();

add this media source as a src parameter to the video player tag

      const url: string = URL.createObjectURL(myMediaSource);
      videoTag.src = url;

once the MediaSource reaches the open state, add a SourceBuffer to it

      myMediaSource.addEventListener('sourceopen', sourceOpen);
      ...
      function sourceOpen() {
          const videoMime = ... // video mime should be specified here
          videoSourceBuffer = myMediaSource.addSourceBuffer(videoMime);
          videoSourceBuffer.mode = "segments"
      }

finally, once the SourceBuffer is added, we can start pushing chunks of video stream into it, allowing the video to play, as follows:
- ```
      const buffer: ArrayBuffer = ... // video buffer goes here
      videoSourceBuffer.appendBuffer(buffer)
```

Unfortunately, that may look simple at first, but in reality, as we all know, the devil is in the details. And as far as details are concerned, we have plenty of those here…

First of all, when creating the SourceBuffer we need to know the MIME type of the video stream that will be played through that buffer. That is the first problem: how to obtain the media source mime type from the file we are about to play.

We should note that the mime type should contain the audio and video codecs, the MP4 file is encoded with.

Second, we also need to know what should be contained in the binary stream buffer, represented as ArrayBuffer, so that the SourceBuffer will be able to play the video.

MP4Box to the rescue

It turns out that there is a JavaScript port of a popular video files manipulation library MP4Box - MP4Box.js

Using this library, it would be possible to read the mime information (and not only) that is encoded in the MP4 container file.

To do that, we need to create an instance of a so-called MP4Box.ISOFile - the object that would represent the video file, and would enable some read operations with this file.

A handy tool that could give some insight into the internal structure of the MP4 file can be found here: https://gpac.github.io/mp4box.js/test/filereader.html

Here is a sample output of this page:

Plenty of information is displayed here, and in particular, the MIME type we are looking for.

What is important for us is also the fact that the video file has been entirely processed locally, without uploading the file to any third-party servers or using external tools.

Before proceeding with video file processing, we need to understand one important thing: the internal structure of an MP4 file.

Let’s have a look at what MP4Box online inspection tool has to say about the structure of the loaded file:

We can see that in our case the video file consists of three blocks:

ftyp (FileTypeBox) - the box containing type information of the file
mdat (MediaDataBox) - the actual video content
moov (MovieBox) - the header section, which contains all the important metadata of the file, such as audio/video codecs, information about available tracks in the file, and so on

And the important thing here is that the locations of those boxes are not predefined or standardized, and potentially can be found anywhere in the video file.

We should note that in the case of the file, given as an example here, the moov box is found at the very end of the file, after all the other boxes.

This is quite naturally the case for the majority of video files, recorded on a smartphone. Indeed, the recording is started, and until it is finished, we cannot know the information to write in the moov box, such as video duration. When the user clicks on the end of recording, the information is written to the file, and the file is completed.

Since the video files can be quite big - several hundred megabytes, or more - we cannot count on the fact that the whole file could be loaded in memory before processing it. Instead, we should incline towards stream-like processing, when chunks of a file are loaded into memory, processed, and then probably discarded.

Ok, enough theory, let’s get our hands dirty and do some code using MP4Box

Reading the video file MIME information

First of all, we need to create an instance of MP4Box.ISOFile which will be the representation of our video file in the code.

const mp4BoxFile: MP4Box.ISOFile<unknown, unknown> = MP4Box.createFile(/*keepMdatData*/true);

Then we need to add important event handlers, since the MP4Box.ISOFile is mostly based on an event-driven model.

    mp4BoxFile.onError = (m: string, msg: string) => {
        console.error(msg);
    };

    mp4BoxFile.onReady = (info: MP4Box.Movie) => {
        console.error(info);
        setMp4BoxFileInfo(info)
    };

The onReady event handler is called once the the ISOFile has received its moov box - that is, the meta information about the video file is ready.

Note that we cannot know in advance where the MOOV box will be located in the file, hence the event-driven model, where an event handler is invoked upon reception and successful parsing of MOOV box.

Once the ISOFile is created and the event handlers are configured, we can start feeding our binary information to the file.

Feeding chunks of data to the MP4Box.ISOFile

The way to feed the chunks of data to ISOFile deserves special attention. Unfortunately, this part is very poorly documented (and not only this part, by the way), so I had to do some searching and experimentation to find that out.

The chunks of binary data, read from the video file, are to be fed to ISOFile using the method ISOFile.appendBuffer(ab: MP4BoxBuffer, last?: boolean): number;

Its first argument is of type MP4BoxBuffer. This is a standard binary array of type ArrayBuffer, with an additional attribute - fileStart.

declare class MP4BoxBuffer extends ArrayBuffer {
    fileStart: number;
    ...
    static fromArrayBuffer(buffer: ArrayBuffer, fileStart: number): MP4BoxBuffer;
}

This attribute is very important, as it indicates to MP4Box.ISOFile the location of the buffer in the original file. In particular, this attribute allows feeding to ISOFile the chunks that are not necessarily located sequentially in the original video file.

Here is an example: imagine we are reading the video file using the standard stream() API, which is part of the File API, like so:

selectedFile.stream().pipeTo(... consume the file chunk by chunk here ...);

The file will be read sequentially, in chunks depending on the buffer size. Suppose the file is read by chunks of 64Kb:

0 - 65536
65536 - 131072
131072 - 196608
… and so on

Hence, in this scenario, we should be calling ISOFile.appendBuffer as follows:

ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 0))
ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 65536))
ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 131072))

If, for some reason, the chunks 2 and 3 were inverted

0 - 65536
131072 - 196608
65536 - 131072

The calls to appendBuffer should have been inverted as well:

ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 0))
ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 131072))
ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, 65536))

Now, what is really confusing here is that the ISOFile.appendBuffer itself also returns a number. And this number is supposed to indicate the next chunk to be fed to ISOFile.

Recall the signature of the appendBuffer method: ISOFile.appendBuffer(ab: MP4BoxBuffer, last?: boolean): number;

As the documentation indicates, the number returned by the appendBuffer method is an indication of the ISOFile expectation. That is, it tries to tell us what it expects the next file chunk location to be.

And indeed, if the MOOV box is not located at the beginning of the file, this number is useful, as it would allow us to directly load the required chunk, instead of making the ISOFile wait for the required chunk throughout the whole file.

Let’s activate the MP4Box logging mode and see how it behaves.

MP4Box.Log.setLogLevel(MP4Box.Log.info)

And here is the result:

[0:00:06.954] [ISOFile] Done processing buffer (fileStart: 0) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.957] [ISOFile] Done processing buffer (fileStart: 65536) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.959] [ISOFile] Done processing buffer (fileStart: 983040) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.959] [ISOFile] Done processing buffer (fileStart: 2031616) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.963] [ISOFile] Done processing buffer (fileStart: 2097152) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.966] [ISOFile] Done processing buffer (fileStart: 4194304) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.968] [ISOFile] Done processing buffer (fileStart: 6291456) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.971] [ISOFile] Done processing buffer (fileStart: 8388608) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.974] [ISOFile] Done processing buffer (fileStart: 10485760) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.976] [ISOFile] Done processing buffer (fileStart: 12189696) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.978] [ISOFile] Done processing buffer (fileStart: 12582912) - next buffer to fetch should have a fileStart position of 16376663
[0:00:06.979] [ISOFile] Done processing buffer (fileStart: 14483456) - next buffer to fetch should have a fileStart position of 16376663
[0:00:07.001] [ISOFile] Done processing buffer (fileStart: 14680064) - next buffer to fetch should have a fileStart position of 16382213

We can see that the ISOFile has been waiting patiently for the block it has been expecting so eagerly - the block starting at position 16376663.

And if we examine the video file, we’ll see why:

The MOOV box, containing the essential information about the video file, is found at the end of the file, at the location 16376663. And MP4Box library knows very well that until that box is found, nothing can be done with the file. Hence, right after the first chunk has been loaded, it knew already that it would be able to do nothing until the MOOV box chunk is located and parsed. That is why it has been indicating right after the first chunk the expected position of 16376663.

However, whatever the ISOFile expectations are, we should clearly understand that when the appendBuffer is called, it should receive the actual chunk location in the file, not the expected one.

We can also conclude that reading a video file in a streaming way, sequentially, is probably not the most optimal one. Instead, we could imagine a random access approach, when the reading process is entirely guided by the expected chunk start, returned by the appendBuffer call.

Like so:

let nextChunkStart = 0
const bufferSize = 65536 / 64Kb
while (the file is not entirely read) {
- chunk = readChunk(nextChunkStart, nextChunkStart + bufferSize)
- nextChunkStart = ISOFile.appendBuffer(MP4BoxBuffer.fromArrayBuffer(chunk, nextChunkStart))
}
ISOFile.flush()

where the function readChunk(start, end) is a hypothetical function capable of reading the file in a random access mannder, given the start and end of the required segment

This way, we can ensure the most efficient way of reading a video file. This approach can (and should!) also be used for reading the video file from a remote location, if the server supports the byte-range requests.

One thing to note here is the call to MP4Box.ISOFile.flush() method once the file is entirely read. This call indicates to MP4Box the fact that no more data will be received, and it should continue processing the remaining chunks, if any.

Here is the code so far:

"use client"

import {JSX} from "react"
import {useState} from "react";
import {ChangeEvent} from "react";
import {MP4BoxBuffer} from "mp4box";
import * as MP4Box from "mp4box"

MP4Box.Log.setLogLevel(MP4Box.Log.info)

export default function TestPlayVideoSegmented(): JSX.Element {

    const [selectedFile, setSelectedFile] = useState<File | null>(null);
    const onFileChange = (event: ChangeEvent<HTMLInputElement>) => {
        setSelectedFile(event.target.files![0]);
    };

    const [mp4boxLoadingProgress, setMp4boxLoadingProgress] = useState<number>(0);
    const [mp4boxBytesRead, setMp4boxBytesRead] = useState<number>(0);
    const [mp4BoxFileInfo, setMp4BoxFileInfo] = useState<MP4Box.Movie | null>(null);

    const mp4BoxFile: MP4Box.ISOFile = createMp4BoxFile(setMp4BoxFileInfo);

    const bytesToProgress = (bytesRead: number) => {
        setMp4boxBytesRead(bytesRead);
        setMp4boxLoadingProgress((bytesRead / selectedFile.size) * 100)
    };

    return (
        <div className={"p-5 flex flex-col items-center justify-center"}>
            <input type="file" onChange={onFileChange} multiple={false} className={"my-3"}/>

            {selectedFile && displayFileInfo(selectedFile)}

            {selectedFile &&
                <div>
                    <button type={"button"} onClick={() => onReadFile(selectedFile, mp4BoxFile, bytesToProgress)}
                            className="mt-10 text-white bg-green-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800"
                    >Read file {selectedFile.name}
                    </button>

                    <div>MP4Box bytes read: {mp4boxBytesRead}</div>
                    <div className="w-full bg-gray-200 rounded-full h-2.5 dark:bg-gray-700">
                        <div className="bg-blue-400 h-2.5 rounded-full"
                             style={{"width": mp4boxLoadingProgress + "%"}}></div>
                    </div>

                    {mp4BoxFileInfo && mp4BoxFileInfo && displayMp4BoxFileInfo(mp4BoxFileInfo)}
                </div>
            }

            <video id={"my-video"} className={"border-4 border-red-700 mx-auto my-3"} controls autoPlay/>
        </div>
    )

}

function displayFileInfo(selectedFile: File): JSX.Element {
    ...
}

function displayMp4BoxFileInfo(info: MP4Box.Movie) {
    ...
}

function onReadFile(selectedFile: File,
                    mp4BoxFile: MP4Box.ISOFile<unknown, unknown>,
                    bytesToProgress: (p: number) => void): void {

    selectedFile.stream().pipeTo(readStreamIntoMp4IsoFile(mp4BoxFile, bytesToProgress));

}

function createMp4BoxFile(setMp4BoxFileInfo: (info: MP4Box.Movie) => void): MP4Box.ISOFile<unknown, unknown> {
    const mp4BoxFile: MP4Box.ISOFile<unknown, unknown> = MP4Box.createFile(true);

    mp4BoxFile.onError = (m: string, msg: string) => {
        console.error(msg);
    };

    mp4BoxFile.onReady = (info: MP4Box.Movie) => {
        console.info(info);
        setMp4BoxFileInfo(info)
    };

    return mp4BoxFile;
}


function readStreamIntoMp4IsoFile(mp4boxFile: MP4Box.ISOFile<unknown, unknown>,
                                  setMp4boxLodingProgress: (p: number) => void): WritableStream<Uint8Array<ArrayBufferLike>> {

    var nextBufferStart = 0;

    function typedArrayToBuffer(array: Uint8Array): ArrayBuffer {
        return array.buffer.slice(array.byteOffset, array.byteLength + array.byteOffset) as ArrayBuffer
    }


    return new WritableStream<Uint8Array<ArrayBufferLike>>({

            start: async (controller: WritableStreamDefaultController): Promise<void> => {
                console.log("readStreamIntoMp4IsoFile.start")
            },
            write: async (chunk: Uint8Array<ArrayBufferLike>, controller: WritableStreamDefaultController): Promise<void> => {
                const ab: MP4BoxBuffer = MP4BoxBuffer.fromArrayBuffer(typedArrayToBuffer(chunk), nextBufferStart)
                nextBufferStart += chunk.length
                mp4boxFile.appendBuffer(ab, false);
                setMp4boxLodingProgress(nextBufferStart)
            },
            close: async (): Promise<void> =>  {
                console.log("readStreamIntoMp4IsoFile.close")
                mp4boxFile.flush();
                setMp4boxLodingProgress(nextBufferStart)
            },
            abort: async (reason: any): Promise<void> => {
                console.log("readStreamIntoMp4IsoFile.abort" + reason)
            },
        },
        {
            highWaterMark: 3,
            size: () => 1,
        },
    );
}

The presented code uses the stream approach to file reading, not the random-access approach depicted above.

And here is the result:

Playing the video using the segmentation process of MP4Box

At this point, we have enough scaffolding to try to finally play our video file. We know the file’s MIME information, including the track and codecs info. And we are capable of feeding our file to MP4Box.ISOFile object.

The Media Source Extension (MSE) is only capable of playing a segmented stream of video file. That is, it cannot accept the whole file; it can only accept segments of it.

The tools, such as ffmpeg and MP4Box, provide command-line tools to segment a file. Then this segmented file can be read into SourceBuffer, segment by segment.

However, in our case, this approach is not suitable, as the idea is to play any MP4 video file without additional preprocessing.

Fortunately, the Javascript version of MP4Box tool - the one we’ve been using before, the MP4Box.js - is capable of segmenting a stream of a video file.

To launch the segmentation process, we need:

add onSegment event handler to the ISOFile instance
start feeding the chunks of file to ISOFile using the process described above - the ISOFile.appendBuffer(…) method
finally call MP4Box.ISOFile.flush() to make sure the remaining chunks are processed
once the onReady has been called (that is the MOOV box has been read)
- specify the segmentation options using ISOFile.setSegmentOptions method
- call the method ISOFile.initializeSegmentation()
- add the buffer, resulting from the initialization, to the SourceBuffer
- call ISOFile.start()

Each time a chunk is created, we’ll need to append it to the MSE SourceBuffer.

Let’s have a look at the resulting code, and then we’ll have a closer look at each part of it:

"use client"

import {JSX} from "react"
import {useState} from "react";
import {ChangeEvent} from "react";
import {MP4BoxBuffer} from "mp4box";
import * as MP4Box from "mp4box"


type InitSegsType = {
    tracks: {
        id: number;
        user: unknown;
    }[];
    buffer: MP4BoxBuffer;
}

type VideoSegment = {
    id: number;
    user: unknown;
    buffer: ArrayBuffer;
    nextSample: number;
    last: boolean;
}

MP4Box.Log.setLogLevel(MP4Box.Log.info)

const supportedTrackTypes = ['audio', 'video']

export default function TestPlayVideoSegmented(): JSX.Element {

    const [selectedFile, setSelectedFile] = useState<File | null>(null);
    const onFileChange = (event: ChangeEvent<HTMLInputElement>) => {
        setSelectedFile(event.target.files![0]);
    };

    const [mp4boxLoadingProgress, setMp4boxLoadingProgress] = useState<number>(0);
    const [mp4boxBytesRead, setMp4boxBytesRead] = useState<number>(0);
    const [mp4BoxFileInfo, setMp4BoxFileInfo] = useState<MP4Box.Movie | null>(null);

    const bytesToProgress = (bytesRead: number) => {
        setMp4boxBytesRead(bytesRead);
        setMp4boxLoadingProgress((bytesRead / selectedFile.size) * 100)
    };

    let myMediaSource: MediaSource;
    let videoSourceBuffer: SourceBuffer;

    let isLastSegment = false;
    const segments: VideoSegment[] =[]

    function onFileInfoReadyCb(info: MP4Box.Movie, initSegs: InitSegsType): void {
        setMp4BoxFileInfo(info)
        myMediaSource = new MediaSource();
        myMediaSource.addEventListener("sourceopen", () => {
            videoSourceBuffer = myMediaSource.addSourceBuffer(info.mime)
            videoSourceBuffer.addEventListener("updateend", () => {
                const segment: VideoSegment | undefined = segments.shift()
                if (segment) {
                    isLastSegment = segment.last;
                    videoSourceBuffer.appendBuffer(segment.buffer)
                } else {
                    if (isLastSegment) {
                        if (myMediaSource.readyState === 'open') {
                            myMediaSource.endOfStream()
                        }
                    }
                }
            })
            videoSourceBuffer.appendBuffer(initSegs.buffer)

        })

        const videoTag: HTMLVideoElement = document.getElementById("my-video") as HTMLVideoElement;
        videoTag.onerror = (e) => {
            console.error("video error", e, videoTag?.error)
        }
        videoTag.src = URL.createObjectURL(myMediaSource);

    }

    function onSegmentReadyCb(id: number, user: unknown, buffer: ArrayBuffer, nextSample: number, last: boolean): void {
        segments.push({id, user, buffer, nextSample, last})
    }

    return (
        <div className={"p-5 flex flex-col items-center justify-center"}>
            <input type="file" onChange={onFileChange} multiple={false} className={"my-3"}/>

            {selectedFile && displayFileInfo(selectedFile)}

            {selectedFile &&
                <div>
                    <button type={"button"} onClick={() => onReadFile(selectedFile, onFileInfoReadyCb, onSegmentReadyCb, bytesToProgress)}
                            className="mt-10 text-white bg-green-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800"
                    >Read file {selectedFile.name}
                    </button>

                    <div>MP4Box bytes read: {mp4boxBytesRead}</div>
                    <div className="w-full bg-gray-200 rounded-full h-2.5 dark:bg-gray-700">
                        <div className="bg-blue-400 h-2.5 rounded-full"
                             style={{"width": mp4boxLoadingProgress + "%"}}></div>
                    </div>

                    {mp4BoxFileInfo && mp4BoxFileInfo && displayMp4BoxFileInfo(mp4BoxFileInfo)}
                </div>
            }

            <video id={"my-video"} className={"border-4 border-red-700 mx-auto my-3 max-w-96"} controls autoPlay/>
        </div>
    )

}

function displayFileInfo(selectedFile: File): JSX.Element {
    ... // didn't change
}

function displayMp4BoxFileInfo(info: MP4Box.Movie) {
    ... // didn't change
}

function onReadFile(selectedFile: File,
                    onFileInfoReadyCb: (info: MP4Box.Movie, initSegs: InitSegsType) => void,
                    onSegmentReadyCb: (id: number, user: unknown, buffer: ArrayBuffer, nextSample: number, last: boolean) => void,
                    bytesToProgress: (p: number) => void): void {

    const mp4BoxFile: MP4Box.ISOFile<unknown, unknown> = createMp4BoxFile(onFileInfoReadyCb, onSegmentReadyCb);

    selectedFile.stream().pipeTo(readStreamIntoMp4IsoFile(mp4BoxFile, bytesToProgress));

}

function createMp4BoxFile(onFileInfoReadyCb: (info: MP4Box.Movie, initSegs: InitSegsType) => void,
                          onSegmentReadyCb: (id: number, user: unknown, buffer: ArrayBuffer, nextSample: number, last: boolean) => void): MP4Box.ISOFile<unknown, unknown> {
    const mp4BoxFile: MP4Box.ISOFile<unknown, unknown> = MP4Box.createFile(true);

    mp4BoxFile.onError = (m: string, msg: string) => {
        console.error(msg);
    };

    mp4BoxFile.onReady = (info: MP4Box.Movie) => {
        console.info(info);

        var options = { nbSamples: 1000/*, sizePerSegment: 1048576  /!*1Mb*!/*/};
        for (let i = 0; i < info.tracks.length; ++i) {
            if (supportedTrackTypes.filter(t => t == info.tracks[i].type).length > 0) {
                console.log("adding segmentation option for track ", info.tracks[i].id, info.tracks[i].type)
                mp4BoxFile.setSegmentOptions(info.tracks[i].id, info.tracks[i].type, options);
            }
        }
        var initSegs: InitSegsType = mp4BoxFile.initializeSegmentation();

        onFileInfoReadyCb(info, initSegs)

        mp4BoxFile.start();
    };

    mp4BoxFile.onSegment = (id: number, user: unknown, buffer: ArrayBuffer, nextSample: number, last: boolean) => {
        console.log("segment received", "id", id, "user", user, "buffer", buffer.byteLength, "nextSample", nextSample, last);
        onSegmentReadyCb(id, user, buffer, nextSample, last)
    }

    return mp4BoxFile;
}


function readStreamIntoMp4IsoFile(mp4boxFile: MP4Box.ISOFile<unknown, unknown>,
                                  setMp4boxLodingProgress: (p: number) => void): WritableStream<Uint8Array<ArrayBufferLike>> {
    ... // didn't change
}

As one can see, the presented code sample is organized around two callback function calls:

onFileInfoReadyCb - called when the ISOFile is ready, i.e., the MOOV box has been received and successfully parsed
onSegmentReadyCb - called when the segment has been emitted by the ISOFile segmentation process

Initializing the segmentation process

First of all, let’s have a look at the set of instructions to initialize the segmentation process.

It consists of the following calls:

ISOFile.setSegmentOptions(trackId, trackType, trackSegmentationOptions): this method allows specifying the segmentation option per track. The segmentation options can contain the following attributes:
- nbSamples: number of samples in segments, defaults to 1000
- sizePerSegment: additional option specifying the size of the segment in bytes. If the segment reaches that limit, it is emitted, even if nbSamples has not been reached yet
ISOFile.initializeSegmentation: this method actually starts the segmentation process, and it returns the initial buffer that should be appended to the SourceBuffer.
ISOFile.start(): this method notifies the ISOFile that each time the buffer is appended via ISOFile.appendBuffer method, the buffer should be processed and the available segments should be emitted (via the ISOFile.onSegment call)

Currently, the segmentation process is configured for tracks of type audio or video.

An MP4 file can contain the tracks of other types, such as text (subtitles), but they are not processed in this example.

Creating the MediaSource and the SourceBuffer

The API of MediaSource and SourceBuffer also heavily relies on the event-driven model.

In particular, the events of particular importance for us are the following:

MediaSource sourceopen event
- fires when the MediaSource is open and ready to receive SourceBuffer’s
SourceBuffer updateend event
- fires when the SourceBuffer is done processing the current video fragment, and is ready to receive a new one.

Let’s have a close look at the callback onFileInfoReadyCb, as this is where the important Media Source setup happens.

    function onFileInfoReadyCb(info: MP4Box.Movie, initSegs: InitSegsType): void {
        setMp4BoxFileInfo(info)
        myMediaSource = new MediaSource();
        myMediaSource.addEventListener("sourceopen", () => {
            videoSourceBuffer = myMediaSource.addSourceBuffer(info.mime)
            videoSourceBuffer.addEventListener("updateend", () => {
                ... // we'll see this more in detail later
            })
            videoSourceBuffer.appendBuffer(initSegs.buffer)

        })

        const videoTag: HTMLVideoElement = document.getElementById("my-video") as HTMLVideoElement;
        videoTag.onerror = (e) => {
            console.error("video error", e, videoTag?.error)
        }
        videoTag.src = URL.createObjectURL(myMediaSource);

    }

Except for the SourceBuffer::updateend handling, the code is quite straightforward: we locate the video HTML5 tag and attach a newly created MediaSource to it.

But an important thing to note here is that we append the initialization buffer, received from the call to ISOFile.initializeSegmentation, to the newly created SourceBuffer. This step is very important, as otherwise this initialization buffer would be lost, and the video would fail to play.

Appending segments to the SourceBuffer

Now let’s have a look at how the segments are treated.

The whole process of reading and parsing a video file, and then appending it to the SourceBuffer, can be imagined as two pipelines, working in parallel:

reading chunks of the video file and feeding them to the MP4Box.ISOFile
- This process is limited by the throughput of the file reading API (or network throughput, in case of reading a file from a remote server)
receiving the segments from the segmentation process of MP4Box, and feeding them to the SourceBuffer
- This process is limited by the speed at which SourceBuffer is capable of accepting and processing the buffer chunks

Since these two processes are not directly related, we cannot synchronously feed the chunks to SourceBuffer as we receive them from the segmentation process.

Indeed, from the experience, the file reading process is normally faster than the SourceBuffer appending process. Hence, we need to store somewhere the segments until we are able to feed them to the SourceBuffer.

We do this thanks to the in-memory queue of video segments - the variable segments

const segments: VideoSegment[] =[]

This array of video segments is used as an in-memory queue:

when a new segment is created, it is added to the queue (see )
when the SourceBuffer is done reading the last segment, appended to it, it retrieves from the queue the next one

This way, the process continues until no more segments to read, i.e., when the queue is empty

The segments are added to the queue in the function onSegmentReadyCb

    function onSegmentReadyCb(id: number, user: unknown, buffer: ArrayBuffer, nextSample: number, last: boolean): void {
        segments.push({id, user, buffer, nextSample, last})
    }

And here is the code of the event handler for SourceBuffer updateend event, which dequeues the segments queue and appends the segment to the SourceBuffer

            videoSourceBuffer.addEventListener("updateend", () => {
                const segment: VideoSegment | undefined = segments.shift()
                if (segment) {
                    isLastSegment = segment.last;
                    videoSourceBuffer.appendBuffer(segment.buffer)
                } else {
                    if (isLastSegment) {
                        if (myMediaSource.readyState === 'open') {
                            myMediaSource.endOfStream()
                        }
                    }
                }
            })

We need to note one important thing here: the call to MediaSource.endOfStream(). This is important as this call notifies the MediaSource that all the video segments have been received, and no more chunks will be appended to the SourceBuffer.

If we fail to notify the end of stream to MediaSource, it would result in an inconsistent state, such as video duration not showing, or an infinite loading sign is shown

Fixing the problems

When I’ve been testing the video player we have built so far, against different video files, I have found out that there is at least one file that didn’t work

In particular, the file frag_bunny-from-nick-desaulniers.mp4 (attached to the source code for this article) wouldn’t play, causing an infinite loop in the segmentation process.

Another problem that would appear with this file is the following: the mime type is not supported:

page.tsx:48 Uncaught NotSupportedError: Failed to execute 'addSourceBuffer' on 'MediaSource': 
The type provided ('video/mp4; codecs="mp4a.40.2,avc1.42e01e,rtp ,rtp "; profiles="mp42,avc1,iso5"') 
is unsupported.
    at MediaSource.eval (page.tsx:48:47)

Ok, the mime type, found in the file, indeed looks not very familiar: 'video/mp4; codecs="mp4a.40.2,avc1.42e01e,rtp ,rtp "; profiles="mp42,avc1,iso5"'

Let’s fix this problem by constructing our own mime type that would only contain the codecs for tracks we will actually be playing - only video and audio.

        let codecs: string[] = [];
        for (let i = 0; i < info.tracks.length; ++i) {
            if (supportedTrackTypes.filter(t => t == info.tracks[i].type).length > 0) {
                codecs.push(info.tracks[i].codec)
            }
        }

        const mime = "video/mp4; codecs=\"" + codecs.join(",") + "\"";

The problem of an infinite loop in the segmentation is more complex to solve.

I have ended up submitting a bug report, and suggesting a fix for this problem, and this has been kindly accepted and released in the release MP4Box.js 1.4.6

Fixing an issue with QuotaExceededError

At some point you might see the following error appearing:

QuotaExceededError: 
Failed to execute 'appendBuffer' on 'SourceBuffer': 
The SourceBuffer is full and cannot free space to append additional buffers.

This error indicates that the memory of the SourceBuffer cannot take up more video segments.

Here is a good article describing the problem and the possible solutions:

Exceeding the buffering quota | Blog | Chrome for Developers

However, it might also happen that the QuotaExceededError is thrown right away, even when the SourceBuffer is empty.

This happens apparently due to the chunk being too big to fit in the SourceBuffer.

In this case, try to use the attribute limiting the size of the segment, like so:

mp4BoxFile.setSegmentOptions(trackId, trackType, { nbSamples: 1000, sizePerSegment: 1048576 });

Conclusion

In this article, we have presented an approach to use the Media Source Extension API in modern browsers to load and play video files, located locally on a device.

The presented solution uses an external library, MP4Box.js, for video stream data extraction and fragmentation.

The article demonstrates the following aspects of the video file play process:

reading the file using the File Stream API
parsing and extraction of the MIME type
fragmentation of the video file, so that it would be suitable for MSE SourceBuffer
and finally appending the fragmented chunks of video to the SourceBuffer, for playback