Building a database with IPFS: A Developer's Tale

Setting the Stage

In ancient Greece, there existed a city called "Polis" where people gathered to discuss a wide range of topics, such as politics, governance, and laws.

Unfortunately, they didn't delve into the art of building a database in IPFS.

That's precisely why, as a group of developers who created a dApp bearing the same name, "Polis," we had to navigate through all the challenges, endure a few curses, and occasionally bang our heads on the keyboard.

In this developers tale , I will attempt to explain in detail our developer journey, focusing most importantly on the thought process, surprises, and gotchas behind the solution. Please note that this is not a step-by-step guide covering the entire development process from booting up your laptop. Of course, this may not be the best solution available, but it can serve as a valuable resource for someone trying to build something similar or explore further experimentation.

what is Polis ?

It was a cool name proposed by @GuiBibeau. As an added benefit, if someone I meet asks me what I'm currently doing, I can simply mention that I'm involved in police work, which is actually easier than explaining how blockchain works!

But Polis's primary function is to gather any dApp and present it to the world. While there are additional plans on the roadmap, for now, anyone can submit their app to Polis. Take a look for yourself here and feel free to submit if you have something to share. Also here is a link to the repo if you want to follow along.

Casting : Selecting a Hero and Uncovering a Villain Within

We had a requirement from the beginning that using dApp should be free for all our users. Further we wanted to make a decentralise database to have reduced reliance on centralised providers.

So we were in a quest to look for an hero!

First one to leave our list of considerations was the blockchain. Mostly because of the gas fees. since the amount of data that we are planing to collect is high too with data points such as descriptions, links and even images, it will be expensive to write to blockchain.

There are other database heroes out there like orbit-db, but we wanted something very simple, minimal and actually something relatively easy to build with what we already know.

And that, my friends, is how, as directors, we conducted our interview with the hero of the day and saviour, IPFS. have you met them too ? If not head over to https://ipfs.io/ to greet and meet. But if you are too much attached to our tale and still want to understand what it is, IPFS is a protocol for organising and transferring data which is designed to use content addressing instead of location addressing and peer-to-peer networking.

One drawback to consider is data mutability. Traditional databases facilitate easy data updates and modifications, whereas IPFS prioritises immutability. Once data is added to IPFS, it cannot be altered. To update data, you typically create a new version with a different content hash.

Our hero is slowly becoming a villain that we cannot control!

Then we searched for a way to build a mutable data storage with IPFS. one way of doing this is to implement a mutable file system MFS. Cool idea but in our case we don’t really a file storage. What we need is a database, which can be just one file.

As we delved deeper, we uncovered IPNS, and it turns out they were a perfect match. Stronger together, our villain transformed into a hero again!

Casting: Supporting roles

Front end

For our frontend, we enlisted Next.js because, let's face it, it's the cool kid in town. To save ourselves from CSS-induced headaches, we brought in Tailwind CSS – because, frankly, we're not CSS superheroes.

infura IPFS

infura is being used as our storage infrastructure. It has reliable and intuitive. with infura IPFS you can be confident that your IPFS files are always kept so that you won’t feel like you are uploading files to the Bermuda triangle.

In order to interact with infura you can try their api endpoints. There is a js client library for ipfs called ipfs-http-client which was deprecated around the time we started building polis. Nice timing ipfs-http-client, nice timing. It is possible to use kubo-rpc-client too for the same purpose but we went with http endpoints. This is a nice blogpost to get started with infura ipfs

There is a dedicated gateway for us to interact with data in infura which is in our case https://polis.infura-ipfs.io

w3name

https://github.com/web3-storage/w3name

One drawback with infura is that they do not support name methods yet. So which means you cannot publish names with infura by the time of writing. So another solution called w3name is being used in polis. w3name is a service and client library that implements IPNS, which is a protocol that uses public key cryptography to allow for updatable naming in an atomically verifiable way.

Here unfolds their adventure!

Act 1: IPFS: Entrusting Our Hero with Precious Treasures

For the sake of simplicity, we are creating a minimal JSON database named applications.json.

This JSON file consists of an array of objects, each representing an application submitted by a fellow web3 enthusiast. Here's an example featuring just one application.

[
    {
            "id": "oMObNGIHGupfgKkwUnK-W", // random string
            "user": "0xc12ca5A8c4726ed7e00aE6De2b068C3c48fA6570", // user's wallet address
            "createdAt": "9/1/2023, 11:34:27 PM",
            "category": [ "Security"],
            "title": "MobyMask ",
            "description": "This snap warns you when interacting with a contract that has been identified as a phisher in the MobyMask Phisher Registry.  ",
            "applicationUrl": "<https://montoya.github.io/mobymask-snap/>",
            "repoUrl": "<https://github.com/Montoya/mobymask-snap>",
            "screenshots": "QmddFhP3od4qRyLpzVQkoCuoGp3XRxHEeS1kckW3uHfiqF",
            "logo": "QmdiM7AnXtShBznk6HbHrTBttbetRYbeixmDjaypreuPCV",
            "isEditorsPick": false
    }

]

While searching for ways to build a simple database with IPFS we came across this awesome blog post by Radovan Stevanovic

Based on this post we built our own implementation considering the data structure we want. We didn’t build the all database functionalities to our database since we wanted to try a MVP of polis first. Here are some main functions in the database just enough to understand where are we heading. located in lib/database.ts

As we continue to receive more applications, this JSON object, and consequently the JSON file, expand in size. While this growth could potentially pose a challenge, we are not overly concerned at the moment. We can address scalability issues at a later stage.

While IPFS excels in its current role, we do need to consider how we'll handle updates, such as adding new applications or making edits to existing ones.

Get ready for some tech magic as we bend IPFS's rigid ways into a more flexible dance, all thanks to IPNS!”

Act 2: IPNS - The Heroic Twist to IPFS's Stiffness

Any update on the json file will result in a brand new file, brand new CID. Simply it is not the same anymore!

How does IPNS helps us to solve this ? IPNS allows users to associate a mutable, human-readable name with a specific IPFS content address. Even though the content linked to the IPNS name changes we can still get the updated content

Here's how IPNS works in brief:

1. An IPFS user generates a cryptographic key pair, where the private key remains secret, and the public key is used to create an IPNS namespace.

2. The user can publish an IPNS record by signing it with their private key. This record contains the desired mutable name and the corresponding IPFS content address (usually a hash).

3. The IPNS record is then distributed across the IPFS network through the Distributed Hash Table (DHT), making it accessible to other users.

4. When someone wants to access the content associated with the mutable name, they look up the IPNS record using the name and retrieve the current content address.

The Teaser

Alright, we've had our fill of technical talk. Now, let's sit back and enjoy a sneak peek at the play in action!

Go to this link in your browser. I am not trying to steal your passwords, I assure you

<https://name.web3.storage/name/k51qzi5uqu5dkzojav2vvqo7hdhdcdy7lfbx6i7rlseeq73nx7ueybagsol6yf>

Here the k51q.....agsol6yf is the ipns name for our database.

You will see the hash of the current database file as the value in the response. Then let’s find what’s inside that file

Here’s the link to do just that

<https://ipfs.io/ipfs/QmSfcCceyXrqHHRYgBR5pCmXNPqy3P2gLCsQBXtsxGbWNt>

What you see is hero’s work; our simple and humble json database.

Final Act : The Dynamic Duo's Front-End Adventure

This is how the duo's adventure unfolds: When the 'Submit Application' modal's submit button is clicked, it brings about a momentary silence, accompanied by a ceaselessly spinning loader icon right on the button. A substantial amount of data, including images, is then transmitted to the submitApplication function, which is where the climax of the process takes place!

export const submitApplication = async ({
  images,
  data,
}: {
  images: FormData;
  data: string;
}) => {
  let logoHash, screenshotsHash;

  const logo = images.get("logo") as File;
  const screenshots = images.getAll("screenshots") as File[];

    // adding the logo to ipfs and get back the hash of the file
  if (logo) {
    logoHash = await add({ file: logo });
  }

    // adding all screenshots to ipfs as a directory and get back the hash of the directory
  if (screenshots.length > 0) {
    screenshotsHash = await add({ files: screenshots });
  }

  const application = JSON.parse(data) as Omit<
    IApplication,
    "id" | "screenshots" | "logo"
  >;

    // retrive the latest database file. 
  const state = await retrieveDatabase();

  const newState = addNode(state, {
    id: nanoid(),
    ...application,
    screenshots: screenshotsHash,
    logo: logoHash,
    createdAt: new Date().toLocaleString("en-US", { timeZone: "UTC" }),
    isEditorsPick: false,
  });

    // creating and add a new file to ipfs, then publish its hash to ipns
  await storeDatabase(newState);

    // nextjs cache revalidation to show the newly added application
  revalidatePath("/");
};

Let me guide you in explaining this

After a few dialogs, the duo works together to retrieve the database.

const retrieveDatabase = async () => {
    // getting the latest db hash
  const hash = await getcurrentHash();

  const json = await cat(hash);
  return deserializeDatabase(json);
};

Our hero, IPFS, is in a state of confusion; he doesn't know the location of the secret scroll (our database file), leading to the treasure. So he asks IPNS and then await for his reply

const hash = await getcurrentHash();

IPNS replies with a hash just using one of his super powers.

const getcurrentHash = async () => {
  const nameServiceUrl = process.env.WEB3_NAME_SERVICE_URL; //https://name.web3.storage/name

  if (!nameServiceUrl) {
    throw new Error("WEB3_NAME_SERVICE_URL is not set");
  }

  const res = await fetch(`${nameServiceUrl}/${process.env.DB_HASH}`);

  return (await res.json()).value; // hash of the latest db file
};

Relived, IPFS then just fetches the scroll and pass it onward to discover the treasure; the json database

while our Duo is resting under a palm tree, rest of the crew is trying to put the the new application data load into the json database.

const addNode = (
  state: Map<string, ApplicationNode>,
  node: ApplicationNode
) => {
  return new Map(state).set(node.id, node);
};

But before loosing, it has to be saved back. Duos rest is over now have to work together again for the storeDatabase adventure.

const storeDatabase = async (state: Map<string, ApplicationNode>) => {
  const json = serializeDatabase(state);
  const jsonDataBlob = new Blob([json], { type: "application/json" });
    // adding the file to IPFS
  **const hash = await add({ blob: jsonDataBlob, fileName: "db.json" });**

  if (!hash) {
    throw new Error("couldn't store database");
  }

    // updating the ipns record with the latest hash: refer the next act
  **await update(process.env.DB_HASH!, process.env.DB_KEY!, hash);**
};

IPFS is showing his power. he is so powerful that he can add one add multiple files as a directory. But for sake of simplicity I am showing just one here, adding one file.

export const add = async ({ blob: Blob, fileName: string}) => {

  // append ?wrap-with-directory=true to the baseUrl to upload a directory
  const baseUrl = `${process.env.INFURA_IPFS_ENDPOINT}/api/v0/add`;

  const formData = new FormData();

  formData.append("data", data.blob, data.fileName);

  try {
    const response = await fetch(baseUrl, {
      method: "POST",
      headers: {
        Authorization:
          "Basic " +
          Buffer.from(
            process.env.INFURA_IPFS_KEY + ":" + process.env.INFURA_IPFS_SECRET
          ).toString("base64"),
      },
      body: formData,
    });

    const hash = (await response.json()).Hash;


    return hash;

  } catch (error) {
    console.error("Error adding file", error);
  }
};

Now it's IPNS's time to shine. He holds some important secrets that only he can access and change using his private key, as he's the guardian of the name.

import * as Name from "w3name";

const update = async (ipns: string, keyStr: string, content: string) => {

    if (!keyStr) {
    return;
  }

  const name = Name.parse(ipns);
  const revision = await Name.resolve(name);

  const nextRevision = await Name.increment(revision, content);

  const key = new Uint8Array(Buffer.from(keyStr, "base64"));

  const name2 = await Name.from(key);
    // publishing the next version of the db with the private key
  await Name.publish(nextRevision, name2.key);
};

In this exhilarating finale, the Dynamic Duo's front-end adventure comes to a close, leaving you with awesome list of application with a nice UI to see. Thank you for joining us on this remarkable adventure!

The Blooper Reel: Things that didn’t make it to the end

Well, what you saw is the “perfect” ending. As real builders know the road to the final ending is not always the smoothest. So These are the things that we have tried, failed and possible shortcomings in the future

• Combine the old world and the new world! What if we store application data in IPFS and keep the data on the user in a web2 database like Supabase? With that, we can easily build authentication and authorization. But it felt like charging a Tesla with a diesel generator while thinking that I am helping the environment.

• In the initial version of the dApp, we tried to keep the data of a single application in a single directory in IPFS (data, screenshots, and logo). Then, we would publish it in IPFS separately. This approach led to the creation of many IPFS files. Additionally, users had to manage their IPNS private keys on their own. The idea was to allow updates on a given application only for its creator. However, this idea was scrapped as it proved difficult to query for applications, and when we did, it was much slower. Another challenge was managing private keys. We explored the idea of incorporating MetaMask Snaps for the same, which was working. However, other issues took precedence, and the entire idea was abandoned.

• Tried Gun, orbitDb and some other database solutions.

• “Just do everything in Supabase!, why bother moving to decentralised solutions:”

      On most frustrated days
  

So, my friend, that's how my developer tale comes to a close. I hope you've enjoyed it. Until next time, take care.