DBOS-Cloud: Fast and Free Automatic Data Aggregator

Vince FulcoVince Fulco
3 min read

This is a simple DBOS app example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

The github repo sets up a cron job which: 1) performs a "SELECT COUNT" on a Supabase Postgres database table and 2) an "INSERT" to a second table. Consider it a starter for a poor man's data lake aggregating data in the receiving table.

Note: This is not the fastest, most compact and recommended way to create this workflow. DBOS-Cloud has terrific in-app full Postgres database functionality. Check out the transactions section for more. I built the script this way since I had pre-existing data stored on Supabase.


Prerequisites

  • Make sure you have Node.js 21.x

  • Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)

  • Have a Supabase account and get your project url (https://foobar.supabase.co/rest/v1) and project API anon key. Check out the Supabase docs to get them.

  • Make sure to have two tables set up, the first can have any data in it as we will be counting rows, and the second table should have the columns "id", "count", and "created_at".


Getting Started

Clone this repository and navigate to the project directory

  • Install the dependencies

  • To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.

  • Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"

  • Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"


Secrets Handling

To use secrets in DBOS, add your variables in the cli like this:

  • export PGPASSWORD=password-when-you-setup-the-remote-database-here

  • export SUPABASE_URL=https://foobar.supabase.co

  • export SUPABASE_ANON_KEY=long-foo-bar-key

  • export SOURCE_TABLE=source-table-name

  • export DESTINATION_TABLE=destination-table-name

These will be picked up at build time and inserted into the dbos-config.yaml fields: ${PGPASSWORD}, ${SUPABASE_URL}, ${SUPABASE_ANON_KEY}, ${SOURCE_TABLE}, ${DESTINATION_TABLE} respectively. Don't use quotation marks wrapping the values. The PGPASSWORD is for the basic user (not admin) Postgres service, which won't be used here.

Be sure not to commit / hard code your secrets to a public repo! This setup is locally developed and directly deployed to the dbos-cloud service.


Get ready to deploy!

  • And finally deploy your app: "npx dbos-cloud app deploy"

After 5 minutes, assuming you have data in your source table, check your destination table for entries.

  • If you are not getting expected results:

Login to the monitoring dashboard using this "npx dbos-cloud dashboard url"and check the collection of logs.

  • To delete the app online,npx dbos-cloud app delete [application-name] --dropdb. Remove the '--dropdb' parameter if you want to retain the database table. If you don't delete it, the cron script will continue to run.

There are so many more benefits to mention re: workflow orchestration, observability, debugging, monitoring and overall speed, security, and costs which can be explored further here: https://docs.dbos.dev/


Reference Docs (From Official Repo)


Resources to learn more --

Awesome-dbos

The first article in this series using cron and Postmark email service provider.

0
Subscribe to my newsletter

Read articles from Vince Fulco directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Vince Fulco
Vince Fulco