Go for Dart with Protocol Buffers

Introduction

In this article, I propose a solution that enables a rapid expansion of the available APIs for Dart to be used on the server, by using existing drivers for Go. I begin by laying out the context of why this is needed. Then, I describe the solution. Finally, I do a walk around mongo_go, the package I implemented following this approach.

The general context

Dart has proven its power on the UI front. However, it still does not have the traction it deserves on the server. There are at least the following two aspects worth considering.

One, it needs a complete server-side framework. There are already some packages that help a lot with this. There's Shelf. There is also the grpc package. While it’s not only focused on server-side development, it also very well supports hosting gRPC services with Dart. There are also frameworks such as Conduit and more. Finally, I am the maintainer of Dartaculous. It is a different kind of full-stack Dart framework that leverages gRPC.

The second aspect is the lack of a set of official drivers to connect to services, like for instance database engines. I will be focusing on connecting to databases, but that is a subset of a broader issue. Normally we would need to also be able to connect to queue managers, caching services, and online APIs, amongst other examples. Even services with REST or gRPC endpoints are sometimes easier to use with language-specific APIs. An example is Firebase's Admin SDK, which has REST and gRPC endpoints, but also packages for C++, Go, and so on. Using external services that require a specific API to work with has been, by far, my greatest challenge when using Dart for the server. Good luck trying to connect Dart to an Oracle database server for instance (just for example, not that I want to). The story around open-source databases has, by nature, been a better one. For instance, there are community-developed packages for PostgreSQL and MySQL.

Other than that, Dart has all the features needed for a server-side language.

Why Dart for the Server

If you have gotten here and are still reading this, you probably do not need any convincing as to why it is appealing to use Dart on the server. Eventually, you may skip this part. However, if you are wondering why not simply limit Dart's use to Flutter and use a language with good support on the server, like Go, here are the reasons.

Most of the applications I develop are line-of-business applications that carry many entities back and forth between the client and the server. This means that the size of my business model tends to be more than trivial. Using the same language for the client and the server means I get to reuse the same implementation of that business model on both ends, including validations.

Even more, there is further immense value in having most of the system represented in a single programming language. It adds flexibility. It significantly lowers the barrier to full-stack development. The benefits are analogous to the ones we get from using document databases instead of having to deal with the mapping between objects and relational. It is like we are removing a different kind of impedance mismatch.

Our specific challenge

There is an excellent Dart community-developed driver for our database server of choice, MongoDB. It is mongo_dart. However, of the few missing features of that driver, streams and transactions were crucial ones for us. These features are present in Mongo DB's official drivers for several languages with official support.

There is a pending issue with the community driver for some time for bringing sessions to it (issues 148, 200 and 274). So, I inferred that if its developers still have not added the feature by now, then it probably would not be trivial for me to dive into that code and do it myself. This led me to think “if only I could reuse an official driver and easily re-expose it for Dart, and by the way, if this could result in a reusable approach to similar issues...”.

The Solution

The Gist

A possible path to solve this issue would be to develop a microservice in Go. It would receive requests from Dart and then use Mongo DB's official driver for Go to execute those requests over the database. A good protocol to connect Dart to Go would be gRPC, using Protocol Buffers. If you are not familiar with gRPC, it is a high-performance RPC framework initially developed and currently stewarded by Google. Protocol Buffers are the data format with which messages are encoded. The structure of that data is defined by an interface definition language (IDL). The IDL is not specific to any programming language. However, at development time, we can compile the IDL definitions into types specific to each of the supported programming languages. For developers that use REST, the IDL is Protocol Buffers' equivalent to OpenAPI specification documents. Implementing a Go microservice would work, but it would require yet another service in the stack.

It just so happens that Go can consume C libraries. Even better, Go packages can be compiled into libraries that expose native C APIs. Both are a part of a feature in Go called cgo. And it just so happens that Dart can invoke dynamic libraries that expose native C APIs. This feature is called Dart FFI. FFI stands for Foreign Function Interface. So, an alternative path would be to have Dart call Go functions via FFI instead of implementing a Go microservice.

The challenge with having Dart call GO methods via Dart FFI is that Dart and Go have wildly different type-systems. The problem is compounded by the fact that both languages have garbage collectors, yet what lies in the middle are C-style structs that require manual memory allocation and deallocation. (We could also try to implement a Dart package that directly uses Mongo DB's C driver through FFI, but that would have been an even more daunting undertaking, and we would be losing out on Go routines - more on that later).

Why not use aspects of both approaches? We could have Dart and Go live under the same process and communicate with each other via FFI. However, instead of marshalling our Dart objects to C structs and then from those to Go objects, a much simpler approach would be to use Protocol Buffers as a lingua franca between both languages. With this approach, we still need to marshal from Dart to C and from C to Go, but with a major difference. We will no longer be converting each different instance of a Dart class to a C struct, and from a C struct to a Go object. Instead, we map the Dart object into a Protocol Buffer message, which is serialized into a byte array. That byte array is sent via Dart FFI to Go. Go then deserializes the byte array into a Protocol Buffer message and maps it into the final Go struct.

So, what are the advantages of this approach? Let’s say we’ve got a graph of objects and we want to pass that graph from Dart to Go. With Dart FFI without using Protocol Buffers, we would need to allocate memory for each of the objects in the graph, do all the conversions, invoke the Go function and then ensure we release all the allocated memory for all the objects we converted to C structs (I’m simplifying things). With Protocol Buffers, we map the objects into Protocol Buffer messages (which are also garbage-collectable objects) and then serialize those into a byte array. This means that the entire graph gets represented by a single array of bytes. Then, there is the single point of manual memory allocation for the array, so we can pass that array to Go. After completing the call, again we have a single point of memory deallocation for the array we passed. It is much easier to use this approach and much safer. Manual memory allocation has been a major source of vulnerabilities. So, it is always best if we can reduce the number of places where we do it to the absolute minimum.

In fact, by always passing arrays, we can have most functions in Go implemented with the same signature. This means we can implement a reduced set of reusable methods in Dart to call all the functions in Go. This reduces the potential for mistakes and for inadvertently introducing a vulnerability (or a memory leak, or a crash) when making calls to Dart FFI.

Because this method is so reusable, we published a Dart package meant precisely to aid the usage of FFI with Protocol Buffers to call Go functions. It is the go_bridge package. There is a corresponding Go package to aid in receiving calls from Dart via FFI with Protocol Buffer messages. It is the dart_bridge package. Now, here’s great news for you if you intend to use this technique. All the manual memory allocation and deallocation you will ever need to do are implemented for you in those packages. This means you may use Dart FFI without worrying about manual memory management.

Asynchronous Execution

Dart is used, by default, as a single-threaded language, with event loops being used for asynchronous execution. Dart does have a feature called Isolates for multi-threading, but unlike other multi-threaded languages, it does not support memory sharing. It uses message passing instead. For the sake of the rest of this discussion, we may ignore Isolates, but here's an excellent explanation. Go, on the other hand, is famous for how easy it is to do multi-threading with Goroutines and channels. The challenge here is that once again we need to use two different execution models and fit them through a C API.

How to get asynchrony between Dart and Go, then? The adopted solution to this uses a Dart ReceivePort together with a Dart Completer. A ReceivePort is a Dart object that allows us to receive messages from other Dart isolates. What is so special about this object is that we can also use it to receive messages from other languages through the C API. Each ReceivePort object has a SendPort attached to it. We can send the SendPort to Go (in reality we are sending its native port). Go can then send messages back through the SendPort. Those messages are handled by the ReceivePort. A Completer is a Dart object we can await until something calls its complete method.

So, here's the sequence of events, from Dart's side:

1. We create a Completer and a ReceivePort. We set up the ReceivePort to complete the Completer when the ReceivePort receives a message.

2. We invoke a Go function, sending the ReceivePort's SendPort and whatever data we want to send to Go.

3. We await the completer and handle the result it returns.

Here is how it goes from Go's side (doesn't this sound cool?):

1. The function called has a parameter with the SendPort it received from Dart and any other parameters containing whatever additional data.

2. It creates a Goroutine passing the port and the data (this is a simplification as right now we are focusing on async execution and leaving the data marshalling aside for now).

3. Whenever the Goroutine does its job, it sends a message through SendPort.

The following sequence diagram illustrates what we have just discussed:

mongo_go

From this point, it will be easier to explain how everything is put together by using a concrete implementation. For that purpose, we will be looking at a package called mongo_go as a reference implementation. mongo_go is the first package that I implemented using the proposed approach in this post. It is in essence MongoDB's GO driver, re-exposed for Dart.

The Go side

The first file we can look into is the facade that Go exposes for Dart. It is the mongo_go.go file. This file defines all the functions that Dart will call. There are all sorts of functions here, that allow us to connect to the DB server, get handles to databases, get handles to collections, insert documents, find documents, and so on. It is easy to identify which functions are meant to be called by Dart, as they all have a comment that instructs cgo to export them, in the form //export [function name]. Here is a very short snippet of the available methods with their implementation removed:

//export initialize
func initialize(api unsafe.Pointer) { 
    /// implementation
}

//export connectMongo
func connectMongo(port int64, buffer *C.uchar, size int) {
    /// implementation
}

//export database
func database(port int64, buffer *C.uchar, size int) {
    /// implementation
}

//export collection
func collection(port int64, buffer *C.uchar, size int) {
    /// implementation
}

//export updateOne
func updateOne(port int64, buffer *C.uchar, size int) {
    /// implementation
}

//// many others

The first function we need to call from Dart is initialize. It needs to be called only once. This call is needed for Go to be able to send messages back to Dart. Dart must call this method at least once before calling any other function in this library. So, with this, let's get initialize out of the way and not think of it again.

Notice that all the functions shown above, except initialize, have the same signature. This does not mean that they receive the same types. It means that the type definitions are not visible in the signature of the functions, but they are present in the Protocol Buffer IDL definitions (more on that in a short while). The parameters are:

• port: The SendPort that the function will use to send the return value to Dart's ReceivePort. If you're confused with all the ports by now, just think of it as "the communication channel through which Go will return its value to Dart".
• buffer: The array of bytes containing the protocol-buffer encoded message.
• size: The size of the array of bytes.

To export this package as a dynamic library that exports these functions as C native functions, we can run the following command in the terminal (you may look at the package's Makefile):

go build -buildmode=c-shared -o mongo_go.so

This will compile the Go package into mongo_go.so. It will also generate a header file containing all the C function declarations. Here is its core:

extern void initialize(void* api);
extern void connectMongo(GoInt64 port, unsigned char* buffer, GoInt size);
extern void disconnect(GoInt64 port, unsigned char* buffer, GoInt size);
extern void startSession(GoInt64 port, unsigned char* buffer, GoInt size);
extern void closeSession(GoInt64 port, unsigned char* buffer, GoInt size);
extern void startTransaction(GoInt64 port, unsigned char* buffer, GoInt size);
extern void commitTransaction(GoInt64 port, unsigned char* buffer, GoInt size);
extern void abortTransaction(GoInt64 port, unsigned char* buffer, GoInt size);
extern void database(GoInt64 port, unsigned char* buffer, GoInt size);
extern void dropDatabase(GoInt64 port, unsigned char* buffer, GoInt size);
extern void collection(GoInt64 port, unsigned char* buffer, GoInt size);
extern void listDatabaseNames(GoInt64 port, unsigned char* buffer, GoInt size);
extern void insertOne(GoInt64 port, unsigned char* buffer, GoInt size);
extern void insertMany(GoInt64 port, unsigned char* buffer, GoInt size);
extern void updateOne(GoInt64 port, unsigned char* buffer, GoInt size);
extern void updateMany(GoInt64 port, unsigned char* buffer, GoInt size);
extern void replaceOne(GoInt64 port, unsigned char* buffer, GoInt size);
extern void deleteOne(GoInt64 port, unsigned char* buffer, GoInt size);
extern void deleteMany(GoInt64 port, unsigned char* buffer, GoInt size);
extern void findOne(GoInt64 port, unsigned char* buffer, GoInt size);
extern void findOneAndDelete(GoInt64 port, unsigned char* buffer, GoInt size);
extern void findOneAndUpdate(GoInt64 port, unsigned char* buffer, GoInt size);
extern void findOneAndReplace(GoInt64 port, unsigned char* buffer, GoInt size);
extern void find(GoInt64 port, unsigned char* buffer, GoInt size);
extern void countDocuments(GoInt64 port, unsigned char* buffer, GoInt size);
extern void estimatedDocumentCount(GoInt64 port, unsigned char* buffer, GoInt size);
extern void aggregate(GoInt64 port, unsigned char* buffer, GoInt size);
extern void watch(GoInt64 port, unsigned char* buffer, GoInt size);
extern void createOneIndex(GoInt64 port, unsigned char* buffer, GoInt size);
extern void listIndexes(GoInt64 port, unsigned char* buffer, GoInt size);
extern void dropOneIndex(GoInt64 port, unsigned char* buffer, GoInt size);
extern void dropAllIndexes(GoInt64 port, unsigned char* buffer, GoInt size);
extern void bulkWrite(GoInt64 port, unsigned char* buffer, GoInt size);

From now on, we will use the insertOne function as a reference example. This corresponds to Mongo DB's insertOne operation. That operation inserts a single document into a collection. Let's start building this function from scratch as if it didn't exist already. So, here's its empty definition:

//export insertOne
func insertOne(port int64, buffer *C.uchar, size int) {
    /// implement insertOne
}

The first thing we need to do is to receive the data that an insertOne operation needs to receive. Without wasting too much time going into the detail of how all the properties we need are going to be used, let's just identify them:

• A key identifying which collection we are adding the document to.

• An optional key identifying the session under which we are inserting the document.

• The document to be inserted.

All this data is to be received within the array of bytes pointed by the buffer parameter. So, how is that array of bytes supposed to be encoded? It's time for a detour so we go into the use of protocol buffer definitions.

Protocol Buffer Definitions

For each of the Go functions, we will have a corresponding Protocol Buffer message definition that defines the structure of the input array of bytes (the buffer parameter) and, optionally, another message definition that defines the structure of the response message Go should send back to Dart.

That is the case for the insertOne method. This function has the following associated message definitions, found in the insert_one.proto file:

syntax = "proto3";

option go_package = "./mongo_stubs";

message InsertOneRequest {
  bytes collectionOid = 1;
  bytes sessionOid = 2;
  bytes document = 3;
}

message InsertOneResult {
  bytes insertedId = 1;
}

Let's call the syntax of this file Proto3 as it is version 3 of the Protocol Buffers Interface Definition Language. A complete reference to this IDL syntax can be found at https://developers.google.com/protocol-buffers/docs/proto3. It is a big coincidence, not a requirement, that all the fields of InsertOneRequest are byte arrays. We can use any of the types supported by Proto3. I am trying to keep us focused on the interplay between Dart and Go and not so much on the specifics of the end goal of using MongoDB. However, it doesn't hurt to quickly explain the meaning of each field in InsertOneRequest:

• collectionId is a key identifying the collection to which the document is to be added. We will not go into detail about how we are identifying connections, databases and collections. However, its type is a MongoDB ObjectId. Because there is no Proto3 type to represent an ObjectId, we are using a byte array.

• sessionId is an optional key identifying which session the insert operation is to be executed under. It is also an ObjectId.

• document is the actual document. We were lucky that MongoDB uses BSON to encode documents. We were even luckier that the developers who created the mongo_dart package also created a package to encode and decode Dart objects to BSON and back, called bson. As you might have guessed by now, a BSON-encoded document is an array of bytes, which is why this field is also an array of bytes.

If any of the fields were, for example, another message type, an int32, a string, or any of the supported types, it would have made sense to represent it in that type instead of an array of bytes. An example of a message definition that uses less generic types than an array of types is the connectionRequest message.

As previously mentioned, Protocol Buffer definitions are independent of any specific language. However, we get to generate representations for all the supported languages. For Go, all we need to do is run the following, from the context of the Go package's directory (this is also in the Makefile):

protoc --go_out=. -Iproto proto/*.proto

If you don't have protoc installed, you can follow the instructions at https://developers.google.com/protocol-buffers/docs/gotutorial#compiling-your-protocol-buffers.

As a result, protoc compiler will generate a corresponding Go struct for each proto message (we will also do the same thing for Dart, but later). In our example, execution protoc over the InsertOneRequest message will result in the following struct:

type InsertOneRequest struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    CollectionOid []byte          `protobuf:"bytes,1,opt,name=collectionOid,proto3" json:"collectionOid,omitempty"`
    Context       *RequestContext `protobuf:"bytes,2,opt,name=context,proto3" json:"context,omitempty"`
    Document      []byte          `protobuf:"bytes,3,opt,name=document,proto3" json:"document,omitempty"`
}

Notice how each of the fields defined in the Proto3 message is also represented in the Go struct. Let's go back to our insertOne implementation and add continue working on it. We can now add the code that converts the array of bytes we received from Dart back into an instance of an InsertOneRequest struct.

//export insertOne
func insertOne(port int64, buffer *C.uchar, size int) {

    // lets first create a variable, 'request',
    // to hold the struct
    var request mongo_stubs.InsertOneRequest

    // and here we deserialize the contents of 'buffer'
    // into properties of 'request'
    ffi.Unmarshal(unsafe.Pointer(buffer), size, &request)

    // we're still not done yet. a lot will still be added here
}

ffi.Unmarshal receives the array of bytes, its size and a pointer to an instance of the InsertOneRequest struct, and fills the properties of that instance with the values contained in the array. In other words, it deserializes the array into the struct referenced by the request local variable. This function belongs to a group of utility functions that are useful anytime we will need to develop any Go library that exposes its interface to Dart using Protocol Buffers. For that reason, they were packed into a separate Go package called dart_bridge, which you can use for your own Go for Dart packages.

At this point, our insertOne method can receive all the data it needs to receive from Dart. If we need to receive additional data, all we need to do is add more fields to the Proto3 definition and re-run the protoc compiler. From this point on, we will not be making any other use of the buffer variable. This means that we can spawn a Goroutine to do the rest of the work. After that, we can immediately return from this function (we expect that as soon as we return, the Dart caller will immediately free the memory it allocated for the array of bytes pointed by buffer). So, here's how the code will look after spawning an empty Goroutine:

//export insertOne
func insertOne(port int64, buffer *C.uchar, size int) {
    var request mongo_stubs.InsertOneRequest
    ffi.Unmarshal(unsafe.Pointer(buffer), size, &request)

    // At this point, 'request' refers to a Go struct of type
    // mongo_stubs.InsertOneRequest. This struct instance
    // is garbage-collectable by Go when it will no longer
    // be needed.

    // Its contents resulted from the unmarshalling of the 
    // array of bytes pointed by buffer.
    // That array of bytes is not garbage-collectable and
    // its memory must be freed by this function's caller.

    go func() {

        /// This is where we will do the work
        /// of inserting a document in the database
        /// and returning a value to Dart

        /// It's ok and expected for us to use 'request' here. 
        /// But, we must never try to use the 'buffer' inside
        /// the go routine. Here we must assume the caller
        /// of this method has already freed its memory.

    }()

    /// after returning the caller is expected to free 'buffer'.
}

We will finally have this function do what it is meant to do, to insert a document in a collection. Even though this iteration of the method is the one that does what the method is intended to do, it is the least significant in terms of understanding how we are creating the interaction between Dart and Go. So, it's OK if you just retain from this iteration that the lines of code we added are the ones that do stuff and don't worry if you don't understand precisely what they are doing:

//export insertOne
func insertOne(port int64, buffer *C.uchar, size int) {
    var request mongo_stubs.InsertOneRequest
    ffi.Unmarshal(unsafe.Pointer(buffer), size, &request)

    go func() {

        /// This is the bit of code that actually 
        /// does the action. 
        ///

        ctx := context.Background()

        /// here we are getting a reference to the collection
        /// from the CollectionOid we received
        oid := helpers.BytesToOid(request.CollectionOid)
        coll, err := cs.GetCollectionProxy(oid)
        if err != nil {
            /// [left out for now]
            return
        }

        /// next, we try to get a reference to the session
        trxProxy, err := _getSessionProxy(coll.databaseProxy.connectionProxy, request.SessionOid)
        if err != nil {
            /// [left out for now]
            return
        }

        /// Finally we do the actual insert of the document
        r, err := coll.InsertOne(ctx, trxProxy, request.Document)

        /// we still need to handle the result

    }()
}

We are almost done and we are returning to a very relevant part of the discussion. To finish the function we need to either return an error or otherwise return a result. To send a message back to Dart, we will be creating an instance of a Go struct that represents the message as it was defined in a Proto3 definition. For this method, we intend that when things go well, we will be sending back to Dart an InsertOneResult as we've seen previously defined in insert_one.proto file. Here we're repeating only the relevant snippet:

message InsertOneResult {
  bytes insertedId = 1;
}

As we've learned previously, the result of the compilation performed by protoc is that we end up having a corresponding Go struct for each message defined in our Proto3 files. So, once again, we will have a Go struct called InsertOneResult. So, to finish our method we will either send an error or we will send an InsertOneResult. For brevity, we will not go into detail about how to send an error, but limit to mention that we also represent errors as any other message type.

Here's the final version of the function:

//export insertOne
func insertOne(port int64, buffer *C.uchar, size int) {
    var request mongo_stubs.InsertOneRequest
    ffi.Unmarshal(unsafe.Pointer(buffer), size, &request)

    go func() {
        ctx := context.Background()
        oid := helpers.BytesToOid(request.CollectionOid)

        coll, err := cs.GetCollectionProxy(oid)
        if err != nil {
            /// Sending an error
            helpers.SendErrorMessage(port, err)
            return
        }
        trxProxy, err := _getSessionProxy(coll.databaseProxy.connectionProxy, request.SessionOid)
        if err != nil {
            /// Sending an error
            helpers.SendErrorMessage(port, err)
            return
        }
        r, err := coll.InsertOne(ctx, trxProxy, request.Document)
        if err != nil {
            /// Sending an error
            helpers.SendErrorMessage(port, err)
            return
        }

        /// The function called here translates
        /// whatever is present in 'r' (the return value
        /// from MongoDB) into an instance of InsertOneResult
        m, err := marshalling.ToInsertOneResult(r)
        if err != nil {
            /// Sending an error
            helpers.SendErrorMessage(port, err)
            return
        }

        /// Finally, we send the InsertOneResult
        ffi.SendMessage(port, m)
    }()
}

It is extremely important to retain that the Goroutine must send a message back to Dart. This is either accomplished by calling helpers.SendErrorMessage(...), by calling ffi.SendMessage(...), or otherwise by calling any other function that sends a message to Dart.

Once again, because ffi.SendMessage(...) is expected to be commonly reused, it is another feature of the dart_bridge Go package. It encapsulates the message into a Response message and, internally, uses the SendUInt8ArrayToPort(...) method. Here's the implementation (you don't need to understand it to use the SendUInt8ArrayToPort(...) function as long as you understand the meaning of its parameters and what it is meant to do):

// #include <stdlib.h>
// #include "stdint.h"
// #include "include/dart_api_dl.c"
//
///// [Other C functions here removed for brevity]
//
// bool GoDart_Send_UInt8Array(Dart_Port_DL port, uint8_t* values, int64_t length) {
//     Dart_CObject obj;
//   obj.type = Dart_CObject_kTypedData;
//   obj.value.as_typed_data.length = length;
//   obj.value.as_typed_data.type = Dart_TypedData_kUint8;
//   obj.value.as_typed_data.values = values;
//     return Dart_PostCObject_DL(port, &obj);
// }

import "C"
import (
    "unsafe"
)

//// [other functions removed for brevity]

func SendUInt8ArrayToPort(port int64, value []uint8) { 
    /// low-level code implementation that uses 
    /// Dart SDK to sends the contents of 'value' 
    /// back to Dart via the 'port'
    C.GoDart_Send_UInt8Array(C.int64_t(port), (*C.uint8_t)(unsafe.Pointer(&value[0])), C.int64_t(len(value)))
}

If you remember the previous discussion about asynchronous execution, you might remember that for Dart to receive a return message from Go, Dart creates a ReceivePort and we send the ReceivePort's SendPort to Go. The port parameter of the SendUInt8ArrayToPort is precisely that SendPort. This is the very first parameter in all the exported Go functions and we've finally reached the place where it is used. It is SendUInt8ArrayToPort's function to send an array of bytes to that port. In other words, it is this method that ends up sending data back to Dart.

The Dart Side

So, we have a Go library that poses itself as a C library and now we need to consume it from Dart. This means we need to integrate that library into our Dart package. So, let's begin by looking at the parts that matter in our pubspec.yaml file:

### all other elements were removed for clarity

dependencies: 
  ffi: ^2.0.1
  go_bridge: ^1.3.0
  bson: ^3.0.0
  # all other dependencies removed for clarity

dev_dependencies:
  ffigen: ^7.2.0
  # all other dev-dependencies removed for clarity

ffigen:
  name: LibMongoProxy
  description: MongoProxy GO driver
  output: 'lib/src/lib_mongo_go.dart'
  headers:
    entry-points:
      - go/mongo_go.h

These additions to pubspec.yaml add FFI and FFIGen packages and configure FFIGen. This configuration instructs the tool to generate a Dart file in lib/src/lib_mongo_go.dart containing the FFI definitions corresponding to the header file generated by Go when we compiled our Go package as a C library. To run Dart FFI, all we need to do is run the following command from the terminal:

dart run ffigen

Every time we change Go's exports, we should run the above command.

Just like we compiled all our Proto3 messages into Go structs, we also need to compile the same Proto3 messages into Dart classes. We can run the following command from the terminal to do so:

protoc --dart_out=./lib/src/gen -Igo/proto go/proto/google/protobuf/*.proto go/proto/*.proto

Every time we change a Proto3 definition, we should run the command above.

We can now start looking at Dart code. Just like we have a facade on Go's side, it is a good idea to also have a corresponding Dart library on Dart's side. We centralize all the calls to Go in this library. In our implementation, it is mongo_go.dart.

To continue our walk around this package, we will continue looking at how to call Mongo DB's insertOne operation. As you might have guessed, resulting from the above execution of protoc, one of the Dart classes we will end with will be InsertOneRequest.

The user's entry point is the Collection.insertOne method:

Future<InsertOneResult> insertOne(
    Map<String, dynamic> document, {
    Session? session,
  }) async {
    final bson = BSON();
    final bytes = bson.serialize(document);

    final result = await p.insertOne(
      collectionId,
      bytes,
      sessionOid: session?.sessionId,
    );
    final ret = InsertOneResult.fromProto(result);
    return ret;
  }

Even though we spread the features through classes where they made sense to the user, Collection, Database, Connection, and so on, the actual calls to Go are concentrated in one Dart library, mongo_go.dart, which in the example is aliased with p. Let's look at a relevant slice of the mongo_go.dart library for this walk around:

final nl = getNativeLibrary();

void initialize() {
  nl.initialize(NativeApi.initializeApiDLData);
}

/// [other functions here removed for brevity]

Future<InsertOneResult> insertOne(
  ObjectId collectionOid,
  BsonBinary document, {
  ObjectId? sessionOid,
}) async {
  final oid = collectionOid.toByteList();
  final soid = sessionOid?.toByteList();

  final request = InsertOneRequest(
      collectionOid: oid, sessionOid: soid, document: document.byteList);
  final response = await callGoFunc(
    request: request,
    goFunc: nl.insertOne,
    responseToFill: InsertOneResult(),
  );
  return response;
}

/// [other functions here removed for brevity]

Let's begin at the very first line, final nl = getNativeLibrary();. It gets a reference to the Dart FFI proxy that represents the native library we built with Go and assigns it to the nl variable. That variable is needed by all other functions as the access point to call Go.

Here is the implementation of the getNativeLibrary function (present in the helpers.dart library):

LibMongoProxy getNativeLibrary() {
  final dylib = bridge.getDynamicLibrary('mongo_go.so');
  final nl = LibMongoProxy(dylib);
  return nl;
}

Back to mongo_go.dart, The very first function we implemented is the call to initialize:

void initialize() {
  nl.initialize(NativeApi.initializeApiDLData);
}

Remember, it is this call that will enable Go to send messages back to Dart. This could be called from anywhere, as long as it is called before any other function in mongo_go.dart. We could have left it to the packages consuming this one. However, because all interactions with this package will necessarily begin with a connection request, we placed the call to the initialize function inside the connect method of the Connection class :

  static Future<Connection> connect(ConnectionSettings settings) async {
    // here's the call to initialize:
    p.initialize();

    // let's ignore the rest and skip to insertOne
    final oid = await p.connect(settings.toConnectionRequest());
    final connection = Connection._(connectionId: oid);
    return connection;
  }

We are finally going to look at how we implemented the call to the insertOne function we created in Go. Here's the method's definition in mongo_go.dart again:

Future<InsertOneResult> insertOne(
  ObjectId collectionOid,
  BsonBinary document, {
  ObjectId? sessionOid,
}) async {
  final oid = collectionOid.toByteList();
  final soid = sessionOid?.toByteList();

  final request = InsertOneRequest(
      collectionOid: oid, sessionOid: soid, document: document.byteList);
  final response = await callGoFunc(
    request: request,
    goFunc: nl.insertOne,
    responseToFill: InsertOneResult(),
  );
  return response;
}

Note how we are creating an instance of InsertOneRequest, the class that resulted from the compilation of the corresponding Proto3 definition done by protoc.

The call to Go is then entirely done by a method called callGoFunc. It receives the object to be serialized to protocol buffers, receives the native Go function to be called and an object to be filled in (by deserializing it from protocol buffers) in case the function runs successfully (I will continue skipping error handling for this explanation). Even though the callGoFunc we are seeing is specific to this package, and it does error handling specific to this package, it, internally, calls a method with the same name from the Go Bridge package that finally does the actual call to Go. All the calls we are doing to Go are following this pattern. Notice that this code doesn't worry about any FFI details as we've been able to sweep all of those worries under the Go Bridge package. Here's the signature of the final callGoFunc that does the call to Go:

Future<TResponse> callGoFunc<TResponse extends GeneratedMessage>({
  required GeneratedMessage request,
  required void Function(int port, Pointer<UnsignedChar> buffer, int size)
      goFunc,
  required TResponse responseToFill,
  List<GeneratedMessage>? errorsToThrow,
}) {
    /// [Implementation left out of brevity]
}

It is within the implementation of this method that we can find the usage of a ReceivePort and a Completer as discussed previously under asynchronous execution. This method also takes care of the memory allocation and deallocation of the array of bytes to be sent to Go. Because it's all inside the go_bridge package, these are all concerns you will no longer need to worry about to implement this approach as long as you use that package.

The Complete Workflow

Putting it all together, we finally get the following workflow:

So, in a nutshell, what the callGoFunc does is:

• Creates a completer that is completed with a message from a ReceivePort

• Serializes the data to be sent with the request to a protobuffer array (and copies that array into a malloc allocated buffer).

• Calls the Go function via FFI, sending it the sendPort of the ReceivePort, the buffer and its length (that is why every single Go function I am exporting has these exact 3 parameters).

• The Go function deserializes the buffer into a Go struct, spawns a go routine, passes it the port and the Go struct containing the data of the request, and returns.

• The Dart method cleans up the memory it previously allocated to store the buffer and awaits the completers.

• In the meantime, when the go routine finishes doing whatever it needs to do, it uses the Dart SDK to send a completion message to the sender port it received.

• When the Dart method's completer is complete, it either returns the result or throws an error if the received message represents an error.

Streams

Some calls to Go may return big results. This may be the case with the find and aggregate methods. For such methods, there is a variant of the callGoFunc method called getGoStream. This method does not return a single callback from Go, but a sequence of callbacks, from which it creates a Stream, thus creating the following workflow:

Conclusion

If we take into account the size of this article, this might seem to be a complex design. However, that apparent complexity might be because of the great level of detail I went through. In practice, it took me a whole lot longer to come up with the ideas and implement the workflow (which is now reusable), than to implement the mongo_go MongoDB driver itself. After having everything set in place, the development of the driver was very speedy.

Hopefully, this approach would be a catalyst for the growth of Dart packages meant to be used on the server by reusing existing Go SDKs.