Building Golang API to Run Untrusted Code in a Sandbox

Run arbitrary Go code inside a sandbox container with real-time output from the gRPC server streaming API

Try it out:https://github.com/nirdosh17/go-sandbox

Design

Gif description

Request Flow

When we receive a request in the gRPC endpoint(Go service):

An available sandbox is reserved. If session_id is present, it will map the session_id to a sandbox and the same sandbox will be used to execute the code for this session until it expires or is cleaned up.
Sandbox is marked as busy after reserving.
Then the code is executed. logs, errors, exit code are recorded.
stdout/stderr is streamed to the client.
Temp files are cleaned up.
Sandbox is freed up and marked as available .

Components

Go Service

A Go service runs as a container and exposes an API where we send an arbitrary code to execute.
```
  {
    "code": "<code>",
    "session_id": "uuid_1"
  }
```
The Go service is a gRPC Server Streaming API that gives a real-time output of executing code simulating the local execution.

2. Sandbox

When the Go service boots up, multiple sandboxes are created inside the container using Isolate. It provides a CLI to execute commands in a restricted isolated environment.
Isolate provides different options to limit access to the resources(network, processes, files, etc) inside the sandbox and the host container.
Let’s say we create 10 sandboxes. For complete isolation, one sandbox serves a single request at a time. So, it has two states: busy | available
We keep track of all sandboxes and their availability in a thread-safe cache.
Files created in one sandbox are not visible in other sandboxes. But, cleaning them up is our responsibility.

3. Sandbox Manager

Maintains a list of reserved(running) and available sandboxes.
Tracks user/session_id and sandbox reserved for the session. So, that when we rerun the code, it runs on the same sandbox.
Cleans up and re-initializes sandboxes when a threshold is reached, such as when they have not been used for a long time or have expired.

// sandbox
{
  "reserved": { 
    "user_id": {
      "box_id": "<int>",
      "lastUsed": "<timestamp>",
      "expiresAt": "<timestamp>",
    }
  },
  "available": ["box_id_1", "box_id_2"]
}

Setting up Isolate Package

Building the Image

Here is the Dockerfile which will set up isolate CLI in the container and Go service run cmd/exec with appropriate parameters.

 FROM ubuntu:24.04

 RUN apt update -y
 RUN apt install wget tar gzip git -y

 # install dependecies and initialize isolate sandbox
 RUN apt install build-essential libcap-dev pkg-config libsystemd-dev -y

 RUN wget -P /tmp https://github.com/ioi/isolate/archive/master.tar.gz && tar -xzvf /tmp/master.tar.gz -C / > /dev/null
 RUN make -C /isolate-master isolate
 ENV PATH="/isolate-master:$PATH"

 # you can copy this default config from: 
 # https://github.com/ioi/isolate/blob/master/default.cf
 COPY ./go-sandbox/isolate/default.cf /usr/local/etc/isolate

 # creates sandbox 0
 # other sandboxes are initialized and managed by the Go service
 RUN isolate --init

 # add installation steps for your Go service e.g. mod install, build ...

 EXPOSE 8000
 CMD ["./cmd/main"]

To check if everything is running, build and run the image and ssh into the container. If you enter isolate command in bash, the command should print out the manual page.

Running a Go file inside the sandbox

SSH into the container and create a sample Go code in a directory e.g. /code/test.go .

To run this using isolate, we need to set the following flags:
- --box-id=1 : To run commands in a specific sandbox. Default: 0
- --dir=/code/test.go : Makes this directory from the host machine visible inside the sandbox process
- --processes=100 : Enable multiple processes inside the sandbox. If not done you will get this error:
```
  runtime: failed to create new OS thread (have 2 already; errno=11)
  runtime: may need to increase max user processes (ulimit -u)
  fatal error: newosproc
```
- --open-files=0 : Specify a higher limit or set it to unlimited(=0) Otherwise, you will reach an open file limit in the sandbox as shown below if you make frequent requests:
```
  go: error obtaining buildID for go tool compile:
  fork/exec /usr/local/go/pkg/tool/linux_arm64/compile: too many open files
```
- Mount Go build cache directory with rwaccess:
- ```
    --env=HOME --dir=/root/.cache/go-build
```
  Otherwise, you will get the following error:
```
  build cache is required, but could not be located:
  GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined

  # Instead of defining HOME dir and mounting build cache dir,
  # you can just specify cache dir --env=GOCACHE=/tmp which will be only present
  # inside the sandbox
  # But in this approach, the sandbox size can increase on subsequent builds.
  # If sandbox size is unpredictable, we cannot specify filesize limit effectively.
```
- Optional configs:
  - --fsize=5120 : max file size that can be created in the sandbox
  - --wait : waits instead of throwing an error immediately when a sandbox is busy running another command

Finally, Running the code with all configs:

    $ isolate --box-id=0 \
    --processes=100 --dir=/code \
    --env=HOME --dir=/root/.cache/go-build \
    --open-files=0 --fsize=5120 --wait \
    --run -- /usr/local/go/bin/go run /code/test.go

    Hello World!
    OK (0.191 sec real, 0.104 sec wall)

Executing isolate via a Go service and streaming results

 cmd := exec.CommandContext(ctx,
  "isolate",
  fmt.Sprintf("--box-id=%v", boxId),
  "--fsize=5120",
  "--dir=/code",
  "--dir=/root/.cache/go-build:rw",
  "--wait",
  "--processes=100",
  "--open-files=0",
  "--env=GOROOT",
  "--env=GOPATH",
  "--env=GO111MODULE=on",
  "--env=HOME",
  "--env=PATH",
  // log package write to stderr which is not forwarded to Go exec. 
  // so, it is better to write all stderr to stdout for out case
  "--stderr-to-stdout",
  "--run",
  "--",
  "/usr/local/go/bin/go",
  "run",
  fmt.Sprintf("/code/%v.go", codeID),
 )

 cmd.WaitDelay = 60 * time.Second

 var stderr bytes.Buffer
 cmd.Stderr = &stderr

 stdoutpipe, err := cmd.StdoutPipe()
 if err != nil {
  stream.Send(&pb.RunResponse{Err: err})
 }

 err = cmd.Start()
 if err != nil {
  stream.Send(&pb.RunResponse{Err: err})
 }

 scanner := bufio.NewScanner(stdoutpipe)
 for scanner.Scan() {
 // streaming output to the API client
 stream.Send(&pb.RunResponse{Output: scanner.Text()})
 }

 err = cmd.Wait()
 if err != nil {
  stream.Send(&pb.RunResponse{Err: stderr.String()}) 
 }

Trying the API from gRPC client

Using a gRPC client like Postman, we can execute a Go code and receive real-time output as shown below:

Try it yourself:https://github.com/nirdosh17/go-sandbox

Parting Thoughts

This post was mainly about Go but we can install appropriate packages in the container, and supply appropriate parameters to isolate and run any programming language.

The sandbox is a work in progress and will be updated as I fix other execution and security issues. It’s a great way to build a playground like go.dev/play and learn more about how sandboxes and processes are isolated in Linux.

I have not explored control groups yet which allows fine-grain control over resources like CPU, Network, Disk I/O etc.

Although outgoing network calls are restricted by default, I need to explore how to allow local connections inside the sandbox. For example, we can execute a code that creates a web server in a Go routine and requests it from the main Go routine.

Building Golang API to Run Untrusted Code in Sandbox

Table of contents