Dagger : Write your pipelines in your own language

Asutosh PandaAsutosh Panda
17 min read

Initially, it's a little overwhelming to understand another language just to write pipelines whether it's Groovy for Jenkins or YAML for other CI/CD tools. But now we have got another tool that enables you to write your pipeline in your language. It's improving day by day as well as adding more and more languages to its gallery.

The tool is Dagger, a programmable CI/CD engine written in Golang, which runs every pipeline inside a container by default, supports cross-language instrumentation, and every operation cached by default. Let's see how it works:-

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1670680172377/8Cidmu3Oc.png align="middle")

Requirements - Go 1.15 or later and Docker

1. Create a Go module for the tool

controlplane $ docker -v
Docker version 20.10.12, build 20.10.12-0ubuntu2~20.04.1
controlplane $ go version
go version go1.18 linux/amd64
controlplane $ mkdir multibuild
controlplane $ cd multibuild
controlplane $ go mod init multibuild
go: creating new go.mod: module multibuild

2. Create a Dagger client in Go

package main

import (
    "context"
    "fmt"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // initialize Dagger client
    client, err := dagger.Connect(ctx)
    if err != nil {
        return err
    }
    defer client.Close()

    return nil
}

Here we are importing the Dagger SDK. main() is providing arguments to the SDK, build() defines the operations of the pipeline such as :-

  • Creates a Dagger client using dagger.Connect() .

  • This client provides an interface by receiving context(ctx) to execute commands with the help of Dagger Engine.

3. Add the Dagger Go SDK to the module

controlplane $ go get dagger.io/dagger@latest
go: downloading dagger.io/dagger v0.4.2
go: downloading github.com/Khan/genqlient v0.5.0
go: downloading github.com/iancoleman/strcase v0.2.0
go: downloading github.com/vektah/gqlparser/v2 v2.5.1
go: downloading github.com/adrg/xdg v0.4.0
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
go: downloading github.com/hashicorp/errwrap v1.1.0
go: added dagger.io/dagger v0.4.2
go: added github.com/Khan/genqlient v0.5.0
go: added github.com/adrg/xdg v0.4.0
go: added github.com/hashicorp/errwrap v1.1.0
go: added github.com/hashicorp/go-multierror v1.1.1
go: added github.com/iancoleman/strcase v0.2.0
go: added github.com/opencontainers/go-digest v1.0.0
go: added github.com/pkg/errors v0.9.1
go: added github.com/vektah/gqlparser/v2 v2.5.1
go: added golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
go: added golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab

Run go mod tidy to add all the necessary dependencies

controlplane $ go mod tidy
go: downloading github.com/stretchr/testify v1.8.1
go: downloading github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883
go: downloading github.com/sergi/go-diff v1.1.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/agnivade/levenshtein v1.1.1

Then run go run main.go which will give a string output.

controlplane $ go run main.go
Building with Dagger

Though it's written Building with Dagger, it's not building a thing. For that, we have to expand the main.go code.

4. Create a single-build pipeline

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        return err
    }
    defer client.Close()

    // get reference to the local project
    src := client.Host().Directory(".")

    // get `golang` image
    golang := client.Container().From("golang:latest")

    // mount cloned repository into `golang` image
    golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

    // define the application build command
    path := "build/"
    golang = golang.WithExec([]string{"go", "build", "-o", path})

    // get reference to build output directory in container
    output := golang.Directory(path)

    // write contents of container build/ directory to the host
    _, err = output.Export(ctx, path)
    if err != nil {
        return err
    }

    return nil
}

Let's see how the build() works.

  • It uses the client's Host().Directory() method to obtain a reference to the current directory on the host. This reference is stored in the src variable.

  • Using the Container().From() the method initializes a container with golang:latest as a base image and returns a struct.

  • Mounts the filesystem of the repo to the container using WithMountedDirectory() .

  • It uses the WithExec() method to define the command to be executed in the container - in this case, the command go build -o PATH, where PATH refers to the build/ directory in the container. The WithExec() method returns a revised Container containing the results of command execution.

  • It obtains a reference to the build/ directory in the container with the Directory() method. It writes the build/ directory from the container to the host using the Directory.Export() method.

controlplane $ go run main.go
Building with Dagger
#1 resolve image config for docker.io/library/golang:latest
#1 DONE 3.3s

#2 mkdir /meta
#2 DONE 0.0s

#3 local:///root/multibuild
#3 transferring /root/multibuild: 13.33kB done
#3 DONE 0.1s

#4 copy / /
#4 DONE 0.0s

#5 docker-image://docker.io/library/golang:latest
#5 resolve docker.io/library/golang:latest
#5 resolve docker.io/library/golang:latest 0.3s done
#5 DONE 0.3s

#5 docker-image://docker.io/library/golang:latest
#5 DONE 0.4s

#5 docker-image://docker.io/library/golang:latest
#5 sha256:2b1c45236c05a077ceb40e15894c0b29065a8368ca5deb6117787fa94691b290 156B / 156B 0.3s done
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 0B / 148.93MB 0.2s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 0B / 85.97MB 0.2s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 0B / 54.59MB 0.2s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 0B / 10.88MB 0.2s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 6.29MB / 85.97MB 0.5s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 9.44MB / 148.93MB 0.8s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 12.58MB / 85.97MB 0.9s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 3.15MB / 54.59MB 0.9s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 6.29MB / 54.59MB 1.1s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 18.87MB / 148.93MB 1.4s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 18.87MB / 85.97MB 1.4s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 9.44MB / 54.59MB 1.4s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 2.10MB / 10.88MB 1.1s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 4.19MB / 10.88MB 1.2s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 14.68MB / 54.59MB 1.7s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 9.80MB / 10.88MB 1.5s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 24.12MB / 85.97MB 1.9s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 10.88MB / 10.88MB 1.7s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 18.87MB / 54.59MB 2.0s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 28.31MB / 148.93MB 2.2s
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 10.88MB / 10.88MB 1.7s done
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 30.41MB / 85.97MB 2.3s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 23.07MB / 54.59MB 2.3s
#5 sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 0B / 5.16MB 0.2s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 36.70MB / 148.93MB 2.6s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 36.70MB / 85.97MB 2.6s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 27.26MB / 54.59MB 2.7s
#5 sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 4.19MB / 5.16MB 0.6s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 30.41MB / 54.59MB 3.0s
#5 sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 5.16MB / 5.16MB 0.8s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 45.09MB / 148.93MB 3.2s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 45.09MB / 85.97MB 3.2s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 34.60MB / 54.59MB 3.2s
#5 sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 5.16MB / 5.16MB 0.8s done
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 0B / 55.04MB 0.2s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 38.80MB / 54.59MB 3.5s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 54.53MB / 148.93MB 3.8s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 50.33MB / 85.97MB 3.6s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 4.19MB / 55.04MB 0.6s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 41.94MB / 54.59MB 3.8s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 55.57MB / 85.97MB 4.1s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 49.28MB / 54.59MB 4.1s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 62.91MB / 148.93MB 4.4s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 54.53MB / 54.59MB 4.4s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 9.44MB / 55.04MB 1.4s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 61.87MB / 85.97MB 4.6s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 12.58MB / 55.04MB 1.5s
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 54.59MB / 54.59MB 4.6s done
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 72.35MB / 148.93MB 5.0s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 67.11MB / 85.97MB 5.0s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 16.78MB / 55.04MB 1.8s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 23.76MB / 55.04MB 2.1s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 81.79MB / 148.93MB 5.6s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 77.59MB / 85.97MB 5.6s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 28.31MB / 55.04MB 2.4s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 83.89MB / 85.97MB 5.9s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 33.55MB / 55.04MB 2.9s
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 85.97MB / 85.97MB 6.0s done
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 36.70MB / 55.04MB 3.0s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 91.23MB / 148.93MB 6.2s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 42.99MB / 55.04MB 3.2s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 47.19MB / 55.04MB 3.4s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 102.76MB / 148.93MB 6.6s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 54.53MB / 55.04MB 3.6s
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 55.04MB / 55.04MB 3.6s done
#5 extracting sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 114.29MB / 148.93MB 7.0s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 125.83MB / 148.93MB 7.2s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 137.36MB / 148.93MB 7.5s
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 148.93MB / 148.93MB 7.8s done
#5 extracting sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 3.6s done
#5 extracting sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037
#5 extracting sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 0.3s done
#5 extracting sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1
#5 extracting sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 0.5s done
#5 extracting sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54
#5 extracting sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 3.3s done
#5 extracting sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487
#5 extracting sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 4.3s done
#5 extracting sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52
#5 extracting sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 8.6s done
#5 extracting sha256:2b1c45236c05a077ceb40e15894c0b29065a8368ca5deb6117787fa94691b290
#5 extracting sha256:2b1c45236c05a077ceb40e15894c0b29065a8368ca5deb6117787fa94691b290 0.1s done
#5 DONE 27.7s

#6 go build -o build/
#6 0.357 go: downloading dagger.io/dagger v0.4.2
#6 0.511 go: downloading github.com/Khan/genqlient v0.5.0
#6 0.678 go: downloading github.com/iancoleman/strcase v0.2.0
#6 0.684 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#6 1.035 go: downloading github.com/adrg/xdg v0.4.0
#6 1.054 go: downloading github.com/hashicorp/go-multierror v1.1.1
#6 1.065 go: downloading github.com/opencontainers/go-digest v1.0.0
#6 1.080 go: downloading github.com/pkg/errors v0.9.1
#6 1.090 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#6 1.385 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#6 1.419 go: downloading github.com/hashicorp/errwrap v1.1.0
#6 DONE 4.6s

#2 mkdir /meta
#2 DONE 0.0s

#3 local:///root/multibuild
#3 transferring /root/multibuild: 13.33kB done
#3 DONE 0.1s

#4 copy / /
#4 DONE 0.0s

#5 docker-image://docker.io/library/golang:latest
#5 resolve docker.io/library/golang:latest 0.3s done
#5 sha256:2b1c45236c05a077ceb40e15894c0b29065a8368ca5deb6117787fa94691b290 156B / 156B 0.3s done
#5 sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 148.93MB / 148.93MB 7.8s done
#5 sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 85.97MB / 85.97MB 6.0s done
#5 sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 54.59MB / 54.59MB 4.6s done
#5 sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 10.88MB / 10.88MB 1.7s done
#5 sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 5.16MB / 5.16MB 0.8s done
#5 sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 55.04MB / 55.04MB 3.6s done
#5 extracting sha256:f2f58072e9ed1aa1b0143341c5ee83815c00ce47548309fa240155067ab0e698 3.6s done
#5 extracting sha256:5c8cfbf51e6e6869f1af2a1e7067b07fd6733036a333c9d29f743b0285e26037 0.3s done
#5 extracting sha256:aa3a609d15798d35c1484072876b7d22a316e98de6a097de33b9dade6a689cd1 0.5s done
#5 extracting sha256:094e7d9bb04ebf214ea8dc5a488995449684104ae8ad9603bf061cac0d18eb54 3.3s done
#5 extracting sha256:3eb5927575302d6dc81a3152fd20e63445540caaefa79af119bef50051351487 4.3s done
#5 extracting sha256:f00fd01201faafb60254828f875da2edc8a08fa27114dea9d17410dcd9ae5e52 8.6s done
#5 extracting sha256:2b1c45236c05a077ceb40e15894c0b29065a8368ca5deb6117787fa94691b290 0.1s done
#5 DONE 27.7s

#6 go build -o build/
#6 0.357 go: downloading dagger.io/dagger v0.4.2
#6 0.511 go: downloading github.com/Khan/genqlient v0.5.0
#6 0.678 go: downloading github.com/iancoleman/strcase v0.2.0
#6 0.684 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#6 1.035 go: downloading github.com/adrg/xdg v0.4.0
#6 1.054 go: downloading github.com/hashicorp/go-multierror v1.1.1
#6 1.065 go: downloading github.com/opencontainers/go-digest v1.0.0
#6 1.080 go: downloading github.com/pkg/errors v0.9.1
#6 1.090 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#6 1.385 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#6 1.419 go: downloading github.com/hashicorp/errwrap v1.1.0
#6 DONE 4.6s

#7 copy /build /
#7 DONE 0.0s

#8 exporting to client
#8 copying files 2.43MB 0.1s

Now the build artifact should be visible

controlplane $ tree build
build
`-- multibuild

5. Create a multi-build pipeline

Now, we will extend the same main.go file to provide more features to the pipeline.

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // define build matrix
    oses := []string{"linux", "darwin"}
    arches := []string{"amd64", "arm64"}

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        return err
    }
    defer client.Close()

    // get reference to the local project
    src := client.Host().Directory(".")

    // create empty directory to put build outputs
    outputs := client.Directory()

    // get `golang` image
    golang := client.Container().From("golang:latest")

    // mount cloned repository into `golang` image
    golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

    for _, goos := range oses {
        for _, goarch := range arches {
            // create a directory for each os and arch
            path := fmt.Sprintf("build/%s/%s/", goos, goarch)

            // set GOARCH and GOOS in the build environment
            build := golang.WithEnvVariable("GOOS", goos)
            build = build.WithEnvVariable("GOARCH", goarch)

            // build application
            build = build.WithExec([]string{"go", "build", "-o", path})

            // get reference to build output directory in container
            outputs = outputs.WithDirectory(path, build.Directory(path))
        }
    }
    // write build artifacts to host
    _, err = outputs.Export(ctx, ".")
    if err != nil {
        return err
    }

    return nil
}

Here we will be building the pipeline using 2 different OS and architectures.

  • build() defines the build matrix, consisting of two OSs (darwin and linux) and two architectures (amd64 and arm64).

  • It iterates over this matrix, building the Go application for each combination. The Go build process is instructed via the GOOS and GOARCH build variables, which are reset for each case via the Container.WithEnvVariable() method.

  • It creates an output directory on the host named for each OS/architecture combination so that the build outputs can be differentiated.

The build process will run 4 times and after that, the build results will be written back to the host. Along with container mechanism, it will be running the pipelines in parallel which makes it even more faster.

controlplane $ vi main.go 
controlplane $ go run main.go
Building with Dagger
#1 resolve image config for docker.io/library/golang:latest
#1 DONE 1.6s

#2 mkdir /meta
#2 CACHED

#3 local:///root/multibuild
#3 transferring /root/multibuild: 3.48MB 0.1s
#3 ...

#4 docker-image://docker.io/library/golang:latest
#4 resolve docker.io/library/golang:latest 0.3s done
#4 DONE 0.3s

#3 local:///root/multibuild
#3 transferring /root/multibuild: 8.84MB 0.3s done
#3 DONE 0.3s

#4 docker-image://docker.io/library/golang:latest
#4 CACHED

#5 copy / /
#5 DONE 0.0s

#6 go build -o build/linux/amd64/
#6 0.841 go: downloading dagger.io/dagger v0.4.2
#6 1.124 go: downloading github.com/Khan/genqlient v0.5.0
#6 1.441 go: downloading github.com/iancoleman/strcase v0.2.0
#6 1.459 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#6 2.618 go: downloading github.com/adrg/xdg v0.4.0
#6 2.672 go: downloading github.com/hashicorp/go-multierror v1.1.1
#6 2.742 go: downloading github.com/opencontainers/go-digest v1.0.0
#6 2.770 go: downloading github.com/pkg/errors v0.9.1
#6 2.811 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#6 4.255 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#6 4.629 go: downloading github.com/hashicorp/errwrap v1.1.0
#6 ...

#7 go build -o build/darwin/amd64/
#0 0.740 go: downloading dagger.io/dagger v0.4.2
#0 1.126 go: downloading github.com/Khan/genqlient v0.5.0
#0 1.453 go: downloading github.com/iancoleman/strcase v0.2.0
#0 1.475 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 2.624 go: downloading github.com/adrg/xdg v0.4.0
#0 2.699 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 2.715 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 2.765 go: downloading github.com/pkg/errors v0.9.1
#0 2.806 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 4.100 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 4.305 go: downloading github.com/hashicorp/errwrap v1.1.0
#7 ...

#8 go build -o build/darwin/arm64/
#0 0.762 go: downloading dagger.io/dagger v0.4.2
#0 1.123 go: downloading github.com/Khan/genqlient v0.5.0
#0 1.533 go: downloading github.com/iancoleman/strcase v0.2.0
#0 1.544 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 2.620 go: downloading github.com/adrg/xdg v0.4.0
#0 2.711 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 2.739 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 2.776 go: downloading github.com/pkg/errors v0.9.1
#0 2.814 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 4.135 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 4.367 go: downloading github.com/hashicorp/errwrap v1.1.0
#8 ...

#9 go build -o build/linux/arm64/
#0 0.765 go: downloading dagger.io/dagger v0.4.2
#0 1.128 go: downloading github.com/Khan/genqlient v0.5.0
#0 1.472 go: downloading github.com/iancoleman/strcase v0.2.0
#0 1.500 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 2.621 go: downloading github.com/adrg/xdg v0.4.0
#0 2.703 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 2.750 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 2.774 go: downloading github.com/pkg/errors v0.9.1
#0 2.809 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 3.869 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 4.175 go: downloading github.com/hashicorp/errwrap v1.1.0
#9 ...

#6 go build -o build/linux/amd64/
#6 DONE 19.8s

#7 go build -o build/darwin/amd64/
#7 ...

#9 go build -o build/linux/arm64/
#9 DONE 109.1s

#7 go build -o build/darwin/amd64/
#7 ...

#8 go build -o build/darwin/arm64/
#8 DONE 109.5s

#7 go build -o build/darwin/amd64/
#7 DONE 109.7s

#2 mkdir /meta
#2 CACHED

#4 docker-image://docker.io/library/golang:latest
#4 resolve docker.io/library/golang:latest 0.3s done
#4 CACHED

#3 local:///root/multibuild
#3 transferring /root/multibuild: 8.84MB 0.3s done
#3 DONE 0.3s

#5 copy / /
#5 DONE 0.0s

#6 go build -o build/linux/amd64/
#6 0.841 go: downloading dagger.io/dagger v0.4.2
#6 1.124 go: downloading github.com/Khan/genqlient v0.5.0
#6 1.441 go: downloading github.com/iancoleman/strcase v0.2.0
#6 1.459 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#6 2.618 go: downloading github.com/adrg/xdg v0.4.0
#6 2.672 go: downloading github.com/hashicorp/go-multierror v1.1.1
#6 2.742 go: downloading github.com/opencontainers/go-digest v1.0.0
#6 2.770 go: downloading github.com/pkg/errors v0.9.1
#6 2.811 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#6 4.255 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#6 4.629 go: downloading github.com/hashicorp/errwrap v1.1.0
#6 DONE 19.8s

#9 go build -o build/linux/arm64/
#9 0.765 go: downloading dagger.io/dagger v0.4.2
#9 1.128 go: downloading github.com/Khan/genqlient v0.5.0
#9 1.472 go: downloading github.com/iancoleman/strcase v0.2.0
#9 1.500 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#9 2.621 go: downloading github.com/adrg/xdg v0.4.0
#9 2.703 go: downloading github.com/hashicorp/go-multierror v1.1.1
#9 2.750 go: downloading github.com/opencontainers/go-digest v1.0.0
#9 2.774 go: downloading github.com/pkg/errors v0.9.1
#9 2.809 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#9 3.869 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#9 4.175 go: downloading github.com/hashicorp/errwrap v1.1.0
#9 DONE 109.1s

#8 go build -o build/darwin/arm64/
#8 0.762 go: downloading dagger.io/dagger v0.4.2
#8 1.123 go: downloading github.com/Khan/genqlient v0.5.0
#8 1.533 go: downloading github.com/iancoleman/strcase v0.2.0
#8 1.544 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#8 2.620 go: downloading github.com/adrg/xdg v0.4.0
#8 2.711 go: downloading github.com/hashicorp/go-multierror v1.1.1
#8 2.739 go: downloading github.com/opencontainers/go-digest v1.0.0
#8 2.776 go: downloading github.com/pkg/errors v0.9.1
#8 2.814 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#8 4.135 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#8 4.367 go: downloading github.com/hashicorp/errwrap v1.1.0
#8 DONE 109.5s

#7 go build -o build/darwin/amd64/
#7 0.740 go: downloading dagger.io/dagger v0.4.2
#7 1.126 go: downloading github.com/Khan/genqlient v0.5.0
#7 1.453 go: downloading github.com/iancoleman/strcase v0.2.0
#7 1.475 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#7 2.624 go: downloading github.com/adrg/xdg v0.4.0
#7 2.699 go: downloading github.com/hashicorp/go-multierror v1.1.1
#7 2.715 go: downloading github.com/opencontainers/go-digest v1.0.0
#7 2.765 go: downloading github.com/pkg/errors v0.9.1
#7 2.806 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#7 4.100 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#7 4.305 go: downloading github.com/hashicorp/errwrap v1.1.0
#7 DONE 109.7s

#10 copy /build/linux/amd64 /build/linux/amd64
#10 DONE 0.0s

#11 copy /build/linux/arm64 /build/linux/arm64
#11 DONE 0.1s

#12 copy /build/darwin/amd64 /build/darwin/amd64
#12 DONE 0.1s

#13 copy /build/darwin/arm64 /build/darwin/arm64
#13 DONE 0.1s

#14 exporting to client
#14 copying files 3.15MB 0.1s
#14 copying files 34.46MB 1.2s done
#14 DONE 1.2s

Now we can check the tree again for the folder structure.

controlplane $ tree build
build
|-- darwin
|   |-- amd64
|   |   `-- multibuild
|   `-- arm64
|       `-- multibuild
|-- linux
|   |-- amd64
|   |   `-- multibuild
|   `-- arm64
|       `-- multibuild
`-- multibuild

6. Create builds targeting multiple Go versions

Modify the main.go file again to provide different versions of Go.

controlplane $ go run main.go
Building with Dagger
#1 resolve image config for docker.io/library/golang:1.19
#1 DONE 1.6s

#2 mkdir /meta
#2 CACHED

#3 local:///root/multibuild
#3 transferring /root/multibuild: 25B
#3 ...

#4 docker-image://docker.io/library/golang:1.19
#4 resolve docker.io/library/golang:1.19 0.3s done
#4 CACHED

#3 local:///root/multibuild
#3 ...

#5 resolve image config for docker.io/library/golang:1.18
#5 DONE 2.8s

#3 local:///root/multibuild
#3 transferring /root/multibuild: 34.46MB 1.4s done
#3 DONE 1.4s

#6 copy / /
#6 DONE 0.2s

#7 docker-image://docker.io/library/golang:1.18
#7 resolve docker.io/library/golang:1.18 0.3s done
#7 DONE 0.4s

#8 go build -o build/1.19/darwin/arm64/
#8 1.495 go: downloading dagger.io/dagger v0.4.2
#8 3.102 go: downloading github.com/Khan/genqlient v0.5.0
#8 4.278 go: downloading github.com/iancoleman/strcase v0.2.0
#8 4.304 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#8 ...

#7 docker-image://docker.io/library/golang:1.18
#7 sha256:3a06c91201dd22ae0ea7eef5937fffadda06ced8d46c724c3a956db8a2f3e72d 141.97MB / 141.97MB 4.3s done
#7 sha256:8419c568f921e324ce8c9d399c64311b937ebf1448230610782e2e083c1950ba 156B / 156B 0.3s done
#7 extracting sha256:3a06c91201dd22ae0ea7eef5937fffadda06ced8d46c724c3a956db8a2f3e72d
#7 ...

#8 go build -o build/1.19/darwin/arm64/
#8 7.602 go: downloading github.com/adrg/xdg v0.4.0
#8 7.811 go: downloading github.com/hashicorp/go-multierror v1.1.1
#8 7.876 go: downloading github.com/opencontainers/go-digest v1.0.0
#8 7.998 go: downloading github.com/pkg/errors v0.9.1
#8 8.260 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#8 ...

#9 go build -o build/1.19/darwin/amd64/
#0 1.704 go: downloading dagger.io/dagger v0.4.2
#0 3.102 go: downloading github.com/Khan/genqlient v0.5.0
#0 3.972 go: downloading github.com/iancoleman/strcase v0.2.0
#0 4.055 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 7.610 go: downloading github.com/adrg/xdg v0.4.0
#0 7.702 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 7.772 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 7.890 go: downloading github.com/pkg/errors v0.9.1
#0 7.954 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#9 ...

#10 go build -o build/1.19/linux/arm64/
#0 1.414 go: downloading dagger.io/dagger v0.4.2
#0 3.151 go: downloading github.com/Khan/genqlient v0.5.0
#0 4.382 go: downloading github.com/iancoleman/strcase v0.2.0
#0 4.462 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 7.609 go: downloading github.com/adrg/xdg v0.4.0
#0 7.709 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 7.792 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 7.863 go: downloading github.com/pkg/errors v0.9.1
#0 7.957 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#10 ...

#11 go build -o build/1.19/linux/amd64/
#0 1.494 go: downloading dagger.io/dagger v0.4.2
#0 3.245 go: downloading github.com/Khan/genqlient v0.5.0
#0 4.343 go: downloading github.com/iancoleman/strcase v0.2.0
#0 4.360 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 7.609 go: downloading github.com/adrg/xdg v0.4.0
#0 7.702 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 7.784 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 7.889 go: downloading github.com/pkg/errors v0.9.1
#0 7.952 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#11 ...

#10 go build -o build/1.19/linux/arm64/
#10 11.37 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#10 12.42 go: downloading github.com/hashicorp/errwrap v1.1.0
#10 ...

#11 go build -o build/1.19/linux/amd64/
#11 12.05 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#11 12.69 go: downloading github.com/hashicorp/errwrap v1.1.0
#11 ...

#7 docker-image://docker.io/library/golang:1.18
#7 extracting sha256:3a06c91201dd22ae0ea7eef5937fffadda06ced8d46c724c3a956db8a2f3e72d 13.8s done
#7 extracting sha256:8419c568f921e324ce8c9d399c64311b937ebf1448230610782e2e083c1950ba 0.0s done
#7 DONE 18.5s

#12 go build -o build/1.18/darwin/amd64/
#12 0.995 go: downloading dagger.io/dagger v0.4.2
#12 1.572 go: downloading github.com/Khan/genqlient v0.5.0
#12 2.117 go: downloading github.com/iancoleman/strcase v0.2.0
#12 2.138 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#12 3.317 go: downloading github.com/adrg/xdg v0.4.0
#12 3.414 go: downloading github.com/hashicorp/go-multierror v1.1.1
#12 ...

#8 go build -o build/1.19/darwin/arm64/
#8 11.64 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#8 12.50 go: downloading github.com/hashicorp/errwrap v1.1.0
#8 ...

#9 go build -o build/1.19/darwin/amd64/
#9 11.75 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#9 12.60 go: downloading github.com/hashicorp/errwrap v1.1.0
#9 ...

#12 go build -o build/1.18/darwin/amd64/
#12 3.507 go: downloading github.com/opencontainers/go-digest v1.0.0
#12 3.576 go: downloading github.com/pkg/errors v0.9.1
#12 3.628 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#12 5.807 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#12 6.157 go: downloading github.com/hashicorp/errwrap v1.1.0
#12 ...

#13 go build -o build/1.18/darwin/arm64/
#0 1.015 go: downloading dagger.io/dagger v0.4.2
#0 1.483 go: downloading github.com/Khan/genqlient v0.5.0
#0 2.143 go: downloading github.com/iancoleman/strcase v0.2.0
#0 2.172 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 3.133 go: downloading github.com/adrg/xdg v0.4.0
#0 3.206 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 3.260 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 3.343 go: downloading github.com/pkg/errors v0.9.1
#0 3.406 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 6.268 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 6.400 go: downloading github.com/hashicorp/errwrap v1.1.0
#13 ...

#14 go build -o build/1.18/linux/arm64/
#0 1.016 go: downloading dagger.io/dagger v0.4.2
#0 1.485 go: downloading github.com/Khan/genqlient v0.5.0
#0 2.005 go: downloading github.com/iancoleman/strcase v0.2.0
#0 2.042 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 3.197 go: downloading github.com/adrg/xdg v0.4.0
#0 3.297 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 3.362 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 3.469 go: downloading github.com/pkg/errors v0.9.1
#0 3.544 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 5.658 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 5.969 go: downloading github.com/hashicorp/errwrap v1.1.0
#14 ...

#15 go build -o build/1.18/linux/amd64/
#0 0.985 go: downloading dagger.io/dagger v0.4.2
#0 1.522 go: downloading github.com/Khan/genqlient v0.5.0
#0 2.045 go: downloading github.com/iancoleman/strcase v0.2.0
#0 2.072 go: downloading github.com/vektah/gqlparser/v2 v2.5.1
#0 3.135 go: downloading github.com/adrg/xdg v0.4.0
#0 3.203 go: downloading github.com/hashicorp/go-multierror v1.1.1
#0 3.257 go: downloading github.com/opencontainers/go-digest v1.0.0
#0 3.348 go: downloading github.com/pkg/errors v0.9.1
#0 3.387 go: downloading golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
#0 5.727 go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
#0 5.973 go: downloading github.com/hashicorp/errwrap v1.1.0
#15 ...

We can see the new artifact folder structure now

controlplane $ tree build
build/
├── 1.18
│   ├── darwin
│   │   ├── amd64
│   │   │   └── multibuild
│   │   └── arm64
│   │       └── multibuild
│   └── linux
│       ├── amd64
│       │   └── multibuild
│       └── arm64
│           └── multibuild
└── 1.19
    ├── darwin
    │   ├── amd64
    │   │   └── multibuild
    │   └── arm64
    │       └── multibuild
    └── linux
        ├── amd64
        │   └── multibuild
        └── arm64
            └── multibuild

If you want to play around then use

Thanks for reading upto here.

More about me - poly.me/measutosh

0
Subscribe to my newsletter

Read articles from Asutosh Panda directly inside your inbox. Subscribe to the newsletter, and don't miss out.

CI/CDDevopsgolangdagger

Written by

Asutosh Panda
Asutosh Panda

Curious about Outages, Distributed Systems, DevOps, Backend, SRE related stuffs.