Embed Static Files in Go Binaries With pkger

One of the best parts of Go is that compiled programs can be distributed and executed as a single dependency-free binary file. Unfortunately, programs requiring access to static files for thing like configuration, web server assets, or database migrations must be distributed alongside those static files, eliminating the single-file benefit. Fortunately, there's a way to bundle these assets inside the binary itself.

This post demonstrates how to embed static assets inside Go binaries using the pkger module.

How pkger works#

Go doesn't actually provide a way to embed static files inside the compiled binary—although there is a draft design in progress—so how is it possible at all? Well, Go does provide a mechanism that can be used to get the functionality we want, and it should look very familiar to you…

// MyFiles maps name->content of each static file
var MyFiles = map[string]string{
    "styles.css": `body {\n  background-color: lightPink;\n }`,
    "main.js":    `console.log('Hello World!')`,
}

That's right… variables. The way to embed static files in Go is to store them in the source code. Now, the above example is definitely not convenient, but it would work for storing a few text-based files that don't change often. We can do better, though.

pkger applies a similar strategy, but uses code generation to automate the process. So, instead of copying file contents into the source manually, pkger scans the codebase for usages of pkger's functions (eg. pkger.Open(path)) and generates a pkged.go file containing the necessary assets.

TIP: If the pkged.go file has not present, pkger will fallback to reading from the filesystem, which is ideal for making frequent changes during development.

Generating pkged.go source file#

Now that we know the mechanism pkger uses to embed static files, we can start using it.

First, install the pkger CLI using the following command.

go get github.com/markbates/pkger/cmd/pkger

Next, create two files: main.go that prints the contents of a file, and example.txt to act as our static asset.

package main

import (
    "fmt"
    "github.com/markbates/pkger"
    "io/ioutil"
)

func main() {
    // pkger will discover that we need example.txt and embed it
    f, err := pkger.Open("/example.txt")
    if err != nil {
        panic(err)
    }

    // Read and print contents of example.txt
    contents, err := ioutil.ReadAll(f)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", contents)
}

Example static asset

Next, run pkger from your project root to generate the pkged.go source file. To check what's happening, we can also run pkger list to preview the assets that will be embedded.

foo@bar:~$ pkger list
demo
 > demo:/example.txt
foo@bar:~$ pkger

NOTE: You may need to run go mod vendor after running pkger to fetch the additional dependencies imported by pkged.go

If successful, you should now see a pkged.go containing encoded versions of your static assets. Note, it won't be human-readable.

// Code generated by pkger; DO NOT EDIT.

// +build !skippkger

package main

import (
    "github.com/markbates/pkger"
    "github.com/markbates/pkger/pkging/mem"
)

var _ = pkger.Apply(mem.UnmarshalEmbed([]byte(`...`)))

If we did everything correctly, we can now run our program with go run main.go and watch it print the contents of the file to the screen. It should have read the contents from pkged.go instead of the filesystem but how can we know for sure?

To verify the behavior, make a change to example.txt and re-run the program. It should print the old contents because it read the bundled one from pkged.go. Now delete pkged.go and try again. You'll now see that pkger fell back to reading the up-to-date contents directly from the filesystem.

Let's summarize what we've learned so far:

• Use the pkger file access APIs to read files and directories
• Run pkger CLI to embed assets within pkged.go source file
• Without pkged.go, pkger will read from the filesystem

Example usages of pkger#

pkger can do a lot more than our example of reading a single file. Here are a few common operations to showcase real-world usage. For a full list, visit the pkger docs.

Open a configuration file#

The following example opens and parses a config.json file.

package main

import (
    "encoding/json"
    "fmt"
    "github.com/markbates/pkger"
)

func main() {
    // Open config file
    f, err := pkger.Open("/config.json")
    if err != nil {
        panic(err)
    }

    // Parse and print config file
    var config map[string]string
    d := json.NewDecoder(f)
    err = d.Decode(&config)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%#v\n", config)
}

List files in a directory#

The following example lists the names of all files located in the project root.

package main

import (
    "github.com/markbates/pkger"
)

func main() {
    // Open project root
    f, err := pkger.Open("/")
    if err != nil {
        panic(err)
    }

    // Print names of all files
    files, err := f.Readdir(0)
    for _, file := range files {
        println(file.Name())
    }
}

Serve static files#

The following example serves a directory of static assets.

package main

import (
    "github.com/markbates/pkger"
    "io"
    "net/http"
    "os"
)


func main() {
    // NOTE: Since file access below is dynamic, pkger
    // can't figure out which assets to bundle. Using 
    // the Include method provides an explicit hint to
    // pkger to bundle everything in /static.
    pkger.Include("/static")

    handler := http.NewServeMux()

    // Navigate to http://localhost:9000/static/<FILE_NAME>
    // to check if it's working.
    handler.HandleFunc("/static/", func(w http.ResponseWriter, r *http.Request) {
        println("Accessing", r.URL.Path)
        file, err := pkger.Open(r.URL.Path)
        if err == os.ErrNotExist {
            http.NotFound(w, r)
            return
        } else if err != nil {
            http.Error(w, "Something bad happened", http.StatusInternalServerError)
            println(err)
            return
        }

        io.Copy(w, file)
    })

    http.ListenAndServe(":9000", handler)
}

Wrap-Up#

Using pkger to embed static assets in Go programs is great because it restores the ability to distribute Go programs as a single binary file. This post covered the basic usage of pkger but there's still more to learn. I recommend taking a look at the pkger README to learn more.