Using the Brave Search API with Go

Recently, I wanted to build a small program to search things through the Brave API. Why? Well, I wanted to begin to understand how APIs work and, two, how search engines work. So, I set out to set up my brave account and make an API key. This is written here if you wish to follow along. https://tinyurl.com/mrxz873w

Saying Hello to the Brave API

I first tried the curl command to see how the API worked. (Just a side note for the sake of this blog: the command is on two lines to make the curl command work: move the -H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE" up a level to make the command all one line.)

curl -s --compressed "https://api.search.brave.com/res/v1/web/search?q=brave+search" -H "Accept: application/json"
    -H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE"

As you can see, the Headers are not the normal restful headers when using an API. For example, most restful headers accept the following Bearer format when working with a token. -H "Authorization: Bearer <ACCESS_TOKEN>"

Using this Curl command will only get you authenticated to the API and will return the following: (This is just a snippet of what is returned; the whole JSON is very long and would make this a very long article.)

{"query":{"original":"brave search","show_strict_warning":false,"is_navigational":true,"local_decision":"drop","local_locations_idx":0,"is_news_breaking":false,"spellcheck_off":true,"country":"us","bad_results":false,"should_fallback":false,"postal_code":"","city":"","header_country":"","more_results_available":true,"state":""},"mixed":{"type":"mixed","main":[{"type":"web","index":0,"all":false},{"type":"web","index":1,"all":false},{"type":"videos","all":true},{"type":"web","index":2,"all":false},{"type":"web","index":3,"all":false},{"type":"web","index":4,"all":false},{"type":"web","index":5,"all":false},{"type":"web","index":6,"all":false},{"type":"web","index":7,"all":false},{"type":"web","index":8,"all":false},{"type":"web","index":9,"all":false},{"type":"web","index":10,"all":false},{"type":"web","index":11,"all":false},{"type":"web","index":12,"all":false},{"type":"web","index":13,"all":false},{"type":"web","index":14,"all":false},{"type":"web","index":15,"all":false},{"type":"web","index":16,"all":false},{"type":"web","index":17,"all":false},{"type":"web","index":18,"all":false},{"type":"web","index":19,"all":false}],"top":[],"side":[]},"type":"search","videos":{"type":"videos","results":[{"type":"video_result","url":"https://www.youtube.com/watch?v=gwho1EuwkRQ","title":"How To Change The Default Search Engine In The Brave ...","description":"How To Change The Default Search Engine In The Brave Web Browser | PC | *2022*This is a video tutorial on how to change the default search engine in the Brav...","age":"September 20, 2022","video":{},"meta_url":{"scheme":"https","netloc":"youtube.com","hostname":"www.youtube.com","favicon":"https://imgs.search.brave.com/Ux4Hee4evZhvjuTKwtapBycOGjGDci2Gvn2pbSzvbC0/rs:fit:32:32:1/g:ce/aHR0cDovL2Zhdmlj/b25zLnNlYXJjaC5i/cmF2ZS5jb20vaWNv/bnMvOTkyZTZiMWU3/YzU3Nzc5YjExYzUy/N2VhZTIxOWNlYjM5/ZGVjN2MyZDY4Nzdh/ZDYzMTYxNmI5N2Rk/Y2Q3N2FkNy93d3cu/eW91dHViZS5jb20v","path":"› watch"},"thumbnail":{"src":"https://imgs.search.brave.com/6g23ttPyzgvCVreUZ2pcnuTnGAhNHt8yQLyVNL3mlsc/rs:fit:200:200:1/g:ce/aHR0cHM6Ly9pLnl0/aW1nLmNvbS92aS9n/d2hvMUV1d2tSUS9t/YXhyZXNkZWZhdWx0/LmpwZz9zcXA9LW9h/eW13RW1DSUFLRU5B/RjhxdUtxUU1hOEFF/Qi1BSC1DWUFDMEFX/S0Fnd0lBQkFCR0Vn/Z0V5aF9NQTg9JmFt/cDtycz1BT240Q0xB/dzdIUkZsVTd1OUZT/YUR2WERZSE1TNnNj/U0p3"}},{"type":"video_result","url":"https://www.youtube.com/watch?v=ux4VacF3hxc","title":"Brave Browser And Search Engine Keep Getting Better - YouTube",

Lets Get Searching With Go

As you can see, we have yet to get searching. For that, we turn to Go.

In Go, you can use specific packages to interact with APIs. They are: "encoding/json", "net/http", and "net/url." There are a few others we are going to use, but they are just there to make this all work. I won't go into detail about them here; maybe in a future blog post.

Essentially, what we are dealing with here is JSON being passed to us, and we have to do something with that JSON. Now you have structured JSON and Unstructured JSON. To learn more about the difference and more about JSON, there is a great article on the Go Blog here: https://go.dev/blog/json

As we know the fields we are getting from the CURL command, we know to use a structured JSON approach. To do that, we set structs to match the fields of the Returned JSON. We do that like this:

type Searchrequest struct {
    base       *url.URL
    token      string
    searchterm string
}

type Main struct {
    Type  string `json:"type"`
    Index int    `json:"index"`
    All   bool   `json:"all"`
}

type Query struct {
    Original             string `json:"original"`
    ShowStrictWarning    bool   `json:"show_strict_warning"`
    IsNavigational       bool   `json:"is_navigational"`
    IsNewsBreaking       bool   `json:"is_news_breaking"`
    SpellcheckOff        bool   `json:"spellcheck_off"`
    Country              string `json:"country"`
    BadResults           bool   `json:"bad_results"`
    ShouldFallback       bool   `json:"should_fallback"`
    PostalCode           string `json:"postal_code"`
    City                 string `json:"city"`
    HeaderCountry        string `json:"header_country"`
    MoreResultsAvailable bool   `json:"more_results_available"`
    State                string `json:"state"`
}

type Mixed struct {
    Type string        `json:"type"`
    Main []Main        `json:"main"`
    Top  []interface{} `json:"top"`
    Side []interface{} `json:"side"`
}

type MetaURL struct {
    Scheme   string `json:"scheme"`
    Netloc   string `json:"netloc"`
    Hostname string `json:"hostname"`
    Favicon  string `json:"favicon"`
    Path     string `json:"path"`
}

type Results struct {
    Title          string  `json:"title"`
    URL            string  `json:"url"`
    IsSourceLocal  bool    `json:"is_source_local"`
    IsSourceBoth   bool    `json:"is_source_both"`
    Description    string  `json:"description"`
    Language       string  `json:"language"`
    FamilyFriendly bool    `json:"family_friendly"`
    Type           string  `json:"type"`
    Subtype        string  `json:"subtype"`
    MetaURL        MetaURL `json:"meta_url"`
    Age            string  `json:"age,omitempty"`
}

type Web struct {
    Type           string    `json:"type"`
    Results        []Results `json:"results"`
    FamilyFriendly bool      `json:"family_friendly"`
}

type Response struct {
    Query Query  `json:"query"`
    Mixed Mixed  `json:"mixed"`
    Type  string `json:"type"`
    Web   Web    `json:"web"`
}

This gets us ready to receive and send data and do something with it. We will touch on that in a minute first to tell Go to go (no pun intended) and interact with the API. We have to do what I like to call building the request.

Let's Build our Request

To build the request, we use the "net/http" package. We essentially tell the package this is going to be a GET request with a token, and in the "net/http" package, there is a function called http.NewRequest this basically controls our request and allows us to pass certain parameters to the function. We use http.MethodGet to be able to pass in a token and our search query.

The following code builds our request:

req, err := http.NewRequest(http.MethodGet, encodedurl, nil)
    if err != nil {
        log.Fatal(err)
    }
    req.Header = http.Header{
        "Accept":               {"application/json"},
        "X-Subscription-Token": {searchrequest.token},
    }

As you can see, our req variable receives the encoded URL (we will touch on this after and nil for error handling). We then pass the variable into a struct and add the following two headers "Accept:" which is how we wish to receive our data in this instance json.

The other header "X-Subscription-Token" is the token we are passing in. We are getting this token from a flag when we run the program as it is not good practice to put API keys into your code; we pass the value from this flag into a struct field called searchrequest.token

Passing in Flags

This piece of code is at the beginning of our program and looks like this:

    // Ask For Token and Search Term
    searchrequest := Searchrequest{
        token:      "",
        searchterm: "",
    }

    flag.StringVar(&searchrequest.token, "token", "", "token")
    flag.StringVar(&searchrequest.searchterm, "searchterm", "", "searchterm")
    flag.Parse()

This asks our user for the token and the search term. Input is as follows:

go run getwebpage.go -token "VALUE" -searchterm "VALUE"

Working with Search Engines Search Terms

    encodedsearchterm := url.PathEscape(searchrequest.searchterm)

    encodedurl := fmt.Sprintf("https://api.search.brave.com/res/v1/web/search")
    queryparam := url.Values{}
    queryparam.Set("q", encodedsearchterm)
    if len(searchrequest.searchterm) > 0 {
        encodedurl += "?" + queryparam.Encode()
    }

The reason why you see in the code the following variable encodedurl is not because of security but because of a simple problem when working with search engines. When you enter a search term into the search engine, you put your term like this ** Show Me Some Cats**. As you can see, you put your text in with spaces.

Now, the search engine automatically encodes your search term to something like this: show%20me%20some%20cats.This tells the rest of the engine this is a whole string with spaces.

Using the "net/url" package, you can encode the term first by passing it to the url struct called url.Values{} (this is in the net/url package). Then you get the search query and pass it to url.Values{} but not before you have encoded it with the Encode function. Now, for this to work, you also have to attach it to the "q" header, as this is what the Brave API only accepts to be able to use your term to search. The code that attaches our encoded query to the header of "q" is queryparam.Set("q", encodedsearchterm)

Finishing our Request

Now we have the API token and the query (our search term), we build a client to handle our request and then pass the req variable to that client.

// Send Request
    client := &http.Client{}
    repsonse, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer repsonse.Body.Close()

Receiving The Request

Essentially, by this point, all I wanted to do was receive my data and write to a file in the format of JSON.

To do that, I used the "json/unmarshall" package, and in this piece of code, I received the JSON and unmarshal it: json.Unmarshal([]byte(body), &res)

Then, this code will indent my JSON so that It is semi-readable, and it will write it to a file in the same directory as my program and call it reply.json.

content := fmt.Sprintf("Query: %+v\n, Results: %+v\n", res.Query, res.Web.Results)
    bytes, _ := json.MarshalIndent(content, "", "")
    contentjsonerr := os.WriteFile("reply.json", bytes, 0644)
    if contentjsonerr != nil {
        log.Fatal(contentjsonerr)
    }

You can do a ton with that data, but this Blog post is already long enough, so I will stop here. I think I might write another post soon about what we can do with our cat's data.

If you want to write this program or run it yourself, you can find the code here: https://github.com/jasric89/headfirstgo/tree/master/Chapter13/HTTPProg

I wish to thank the Slack Gopher community (Join here: gophers.slack.com), particularly a fellow Gopher called Peter Hellberg, who helped me understand how APIs work. Without him, this blog post would not exist.

Happy Coding and The Cloud Dude will see you on the next Post.