Wrangle log data in cli with pb

We released pb, a modern, fast and intuitive CLI for log management couple of months ago. pb is built for developers who like the comfort of their terminal. Log data is primarily textual, and hence it was important that pb is pluggable via the linux pipe | to other text manipulation tools like grep, awk, sed etc.

In this post, we will see how pb can be used to wrangle log data in the terminal. We'll also explore the new pb tail command to see how it can be used to tail logs in the terminal.

Pre-requisites

pb is a terminal client for Parseable. So you'll need a Parseable installation up and running to use pb. You can follow the official Parseable documentation for installation instructions.

Installation

pb is available for Linux, MacOS and Windows. You can download the latest release from here. Once downloaded, extract and move the binary to a location in your $PATH and rename it to pb.

Get started

pb allows you to manage streams, users, roles, alerts and more. In this post, we'll see how to manage streams, tail logs and query logs using pb.

➜  pb help           

pb is the command line interface for Parseable

Usage:
  pb [flags]
  pb [command]

Available Commands:
  help        Help about any command
  profile     Manage different Parseable targets
  query       Run SQL query on a log stream
  role        Manage roles
  stream      Manage streams
  tail        tail a log stream
  user        Manage users
  version     Print version

Flags:
  -h, --help      help for pb
  -v, --version   Print version

Use "pb [command] --help" for more information about a command.

Set up a profile

By default, pb ships with a profile called demo. This profile points to the Parseable demo installation. We'll use this profile (i.e. the Demo server) to get started with pb right away in this blog post.

If you'd rather use your own Parseable installation, you can create a new profile using the pb profile create command. For example, to create a profile called local pointing to a Parseable installation running on http://localhost:8000, run the following command.

pb profile add local http://localhost:8000 admin admin

Stream management

pb allows creating, deleting, listing stream on the Parseable server. You can also fetch info about a stream using pb. To list all streams

➜  pb stream list  

•  frontend
•  druide2e
•  playminio
•  backend

To create a new stream, run the following command

➜  pb stream create teststream

Fetch info about a stream using the info command. For example, to fetch info about the backend stream, run the following command

➜  pb stream info backend

Info:
  Event Count:     521311
  Ingestion Size:  171 MB
  Storage Size:    23 MB
  Compression Ratio:    86.58%

No retention period set on stream

No alerts set on stream

Tail logs

pb can tail logs in real-time. The Parseable server exposes Arrow Flight based streaming endpoint that pb hooks on, to fetch logs in real-time. This means these logs are stored in Parseable and are queryable at a later point in time. But with live tail, you can see the logs in real-time as they hit the Parseable server.

The tail command takes only the stream name, and returns the logs in real-time. Run the following command to tail logs from a given stream

pb tail frontend | jq .

To filter logs with jq, you can use below approach

pb tail frontend | jq '. | select(.method == "PATCH")'

You can also add additional filters to the tail response with grep. For example

pb tail frontend | grep "POST" | jq .

To stop tailing logs, press Ctrl + C.

Query logs

Above we saw how to tails logs from a given stream. Lets now see how to query logs using pb. Let's start with a simple query. Run the following command to query all logs from the last 1 minute.

pb query "select * from frontend" --from=1m --to=now | jq . | less

The pb query command takes a SQL query as input and returns the results in JSON format. The --from and --to flags are used to specify the time range for the query. The jq command is used to format the JSON output. Finally, the less command is used to paginate the output.

Next, let's try a filter in the SQL query. Run the following command to query all logs from the last 1 minute, where the status field is 500.

pb query "select host, id, method, status from frontend where status = 500" --from=1m --to=now | jq . | less

You can now also direct the output to a file. Run the following command to query all logs from the last 1 minute, where the status field is 500 and direct the output to a file called 500.json.

pb query "select host, id, method, status from frontend where status = 500" --from=1m --to=now | jq . > 500.json

You can also add additional filters to the query response with grep. For example

pb query "select host, id, method, status from frontend where status = 500" --from=1m --to=now | grep "POST" | jq . | less

Conclusion

In this post, we saw how to wrangle log data in the terminal with pb. We also saw how to tail logs in real-time and query logs using pb. pb is a modern, fast and intuitive CLI for log management. It is built for developers who like the comfort of their terminal. pb is pluggable via the linux pipe | to other text manipulation tools like grep, awk, sed etc.