Introduction

Automated testing was a problem I ran into early on with pyonepassword. The way pyonepassword works is by driving the 1Password CLI tool, op, which talks to your 1Password cloud account. This introduces all sorts of problems for testing, including:

• The op command requires authentication for most operations

• The performance implications of op's live authentication and interaction with the 1Password cloud

• Lack of testing isolation, since a live 1Password account pre-populated with data is required

I needed a tool that would pretend to be op but would:

• Use self-contained data, ideally added to the tested project's version control

• Not require network connectivity

• Not, under any circumstances, require console interaction

• Produce the same output every time given a particular input

So I developed mock-op, which does the above, PLUS provides a DVR-style capability to observe and record op to play back its behaviors.

Overview

The mock-op package comes two significant parts:

• A mock-op console utility that plays back responses identical to the real op command's output for a given set of command-line arguments. Additionally, the same exit statuses are returned so that errors can be handled appropriately.

• A response-generator utility to execute op under various scenarios and record its responses for use by mock-op

Usage and Installation

Install using pip from GitHub:

$ pip install git+https://github.com/zcutlip/mock-op

mock-op CLI Tool

For mock-op to simulate the real op it needs a directory of responses to look up and playback. The anatomy of the response directory is discussed in the README for mock-cli-framework, but below is an example:

JSON Dictionary of command invocations:

{
  "meta": {
    "response_dir": "./responses"
  },
  "commands": {
    "item|get|Example Login 1|--vault|Test Data": {
      "exit_status": 0,
      "stdout": "output",
      "stderr": "error_output",
      "name": "item-get-[example-login-1]-[vault-test-data]"
    },
    "item|get|nok7367v4vbsfgg2fczwu4ei44|--fields|username,password": {
      "exit_status": 0,
      "stdout": "output",
      "stderr": "error_output",
      "name": "item-get-[example-login-2]-[fields-username-password]"
    },
    "item|get|Invalid Item": {
      "exit_status": 1,
      "stdout": "output",
      "stderr": "error_output",
      "name": "item-get-[invalid-item]"
    }
  }
}

An on-disk repository of response files:

responses
├── item-get-[example-login-1]-[vault-test-data]
│   ├── error_output
│   └── output
├── item-get-[example-login-2]-[fields-username-password]
│   ├── error_output
│   └── output
└── item-get-[invalid-item]
    ├── error_output
    └── output

With that directory in place, the mock-op utility will look up responses and exit status based on the set of command-line arguments given.

Let's emulate an "item get" for item UUID nok7367v4vbsfgg2fczwu4ei44, requesting fields 'username' and 'password':

$ mock-op item get nok7367v4vbsfgg2fczwu4ei44 --fields username,password
{"password":"weak password","username":"janedoe123"}

The output is the exact JSON that op would give.

Let's try an "item get" for an item that doesn't exist:

$ mock-op item get "Invalid Item"
[ERROR] 2021/02/03 13:39:42 "Invalid Item" doesn't seem to be an item. Specify
the item with its UUID, name, or domain.
$ # check the exit status...
$ echo $?
1
$

The output on stderr is the same error message op would give, and the exit status is 1, as expected.

Automated Response Generation

The response file & directory structure was designed to be fairly straightforward so that one could create it by hand or easily script it. However, mock-op comes with a tool to generate responses. You provide a configuration file, and it will connect to your 1Password account (using the real op tool), perform the queries, and record the responses.

Note: response generation requires you to install the pyonepassword Python package. It's responsible for driving the op command under the hood so its responses can be captured.

It's only required during response generation and is not required for subsequent use of mock-op.

It can be found in PyPI and installed via: pip3 install pyonepassword.

Here's an example configuration file for generating responses. Note that the invalid item definition has an expected-return value of 1. This tells response-generator that an error is expected and should be captured rather than failing.

[MAIN]
config-path = ./tests/config/mock-op
response-path = responses
response-dir-file = response-directory.json

[whoami]
type=whoami

[item-get-example-login-1-vault-test-data]
type=item-get
item_identifier = Example Login 1
vault = Test Data

[item-get-invalid]
type = item-get
item_identifier = Invalid Item
expected-return = 1

Then you can run response-generator and have it create your response directory:

❱ response-generator ./examples/example-response-generation.cfg
Running: op --version
Running: op --format json account list
Running: op --format json whoami
Running: op signin --raw
Running: op --format json whoami
Signed in as User ID: *********************ERLHI
About to run: op --format json whoami
About to run: op --format json item get 'Example Login 1' --vault 'Test Data'
About to run: op --format json item get 'Invalid Item'

The response generator records the output, standard error, and exit status of each invocation of op. It then records that in the JSON response directory. We end up with a directory that looks like:

$ tree tests/config/mock-op
tests/config/mock-op
├── response-directory.json
└── responses
    ├── item-get-example-login-1-vault-test-data
    │   ├── error_output
    │   └── output
    ├── item-get-invalid
    │   ├── error_output
    │   └── output
    └── whoami
        ├── error_output
        └── output

Limitations

The main limitations are:

• mock-op can't simulate various authentication flows

  • It can simulate being authenticated or not, resulting in failures where appropriate

• mock-op can't currently simulate op operations that change state

  • In other words, if you need to simulate a successful "item get," followed by an "item delete," and finally an "item get" that fails, mock-op can't currently do this

  • Support for state-changing operations is currently under development, so stay tuned!

Summary

If you have a situation where you need to simulate the op command, but that's impractical for performance or other reasons, mock-op may be for you.

My pyonepassword project has over 455 automated test cases, most of which use mock-op in some way. You Can browse that real-world mock-op configuration here.

Check out mock-op at its GitHub project page.