Solr CLI Revamp: Updated Progress

Introduction

Origin of Solr's CLI

Solr's CLI originated when Solr moved from being a WAR file that was deployed in any Java web server to being a standalone server application. That migration required the creation of a script to start Solr, and thus the bin/solr command was created, along with bin/solr.cmd for Windows. Over the years more commands were added, like being able to create a collection, manage security authentication capabilities for Solr, and even growing to interesting features like a Package Manager for Solr!

Time passed, and we started having fragmentation in how we interacted with Solr from the command line. We started having additional scripts, some that only worked on Linux, others that had both Windows and Linux versions (thinking of you bin/post.sh!). We also started using competing Java libraries for building CLIs. Plus, the bin/solr scripts exploded into 4000+ line monsters that became very brittle and difficult to maintain.

In April 2023 it was time to get Solr's CLI back under control.

Setting goals and objectives for the new CLI

Fortunately a lot of ground work had been done by various members of the community when I created SOLR-16757: Umbrella Ticket for Revamping Solr CLI's for the Future back in April 2023:

SOLR-11206: Migrate logic from bin/solr scripts to SolrCLI - Opened in 2017, it was the first effort to tame the CLI! I was able to pull a LOT of code and a lot of ideas from this work!
SOLR-11749: integrationTests (BATS) fail on GitHub and Jenkins Export: From late 2022 this established the testing framework on Unix for the bin/solr script, making it possible to start iterating on the scripts with less risk.

So then it was time to think about what success looks like in revamping Solr's CLI.

Establish common patterns in the CLI. Some examples of this is ensuring that all commands work in both Windows and Unix/OSX environments.
All commands should work with all authentication schemes for Solr. We have Basic Auth done for Solr 10!
Discovering how to use commands should work the same for commands.
Ref Guide should mention all the various commands that are available.
Use Commons CLI library for option parsing across all of our commands.
Introduce unit and integration tests across all commands.
Clean up the long list of JIRA tickets related to the CLI!

Key Wins So Far

Enhanced user interface and experience

The bin/solr and bin/solr.cmd are now the one stop shop for all of the CLI commands that ship with the core Solr. As of Solr 9.7 we've eliminated the separate zkcli.sh, post.sh, postlogs.sh in favor of bin/solr command equivalents. Just missing the cut off date, and therefore landing in 9.8 is the removal of the last custom script, snapshotcli.sh!

We've also established significant more commonality across the various commands. We've embraced the Linux standard of -v and --verbose for formatting our commands. And for longer ones, we are using "kebab case", going from -noprompt to --no-prompt. Don't worry, if you are on the 9x line of Solr, this is a backwards compatible change, so you have time to update your scripts!

Plus, we improved discoverability of the options for commands by leaning into HelpFormatter that the Commons CLI library provides, plus some of own visual formatting so that invoking a command with -h looks it's best. Some commands, like bin/solr package had provided a different, and richer, style of help than the default that Commons CLI provides, and now we have both styles integrated into one holistic whole.

Improved commands and functionalities

Interacting with Solr? We've revamped how the CLI connects and you can now pass either a --zk-host Zookeeper address or --solr-url with any command to establish a connection to Solr.

Need to create a core or collection? Yeah, we have you covered with just a single bin/solr create command.

Working with Snapshots? Coming in Solr 9.8 we have a new bin/solr snapshots set of commands that let you manage them from the command line.

Lastly, there are so many bug fixes! There was a fair number of edge cases (like uploading to ZK a configset with symlinks on the file system) that gave odd errors.

Where are We?

We have seen a steady stream of improvements starting with Solr 9.2, and continuing in each release up to 9.6. The upcoming 9.7 release is the big one, where we introduce the use of kebab formatting for our CLI commands. -solrURL becomes --solr-url for example. Don't worry, we still have backwards compatibility in the 9x line for this. I think of it as "the big one" because it's where we are making significant changes to how folks interact with the CLI, and I expect a lot of feedback.

In a forth coming blog, I'll talk about the changes that have already hit by individual Solr release.

Whats Coming?

As I mentioned earlier, the handling of Snapshots via bin/solr just missed the cut off for 97, so you will see that in 9.8.

In SOLR-17359 we already have made major improvements in how the bin/solr CLI handles the ZK subcommands. Previously that was through a lot of custom shell scripting, and now that is in Java code. This has dropped the bin/solr script from 4000+ lines to "just" 1482 lines.

Christos has been doing fantastic work on continuing to rationalize the various options used in the CLI and identifying duplicate usages with different meanings or inconsistencies. You can check out this great spreadsheet: https://docs.google.com/spreadsheets/d/1ws44kN51WnGwQzOXc8KKRQ93TMgHSqIGb02MRWFel_U/edit?usp=drivesdk

We also need to restructure the Ref Guide a bit. We have multiple pages that talk about various tools, for example the section on the Package Management, while the original page that was focused on starting/stopping of Solr, has grown way beyond that scope to cover the API Tool, Zookeeper operations, exporting/importing docs etc. We need someone to take a fresh look at how to structure this documentation.

I'd like to see Solr's CLI start consuming Solr's V2 API. It would be great to use the V2 API Java client in the Solr CLI instead of our hand crafted JSON requests.

And lastly, we need to remove the deprecated CLI code from main so that reduce the maintenance burden!

Keep an eye on https://issues.apache.org/jira/browse/SOLR-16757: Umbrella Ticket for Revamping Solr CLI's for the Future.

Are we there yet? An update on revamping Solr's CLI.