A Trip Down Memory Lane: Improvements to Solr CLI from 9.2 to 10.x

Introduction

I want to share some of the changes to Solr CLI since I embarked on revamping the CLI in Solr 9.2, and to highlight some of the folks who came along with me on this journey! This isn't an exhaustive review, subject to my many memory failures, and mostly informed by https://solr.apache.org/docs/9_6_1/changes/Changes.html. I'm going to miss some changes and some of the folks who contributed!

Looking back helps me stay excited for the future!

Solr 9.2 CLI Changes

First sign of this effort.

SOLR-16672: Simplify SolrCLI Healthcheck command's logic for obtaining a target collection

Solr 9.3 CLI Changes

Now we are getting started! Found that the package tool doesn't support -h or -help (this is before we decided to move to --help) style. We eliminate -usage in favour of -help as well. We get the CLI to use the Jetty HTTP2 client and break SolrCLI.java from one class with all the tools into a new org.apache.solr.cli with each tool having it's own java file, i.e AuthTool.java, CreateTool.java etc. The ExportTool now properly exports in JSON and JSON With Lines documents.

First contributions from the community, from Jesse Gorzinski and Bence Szabo on the CLI are merged.

Solr 9.4 CLI Changes

Now we are getting somewhere. bin/solr auth now creates reasonable roles and permissions for security: 'search', 'index', 'admin', and 'superadmin' roles are created and assigns the default user superadmin role. We are finally thinking about what would our users want by default from Security!

In our quest to properly support Windows, the postlogs.sh script is migrated to bin/solr postlogs command. Postlogs lets you store Solr's logs in a Solr collection for further analysis.

Running the Solr CLI on the box running Solr? Now the CLI looks up the environment variables to connect to Solr so you don't need to pass in the Zookeeper or Solr urls to the commands.

Alex added support to Solr Exporter script to connect to Solr with basic authentication. Adding additional support for basic auth to all of the Solr tools is critical to making running a secure Solr the default option!

Thanks Jan, Will White, Alex Deparvu, and Houston for the improvements.

Solr 9.5 CLI Changes

The migration from bin/post to bin/solr post was slightly rough. In 9.5 we updated the Ref Guide to consistently refer to bin/solr post, and we made sure the options in bin/solr post mirrored those in the deprecated bin/post script.

We also discovered bin/solr script didn't properly support the "ps" command on some systems.

We introduced the idea that in the CLI, a Solr url doesn't have to end in /solr. -url http://localhost:8983 is the same as -url http://localhost:8983/solr. This is partly to simplify the V2 apis (which live under http://localhost:8983/api). You now get a nice message if you append the /solr in the URLs, and we started converting the various tools that look up Solr urls to be aware and remove the trailing /solr.

I think it was around this time that I started having a LOT of conversations with Jason Gerlowski about "how do we handle backwards compatiblity?". I really appreciate all the time he spent working with me on this topic. I don't know that we perfectly nailed it, but it's much better than if I hadn't had his perspective. Thanks Jason!

Thanks Vincenzo D'Amore for contributing the ps fix!

Solr 9.6 CLI Changes

According to the changelog, this was pretty quiet. Checking JIRA via https://issues.apache.org/jira/browse/SOLR-17017?jql=project%20%3D%20SOLR%20AND%20fixVersion%20%3D%20%229.6%22 there are some more changes, but mostly small. I think this was a period where the PR's had a lot of change, and they hung out open for a LONG time.

We enhanced the unit tests for the PostTool command, and introduced a nice --dry-run command to simulate posting documents to Solr.

We fixed some issues with our example Solr setups like bin/solr start -e films.

Solr 9.7 CLI Changes

This brings me up to today, late August 2024, and the impending Solr 9.7 release. This is the big one!

SOLR-16824: Adopt Linux standard pattern of -- for long option commands, and make all commands "kebab" formatting. This was a big effort by Eric, Jan, Jason, and new contributor Christos Malliaridis. I want to especially thank Rahul Goswami for his work on making sure the CLI works on Windows. He generated some great PRs and made me a lot more confident about shipping these changes. Thanks Jan for digging in on the backport to 9x, I was definitely floundering on that.

With this change options such as -zkHost are now kebab formatted as -zk-host. The old parameters such as -zkHost continue to be supported in the 9.x line of Solr.

I can't tell you enough how nice it is to have consistency in how our parameters work, and get away from the mix mash of styles that we had. Previously some tools like the PackageTool embraced the kebab, while the AuthTool had camelCase style.

We also decided that for passing in user credentials, we are going with -u and --credentials across all our Tools, which was inspired by what Curl uses. Thanks Curl!

We aren't done on the migration yet however. Big thanks to Christos for going through the code and analyzing all the various command lines. He has put together this spreadsheet to track the work: https://docs.google.com/spreadsheets/d/1ws44kN51WnGwQzOXc8KKRQ93TMgHSqIGb02MRWFel_U/edit?gid=0#gid=0 and I'm very excited to have his eyes on this problem.

We also finally have AuthTool, DeleteTool, and PackageTool all using Java code for arg parsing instead of doing it in the bin/solr and bin/solr.cmd shell scripts, continuing to shrink the lines of code and reduce the duplication of logic.

We have deprecated zkcli.sh by adding linkconfig, cluster, and updateacls sub commands to bin/solr zk. You no longer need zkcli.sh for certain specialized tasks. Also finally have better docs in the Ref Guide around this.

We also added support to use a --solr-url with all the bin/solr zk commands. So now you don't need to know the ZooKeeper host, you can just point your zk commands to Solr and it looks up the required information under the covers. This may help make it easier for folks to interact with ZooKeeper without becoming experts in it. Longer term, the need for separate ZooKeeper prevents teams from using Solr, and we need to solve this.

Solr Exporter saw some improvements, with the addition of supporting using it via SSL with Solr and the migration from the argparse4j library to Commons CLI. One less dependency for Solr!

Thanks Eivind Bergstøl, Rahul, Christos, and everyone else who worked on this release!

Conclusion

I am very excited about the upcoming 9.7 release, and a bit nervous as this will be the version where we start getting a LOT of feedback on the CLI changes. There is already a lot of great work teed up for 9.8, and of course some NEW features that are only in 10. I'll blog about them in the future, but you can always get a peek via the Changes.txt file and keeping an eye on the umbrella JIRA issue.