Use Ollama to generate commit messages from Changesets 🦋

In the world of continuous software development, efficiently generating commit messages can save valuable time (μS - microseconds). This article explores how to automate this process using large language models (LLMs).

For products that have continuous releases and work with changeset 🦋, I have some good news for you. Now you can save some μS per release by automatically generating our commit messages.

I started this exploration just for fun, just to learn more about creative prompts, nudge models to return JSON responses, and how to modify temperatures.

Tools I used

1. Ollama to serve some LLMs locally

2. HTTPie to make the requests to the Ollama server

Full code

Here you can find the GitHub gist of the bash file I am using. There are some parts that can be of interest.

Prompting

Conventional commits

At Neo4j's Design System repo we work with Conventional Commits. To instruct the LLM to work with this format I included in the prompt:

You follow the Conventional Commits specification.
feat: for new features
chore: for maintenance work
fix: for bug fixes

JSON response

Then I wanted the result to be a JSON response that I could parse. That part of the code that instructs the LLM to do this:

Example commit message: 'feat: added new disabled boolean variable to button'
Return the commit message as a json response.
THE RESULT SHOULD ONLY BE A VALID JSON!

Example response:
{
  "commit": "feat: added new disabled boolean variable to button"
}

Turns out, that when you SHOUT, LLMs listen better? 📣

Commit length and tuning

By default Ollama temperature is set to 0.8, with the higher the value, the more creative the output of LLM.

Turns out that for maintaining a short commit message, and allowing an LLM to summarise properly, a temperature of around 0.9 produces better results.

For an input of:

Keep the generated commit message strictly under 50 characters, if its more summarise it.
THE COMMIT MESSAGE SHOULD NOT BE MORE THAN 50 CHARACTERS!!!!!

The responses where the following:

# Temperature 0.8 (default)
"fix: updated wizard's layout with new flex gap and gradient color"

# Temperature 0.9
"fix: improved wizard's layout and styling"

Full demo

Below you can see a result from an example changeset file, and the resulting commit it generates. Not bad 🦹‍♂️, micro seconds are just saved.