In truth, this isn’t a minimal viable product; it’s a minimum viable backend. I’ve been a platform engineer for too long to start throwing together UIs, even in my spare time.

If you're interested in a full-fledged product, at least one startup is doing something similar.

What Does it Do?

My project shortlist isn’t what most people would call user-friendly. It was originally a way to get better at Kubernetes while solving a real problem in my life.

This is how to provision the Kubernetes cluster that does all the magic, using OpenTofu/Terraform:

The AWS-related variables allow workloads inside the Google Cloud Kubernetes cluster to send emails using AWS SES. The rest relate to where to scrape real estate ads from and how the large language model should decide whether or not to email me about an ad.

A periodic job scrapes the new property ads from the provided results URL. These property profiles are sent to a service that spins up a job that feeds the property description to an LLM with the provided system prompt. If the LLM outputs “yes”, I receive an email like this one:

Yes, it's rudimentary HTML. I thought about adding the images from the adverts, but spent too much time getting the system working. Even with the MVP in place, I’d be more excited about using a multimodal LLM to analyse the images than put them in the email.

How Does it Work?

The working version of my project is quite close to what I envisioned last year in this post. Here’s a high-level diagram of how it ended up:

So Many Repositories!

I chose a multi-repo approach for this project, which spawned eight repositories, including a new Helm charts repo. Here are some details on the repos and what they contain.

Development Feedback Loop

One thing I haven’t covered is what development looked like. In some cases, it could be quite laborious. Here’s the typical feedback loop for application changes:

1. Use the ingester to replay some recorded property profiles

2. Based on the state of the Kubernetes resources and logs of the different workloads, I would work out what needed changing

3. If it were application code, I would submit a PR to the application repository and trigger a new container image build

4. Make any updates to helm charts, the most common being bumping the application version

5. I would update the OpenTofu IAC with the new Helm chart version

6. Run tofu apply

7. Repeat

Some of this toil may have been avoided by writing automated integration tests. It would have been difficult to test the desired behaviour (email receipt), but doing so would have saved some manual work.

Another thing that could have simplified things for me is using Kustomise rather than Helm, given the fact that I wasn’t packaging applications for 3rd parties.

The record/replay step

To make it easy to repeatedly test the system using the same data, I added commands to the ingester program that would record profiles scraped from a real estate website to Redis and replay these profiles by sending them to the runner.

I would initiate recording using kubectl like this:

kubectl -n shortlist run test-ingest-record \
  --image=ghcr.io/simoncrowe/shortlist-rm-ingester:1.0.5 \
  --env=BASE_LISTING_URL=https://www.rightmove.co.uk/properties \
  -- \
  --redis-host=redis-dev-master.shortlist.svc.cluster.local \
  --redis-pass="$(kubectl get secret --namespace shortlist redis-dev -o jsonpath="{.data.redis-password}" | base64 -d)" \
  record 'https://www.rightmove.co.uk/property-to-rent/find.html?maxPrice=2500&index=0&sortType=6&channel=RENT&transactionType=LETTING&locationIdentifier=REGION%5E93917&displayLocationIdentifier=London&minBedrooms=1&maxBedrooms=2&dontShow=retirement%2Cstudent%2ChouseShare&furnishTypes=unfurnished&propertyTypes=flat' \
  -n 5 \
  --db-key=test-2025-05-17

And replaying looked like this:

kubectl -n shortlist run test-ingest-replay \
  --image=ghcr.io/simoncrowe/shortlist-rm-ingester:1.0.5 \
  --env=BASE_LISTING_URL=https://www.rightmove.co.uk/properties \
  -- \
  --redis-host=redis-dev-master.shortlist.svc.cluster.local \
  --redis-pass="$(kubectl get secret --namespace shortlist redis-dev -o jsonpath="{.data.redis-password}" | base64 -d)" \
  replay http://run-dev-shortlist-runner.shortlist.svc.cluster.local/api/v1/profiles \
  --db-key=test-2025-05-17

Conclusion

This wasn’t just a tech experiment. It was a personal tool built to solve a real problem, using tools and practices that are important to me as a software engineer. From scraping real estate listings to assessing them with an LLM and sending actionable alerts via email, the whole thing runs automatically in the cloud. I’m the first to admit it’s not pretty, but it works, and it taught me a lot about managing complexity in small, polyglot systems and the basics of MLOps for inference.

I’m not sure I’d recommend this approach to everyone. It may have been quicker and cheaper to cobble something like this together using 3rd party tools. However, for someone like me who’s been doing backend and platform engineering for almost eight years, it provided an excellent opportunity to explore new technologies and experiment with greenfield code. I may pick the project up again in the coming years and improve on it, whether this be replacing the HTTP communication with an event bus or experimenting with whatever the future holds for AI.

If you're working on or thinking about something similar, I hope this gave you a few ideas. And if you want to dive deeper, the linked posts below cover many of the technical challenges in more detail.

Further Reading

I’ve written a few detailed articles on the problems I solved while working on this project:

• Programmatically Creating Kubernetes Jobs With Go

• On Python Testing

  • Part 1: isolating external calls and minimising mocks

  • Part 2: Dependency Inversion with Service Functions

• Multi-Cloud IAM: Securely Using AWS SES From Google Cloud Kubernetes

• Running the Llama 3.1 8B Large Language Model Cheaply on Google Cloud Kubernetes