Building a Flat-Hunting Shortlist with AI and a Polyglot Cloud Backend


In truth, this isn’t a minimal viable product; it’s a minimum viable backend. I’ve been a platform engineer for too long to start throwing together UIs, even in my spare time.
If you're interested in a full-fledged product, at least one startup is doing something similar.
What Does it Do?
My project shortlist isn’t what most people would call user-friendly. It was originally a way to get better at Kubernetes while solving a real problem in my life.
This is how to provision the Kubernetes cluster that does all the magic, using OpenTofu/Terraform:
The AWS-related variables allow workloads inside the Google Cloud Kubernetes cluster to send emails using AWS SES. The rest relate to where to scrape real estate ads from and how the large language model should decide whether or not to email me about an ad.
A periodic job scrapes the new property ads from the provided results URL. These property profiles are sent to a service that spins up a job that feeds the property description to an LLM with the provided system prompt. If the LLM outputs “yes”, I receive an email like this one:
Yes, it's rudimentary HTML. I thought about adding the images from the adverts, but spent too much time getting the system working. Even with the MVP in place, I’d be more excited about using a multimodal LLM to analyse the images than put them in the email.
How Does it Work?
The working version of my project is quite close to what I envisioned last year in this post. Here’s a high-level diagram of how it ended up:
So Many Repositories!
I chose a multi-repo approach for this project, which spawned eight repositories, including a new Helm charts repo. Here are some details on the repos and what they contain.
Name | Purpose | Language |
shortlist-rm-ingester | Ingests property listings and forwards them to the (assessor) runner. | Python |
shortlist-runner | Creates jobs to run assessors using the Kubernetes API when profiles are POSTed to its API (i.e. by the ingester). | Go |
shortlist-llm-assessor | Runs an LLM with a given system prompt over profile text and either accepts or rejects the profile. Acceptance means forwarding the profile to a URL (the notifier). | Python |
shortlist-dummy-assessors | Some simple assessors to use during development. | Go |
shortlist-schema | Contains shared Go structs and serialisation/deserialisation code. (Doesn’t mesh well with Python; OpenAPI specs may be better for a polyglot project.) | Go |
shortlist-rm-email-notifier | Exposes an API allowing clients to trigger emailing the property profile to a given address. | Python |
shortlist-infra | The OpenTofu code for the AWS and Google Cloud resources needed to run the project. | HCL |
helm | Helm repo for the ingester, runner and notifier charts — Made using the chart-releaser GitHub action | YAML + Go templates |
Development Feedback Loop
One thing I haven’t covered is what development looked like. In some cases, it could be quite laborious. Here’s the typical feedback loop for application changes:
Use the ingester to replay some recorded property profiles
Based on the state of the Kubernetes resources and logs of the different workloads, I would work out what needed changing
If it were application code, I would submit a PR to the application repository and trigger a new container image build
Make any updates to helm charts, the most common being bumping the application version
I would update the OpenTofu IAC with the new Helm chart version
Run
tofu apply
Repeat
Some of this toil may have been avoided by writing automated integration tests. It would have been difficult to test the desired behaviour (email receipt), but doing so would have saved some manual work.
Another thing that could have simplified things for me is using Kustomise rather than Helm, given the fact that I wasn’t packaging applications for 3rd parties.
The record/replay step
To make it easy to repeatedly test the system using the same data, I added commands to the ingester program that would record profiles scraped from a real estate website to Redis and replay these profiles by sending them to the runner.
I would initiate recording using kubectl
like this:
kubectl -n shortlist run test-ingest-record \
--image=ghcr.io/simoncrowe/shortlist-rm-ingester:1.0.5 \
--env=BASE_LISTING_URL=https://www.rightmove.co.uk/properties \
-- \
--redis-host=redis-dev-master.shortlist.svc.cluster.local \
--redis-pass="$(kubectl get secret --namespace shortlist redis-dev -o jsonpath="{.data.redis-password}" | base64 -d)" \
record 'https://www.rightmove.co.uk/property-to-rent/find.html?maxPrice=2500&index=0&sortType=6&channel=RENT&transactionType=LETTING&locationIdentifier=REGION%5E93917&displayLocationIdentifier=London&minBedrooms=1&maxBedrooms=2&dontShow=retirement%2Cstudent%2ChouseShare&furnishTypes=unfurnished&propertyTypes=flat' \
-n 5 \
--db-key=test-2025-05-17
And replaying looked like this:
kubectl -n shortlist run test-ingest-replay \
--image=ghcr.io/simoncrowe/shortlist-rm-ingester:1.0.5 \
--env=BASE_LISTING_URL=https://www.rightmove.co.uk/properties \
-- \
--redis-host=redis-dev-master.shortlist.svc.cluster.local \
--redis-pass="$(kubectl get secret --namespace shortlist redis-dev -o jsonpath="{.data.redis-password}" | base64 -d)" \
replay http://run-dev-shortlist-runner.shortlist.svc.cluster.local/api/v1/profiles \
--db-key=test-2025-05-17
Conclusion
This wasn’t just a tech experiment. It was a personal tool built to solve a real problem, using tools and practices that are important to me as a software engineer. From scraping real estate listings to assessing them with an LLM and sending actionable alerts via email, the whole thing runs automatically in the cloud. I’m the first to admit it’s not pretty, but it works, and it taught me a lot about managing complexity in small, polyglot systems and the basics of MLOps for inference.
I’m not sure I’d recommend this approach to everyone. It may have been quicker and cheaper to cobble something like this together using 3rd party tools. However, for someone like me who’s been doing backend and platform engineering for almost eight years, it provided an excellent opportunity to explore new technologies and experiment with greenfield code. I may pick the project up again in the coming years and improve on it, whether this be replacing the HTTP communication with an event bus or experimenting with whatever the future holds for AI.
If you're working on or thinking about something similar, I hope this gave you a few ideas. And if you want to dive deeper, the linked posts below cover many of the technical challenges in more detail.
Further Reading
I’ve written a few detailed articles on the problems I solved while working on this project:
Subscribe to my newsletter
Read articles from Simon Crowe directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by

Simon Crowe
Simon Crowe
I'm a backend engineer currently working in the DevOps space. In addition to cloud-native technologies like Kubernetes, I maintain an active interest in coding, particularly Python, Go and Rust. I started coding over ten years ago with C# and Unity as a hobbyist. Some years later I learned Python and began working as a backend software engineer. This has taken me through several companies and tech stacks and given me a lot of exposure to cloud technologies.