Unlocking Hidden Data: How I Scraped My Way to New Zealand's Housing Insights

Earlier this year, I set out to analyze the New Zealand housing market. I wanted to uncover spatial trends, track price changes over time, and figure out what’s really stopping me (and many others) from getting onto the property ladder.

The problem? The kind of data I needed — detailed, granular, the good stuff — isn’t easy to come by.

There is an organization that holds it: REINZ (the Real Estate Institute of New Zealand), And their data doesn’t come cheap. To get access, you either need to:

• Be a licensed real estate agent

• Work as a property manager

• Or pay for a tailored business subscription

I wasn’t exactly keen to change careers just to get my hands on some data.
So… was that the end of the project? Not a chance.

---

Enter: Web-Scraping

When official data is locked away behind paywalls, there’s another option: web scraping. If information is visible in your browser, chances are you can write a script to collect it automatically. It’s a bit of a grey area ethically, but when done responsibly, it’s a powerful way to build your own dataset.

I found a website that contains all of the data that I am looking for, allowing you to filter for recently sold and listed properties all across the country.

On the front page, you can search across all of New Zealand. Sure, you could spend countless hours clicking through listings and copying down property details — but there’s a better way.

Most websites have what’s called a sitemap.
Think of it as a directory that tells search engines exactly where every page lives. And lucky for me, this website had one at it’s /sitemap.xml.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://www.realestate.co.nz/agents.xml</loc>
<lastmod>2025-08-19T12:14:24+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://www.realestate.co.nz/auckland-insights.xml</loc>
<lastmod>2025-08-19T12:19:35+00:00</lastmod>
</sitemap>
<sitemap>
<loc>https://www.realestate.co.nz/auckland-suburbs.xml</loc>
...

This sitemap is essentially a giant list of every page they want search engines to find — perfect for scraping. Using it as a base, I wrote a Python script that collected:

• Historic sales

• Council valuations

• Property attributes (bedrooms, bathrooms, land/floor area, parking, pools, etc.)

• Location data (latitude & longitude)

Keeping the database fresh was simple: re-run the script, and it would pull in the latest updates. Nice!

The catch was rate limits. I could only pull data for about three properties per second. And this is where the grey area of web scraping comes in.

Fire thousands of requests at a site every second and you’ll almost certainly get your IP blocked — not to mention the strain it puts on their servers. The rule of thumb is to stick to 2–3 requests per second. Slow, yes, but safe.

So I played by the rules. My script chugged away for hours each day, gradually collecting data until I had every property across the country.

I then transformed the raw data into structured tables — regions, properties, sales, valuations — ready for analysis:

Regions

Properties:

Sales:

Valuations:

From there, I ran some quick exploratory analysis in Python:

I also pulled in regional boundaries, which made it possible to map the data on top of the spatial boundaries.

Seeing the patterns on a map sparked an idea: why not turn this into an interactive dashboard?

Using Power BI (with regional map files I sourced from LINZ and Stats NZ), I built a tool where you can toggle between sale prices and council valuations, switch between yearly or monthly data, and drill all the way down from regions → districts → suburbs.

Very cool!

You can explore all of the code here: GitHub – property_tracker

What started as a personal project in housing analysis turned into an exercise in scraping, data modelling, and visualization — and the same approach could be applied anywhere detailed data is locked away.

If you’d like to see more projects like this, hit subscribe and follow — you’ll get an email whenever I post something new.