Measurement. Your Foundation for IA Success.

Hello and welcome to Ayo’s Notebook. I am currently participating in Abby Covert’s IA in Practice. This is a monthly workshop learning about all things Information Architecture. This month’s notebook is on Measurement. You can read last month’s notebook on Auditing here.

Why Do We Measure?

Almost anything can be measured; these days, most things are measured for both good and bad. How many steps did you take, how many hours did you spend working, how much did that cost, and what is the ROI of this? There’s measurement everywhere to show what matters. But it is important to remember that not all measurement is meaningful without context.

One key lesson I took from this workshop session was how easy it is to measure the wrong things. Why? It is simple to do, and they’re right there. Someone shared how a team measured the low-hanging, easy metrics and used it to make important decisions. The result was as expected. Poor.

Avoiding convenient measurements and taking your time to decide what matters is vital. Know what is useful and meaningful to measure. That will help you make significant progress in your project. So how do you know what makes measurement meaningful?

You start with Baseline Measurement.

Baseline Measurement: The Foundation

The best way to describe baseline measurement is a “before” picture. Transformations are all the rage. If you’ve never come across a ‘Before vs After” picture, I would like you to share your settings for how you access the internet. Think of baseline measurement as where you currently are. Before making any changes, before all the buzzwords, the KPIs, BYTs, OKRs, etc. (One of those might be made up). Baseline measurement is your position before taking a step.

Without measuring your baseline, almost any measurements you take after this will just be floating in space, not rooted or connected to anything. In its void. Something people do is skip the baseline measurement and head straight to benchmarking. What’s the difference?

Baseline measurement is capturing the system in its present state. Measure what exists right now without any additions or subtractions.
Benchmarking is comparing your system to a standard or ideal.

Your baseline can be a benchmark, but it is not always that way. Most times, a benchmark is an external factor you include. A goal you set based on industry standards or competitive analysis.

Types of Measurement

Once you have your baseline, you need to figure out what to measure. A common mistake here is thinking that everything that can be measured should be measured. Luckily, the workshop host came up with some great measurement categories that I think capture the key measurement types.

They are:

Output: What we produce and its impact. This could be either positive or negative
Cost: This usually has a connection to money. Sometimes, it is hidden, but most times, it accumulates over time
Sentiment: This is a measure of how people feel. Taking the feelings and putting it into a number. For example, the satisfaction score.
Adoption: This type of measurement deals with how readily people accept new tools, processes, or systems. Measuring how long it takes people to adopt and change.
Engagement: This focuses on ongoing interaction and sustained use over time.
Time: Time focuses on identifying and reducing wasted effort. It could be the time spent searching for information to work.
Attrition: This is a measure of loss. Loss of people, resources, or even momentum.
Extensibility: This is a measure of how well the solution adapts, grow,s and measures over time.

These categories cover the majority of what will need measuring. This leads me to want to discuss engagement metrics, which I learnt are one of the most misused categories.

Engagement Metrics

Engagement metrics require a lot of nuance to be able to make sense. They are often the most used metrics because of how easy they are to capture. Clicks, Views, Time spent on page, Number of readers. These numbers are easy to track, and they always look good on reports. You should remember: engagement does not necessarily mean effectiveness.

For example, if a knowledge base has a high time on-site, does that mean it’s good? Yes? No? Maybe? It could be that the content is confusing and users are struggling to find what they need. Engagement metrics should be used carefully. A recommendation could be using them with qualitative measurement (e.g user feedback, behavior tracking, etc.).

Going purely off engagement metrics without nuance increases the risk of optimizing the wrong thing.

The Process of Meaningful Measurement

So with all that information, how can we measure the right things in the right way? We follow a simple process.

Start with Intention: Write down what you want to measure. Be as specific as possible. Make it time-bound.
Get Real about your Why: You know what you want to measure. Why do you want to measure it? Why does it matter? Who benefits from this process? If you can’t answer these questions. Rethink your metric and go back to step 1.
Ask a Measurable Question: Key questions. “How will I know if this is working?” “How many …” “How much …” Ask questions whose answers can be measured
Set up Your Measurement System:

Pick one thing to track (metric)
Record where you are now (baseline)
Set a target (goal)
Define the timeline (by when)

Create a Warning System: Data changes, sometimes it requires constant attention. Sometimes, it does not. You should set flag conditions and what to do when these conditions are triggered. Flag conditions are situations that let you know when to look closer at the data. It could be something going wrong or something going well.

Bringing It All Together

Once you’ve started measuring, you have to decide on a format to store and share the data. Your needs decide what format will work best.

Spreadsheets - Best for manual aggregation of data.
Workbooks - This is best when you’re working on a measurement over time. If you have multiple spreadsheets, it is best to put them into a workbook.
Reports - This is best for when you’re communicating measurement to a wide audience. It allows you to include narrative tools such as text or graphics.
Dashboards - Dashboards are the best for monitoring live data

It is impossible to create a perfect measurement system. It’s best to build something that helps you track what’s happening so you can make better decisions. The goal isn’t to measure, it’s to use the measurement to generate impact.

Remember that sometimes we measure things just because we can measure them, but the real work is in figuring out what needs to be measured. Until next month, thank you for reading!

Ayo’s Notebook: Measuring Before Change Matters More Than You Think