As product managers, we face many challenges and uncertainties in our work. We have to deal with changing customer preferences, market trends, competitor actions, technical constraints, and stakeholder feedback. We also have to balance the trade-offs between speed, quality, cost, and scope of their products.

We are constantly faced with ambiguity. We have to make decisions based on incomplete information, assumptions, and hypotheses. We have to balance the needs and wants of various stakeholders, such as customers, users, engineers, designers, and executives.

In such a complex and dynamic environment, it is not easy to find the optimal solution for every problem or opportunity.

How can we navigate this complex and dynamic environment?

How can we reduce the risk of failure and increase the chances of success?

How can we learn fast and iterate quickly?

One of the most valuable skills that a product manager needs is Brute Force Experimentation. Brute Force Experimentation is the ability and willingness to test ideas and assumptions with real data and feedback from the market. It is the mindset and practice of learning by doing, rather than by planning or guessing.

Brute Force Experimentation helps us to:

• Quickly validate or invalidate our assumptions and hypotheses about the problem, the solution, and the market

• Discover new insights and opportunities that we might have missed otherwise

• Optimise our product features and value proposition based on what works and what doesn’t

• Build trust and credibility with our stakeholders by showing evidence of our progress and results

• Focus on delivering products to create value for customer and the business

• Adapt to changing customer needs and market conditions

• Avoid wasting time and resources on building products that nobody wants or needs

Popular Frameworks

The Lean Startup methodology, pioneered by Eric Ries.

The Lean Startup is based on the idea of building a minimum viable product (MVP) that delivers the core value proposition to a subset of early adopters, and then measuring and learning from their feedback.

The feedback loop consists of three steps: build, measure, and learn.

The goal is to iterate rapidly and pivot when necessary, based on validated learning.

Design Thinking, popularized by IDEO and Stanford d.school.

Design Thinking is a human-centered process that focuses on understanding the needs and emotions of the users, and then ideating and prototyping solutions that address those needs.

The process consists of five stages: empathize, define, ideate, prototype, and test.

The goal is to create products that are desirable, feasible and viable.

Growth Hacking, coined by Sean Ellis.

Growth Hacking is a data-driven strategy that aims to optimize the growth of a product or service by finding the most effective ways to acquire, retain and monetize users.

The process involves identifying the key metrics that indicate growth (such as user acquisition, activation, retention, referral, and revenue), and then running experiments to improve those metrics.

The goal is to achieve product-market fit and scalable growth.

Popular Methods and Approaches

There are many methods and approaches for brute force experimentation that product managers can use depending on the goals, context, and resources. Some of the most popular ones are:

A/B testing: A/B testing is a method of comparing two versions of a product or a feature to see which one performs better on a specific metric or outcome. For example, product managers can use A/B testing to compare different headlines, colours, layouts, or buttons on a website or an app to see which one generates more clicks, conversions, or engagement.

Multivariate testing: Multivariate testing is a method of comparing multiple variations of a product or a feature at the same time to see which combination performs best on a specific metric or outcome. For example, product managers can use multivariate testing to compare different combinations of headlines, images, and calls to action (CTA) on a landing page to see which one generates more sign-ups or sales.

Prototyping: Prototyping is a method of creating a simplified or partial version of a product or a feature to test its feasibility, usability, or desirability with potential users or customers. For example, product managers can use prototyping to create mockups, wireframes, sketches, or interactive demos of a new product or feature to get feedback from users or customers before building the full version.

User testing: User testing is a method of observing how users or customers interact with a product or a feature in a realistic setting or scenario. For example, product managers can use user testing to watch how users or customers use a product or feature in their homes, offices, or other environments to understand their needs, pain points, behaviours, and emotions.

Beta testing: Beta testing is a method of releasing a new or improved version of a product or a feature to a limited group of users or customers before launching it to the public. For example, product managers can use beta testing to get feedback from early adopters or loyal customers on a new or improved product or feature before rolling it out to the wider market.

Adopting Brute-Force Experimentation

Despite the benefits, Brute-Force Experimentation brings its own set of challenges and limitations. It can be time-consuming, costly, and risky to try many different options without a clear direction or criteria. It can also lead to suboptimal results or missed opportunities if the product manager does not have a good way of evaluating and learning from the experiments.

Therefore, product managers who want to learn and adopt brute-force experimentation need to follow some best practices and principles. Some tips to get started with learning and adopting brute force experimentation:

Define goal and scope: Define clear and measurable goals and hypotheses for their products or features. For example, product managers can use the SMART framework (Specific, Measurable, Achievable, Relevant, Time-bound) to define their goals and hypotheses for their products or features.

Prioritize and iterate: We don’t have to try everything at once. We can use some criteria or heuristics to prioritize the most promising or feasible options and test them first. Choose the most appropriate method or approach for their experiments based on their goals, context, and resources. Then, we can use the feedback and data from the experiments to refine our options and test them again. This way, we can gradually converge to the best solution or discover new insights along the way.

For example, product managers can use the RITE framework (Rapid Iterative Testing and Evaluation) to choose the most suitable method or approach for their experiments based on their research questions, target audience, data sources, and time constraints.

Design and execute: We need to have a way of measuring the outcomes and impacts of the experiments with rigour and validity. Product managers can use the PIE framework (Potential, Importance, Ease) to prioritize their experiments based on their expected impact, relevance, and difficulty.

Analyse and interpret: The results must be measured with objectivity and honesty to understand what worked and what did not. It is imperative to have a mindset of learning from both successes and failures. Product managers can use the HEART framework (Happiness, Engagement, Adoption, Retention, Task success) to measure their results based on user-centric metrics that reflect and evaluate any aspects of it user experience.

Brute-force experimentation is not a magic bullet that can solve any problem. It is a tool that can help product managers explore, validate, and solve problems in an empirical and iterative way. Brute force experimentation is not only a skill but also a mindset that product managers need to cultivate and practice. By following these tips, one can learn and adopt brute-force experimentation in a more effective and efficient way.