Why Test the Design with Only 5 Users

The reasoning behind small sample sizes in qualitative usability research.

The sufficiency and thereby reliability of findings derived from testing a feature with just 5 users is a common concern that various stakeholders of the design process share. Before I can delve into justifying the sample size for feature testing, there are few facets of research I would like share.

The problem you are trying to solve defines the research method. The method defines the sample size.

Depending on the problem (stage of the design process), user researchers suggest an appropriate methodology that is best suited to uncover insights. For example, if we want to understand what soap people are using, a survey (quantitative) is a recommended approach to reach a large audience in a short span of time. To understand why they use the soap, running a survey alone will not be enough. Beyond the common factors of price, branding, flavour of the soap, there may be others that require deeper understanding (For example, the way the soaps are stored in the shelves of the market). In a contextual inquiry (qualitative), participants are observed in their natural environment over a large period of time from pre purchase to post purchase. As this takes a longer time, we would have a smaller sample size. What the exact number is for a sample size depends on a lot of factors, which is beyond the scope of this paper.

Some common research methods that come to mind

As with everything else in life — budget, time and resources affect the ability to conduct more research and thereby with more users.

If you work in a setup that is truly agile, all functions need to iterate on small changes and iterate continuously. This is no different for research. The minimum viable product (MVP) for research is to get as many insights as possible (return), with the least amount of time, resources and participants that are required to deliver reliable and valid results (quality) that are actionable. By stretching any of the above mentioned levers, the research project as a whole would be affected. Identifying the right fit without compromising on the integrity of the research is a challenge researchers face every day as do designers, product managers and engineers in their respective domains.

In an ideal world, we could test with many participants over many days and find as many insights as possible. However, this would slow down and possibly stagnate the design process.

MVP for research is something researchers think about

The Magical (or not) Number 5

In the year 1993, renowned usability expert, Jakob Nielsen, published a scientific paper to describe his results from analysing multiple usability studies. He established that the probability of identifying more usability issues in the design reduces when you add more participants. With a sample of 5 users, you can uncover 75–80 % of the usability issues in the design. In an agile setup, the aim is to repeat the tests of 5 with iterations of design. The more users you add, the less value you will get in return.

Nielsen's hypothesis on return value of sample size

When can We Test with 5 Users?

• The research methodology is usability testing only that is qualitative in nature.
• Testing flows, interactions and visuals of the same set of features. (not 2 different apps)

When can We Not Test with Just 5 Users?

• You are trying to understand opinions and attitudes which would be better addressed via a survey, requiring larger sample sizes.
• You are trying to predict future behaviour via data modelling, A/B testing, which cannot be run with 5 users.
• You have distinct user segments such as buyers and sellers on an e-commerce app. You will need to consider these two segments as separate.

With 5 users in usability testing, we are not testing the success of the design rather identify possible failures before going to market. It does not uncover “all” issues that people may face, just the more probable. The approach is a qualitative one, different from quant methods.

Multiple Approaches to Solving the Same Problem

Various stakeholders have different approaches. While engineers may think in terms of technical infrastructure, performance and risk, marketing leaders need to think of outreach and consumer acquisition among other things. Business and product leaders need to build the case for the need in the market for a new feature and designers/researchers translate the business vision into tangible outcomes while data driven people measure the success of everyone’s efforts. Although, a simplified version of the roles of stakeholders, the essence is that everyone is working towards solving the same problem. Understanding each other’s approaches and the value they bring will help us be collaborative and give a deeper meaning to the shared vision and success of a product.

This post was first published on Medium.