How to Design a Comprehension Test

In a previous post we discussed how much research you may have to execute to fulfil your obligations as part of the Financial Conduct Authority’s new Consumer Duty regulations. If you’re a retail financial business with more than 1,000 customers, then there’s a fairly good chance you will need to run comprehension tests for your customers.

The comprehension test has been specifically suggested in the FCA guidance. In this form of research, participants are presented with the same information that is shown to buyers of your products and a series of questions is then asked to determine how much they understood and recalled. For the FCA the key element is that customers or potential customers understand the value of product or service you are trying to sell them.

As one of the key elements of the Consumer Duty testing, what do you need to consider when conducting your own comprehension tests?

Screening

Before you get to the comprehension test stage you will have defined your target audience for the product or service you’re testing. For the FCA regulations you don’t need to test comprehension with real life customers, it’s enough that the profile of research participants match the demographic profile of your target audience.

Recreate the Context

It’s important that you recreate the context of the communication as accurately as possible. unintentional biases can sometimes be found in unlikely places. Anything that might add to cognitive load or emotional framing on a website or through a one-on-one call can affect the comprehension of a product’s features or value. If you can, recreating the digital customer journey can be a vital way in ensuring you are getting a deep and fundamental understanding of comprehension. If you’re testing customer support incidents, it’s much more effective to test in the format it would occur in real life, whether that’s digitally or a one-on-one call.

Test Recall

Recollection should provide you with the overall guide of whether the customer fully understood the product and its value. Between their exposure to the product test and recollection you will want to add a distraction task. This will minimize the participant’s rehearsal of the material and ensures that it is not currently stored in their working memory. The next step is a survey that looks to explore the participants understanding of the product and a recollection of key features and conditions.

Understanding Subgroups

Once the comprehension testing has been finished it’s important to analyse the data not as a homogenous group, but in a way that identities subgroups, particularly those that might be considered vulnerable. It’s very unlikely that everyone will comprehend in the same way, so any changes you will need to make to the communications will need to be adjusted for different groups.

The Score

A comprehension test produces a score for each document or piece of information that you include, as well as an overall score for your product as a whole. The FCA has not laid down a specific threshold, but our recommendation is that a score of 60 (out of 100) reflects a good level of understanding for most markets. If any of your documents don’t reach this score, behavioural science provides a ‘toolkit’ of changes that will improve comprehension for most documents: adding a schema, highlighting and colour-coding key information, and a number of other tools.

The comprehension test is one of four pillars of the new Consumer Duty regulations. At its core, it’s testing how well customers understand the value your products provide: that includes both their positive benefits, and any exclusions or exceptions – and how well the product meets the customer’s needs. The FCA has explicitly stated that just giving a customer all the information in one format or another is not enough, you need to ensure your communications are fair, clear and not misleading for all your customers.

If you’re ready to conduct comprehension tests with your customers, Irrational Agency has a comprehension test platform built and ready to use. You can just drop in your own documents, define your audience and get your comprehension score within 1-2 weeks. Contact us at hello@irrationalagency.com for a demo and to discuss how this can help you get ahead of the timeline and comply with the Consumer Duty in plenty of time.