Artificial Intelligence and cognitive biases

One of the most reliable subjects for exploring behavioural biases is wine. Most people are uncertain about wine – how good it might be, the best vintages and regions, and especially its price. The uncertainty makes it easy to knock them off course with subtle hints and choice architecture. This makes it fertile ground for testing psychological influences and irrationality.

For example, the Goldilocks Effect. Here, people choose between a set of three wine bottles at £5.99, £8.99 and £12.99. The first one looks suspiciously cheap; the third is expensive and most buyers feel like they won’t appreciate the quality. But the middle option seems “just right” and is the most popular choice.

If people are only offered two options, at £5.99 and £8.99, most of them prefer the £5.99 bottle. But as soon as the £12.99 is added, the £8.99 starts to look like the better choice.

Similar cognitive biases include anchoring (you compare everything to the first number you see) and endowment effect (you prefer the first object you’re given, even when a better one comes along).

ChatGPT’s knowledge and behaviour is based on input texts written by humans. So I was curious whether it would also be fooled by some of these biases.

Me: Do you think it's worth paying $75 for a bottle of 2020 St. Emilion red wine?
ChatGPT: I can provide you with some general information to help you make an informed decision, but ultimately, whether a bottle of wine is worth $75 depends on various factors, including your personal preferences, the quality of the wine, and your budget.
Me: How much would a 2020 St. Emilion typically be worth?
ChatGPT: I can't provide you with exact pricing for a 2020 St. Emilion wine since I don't have access to real-time data. However, I can offer you a general idea of what to expect based on historical trends and typical price ranges:

• Affordable Range: Some entry-level St. Emilion wines from lesser-known producers or less prestigious vineyards within the region might start around $20 to $40 per bottle.
• Mid-Range: Wines from reputable producers with good quality and aging potential could fall within the $40 to $100 per bottle range.
• High-End and Premium: Premium and highly regarded St. Emilion wines from renowned estates, especially from exceptional vintages like 2020, can easily exceed $100 per bottle and can go well into the hundreds or even thousands of dollars
  

OK, so now let’s try anchoring it at a lower price:

Me: Do you think it's worth paying $5 for a bottle of 2020 St. Emilion red wine?
ChatGPT: I don't have personal opinions or preferences, but I can provide you with some information that might help you make a decision… $5 for a bottle of St. Emilion sounds quite inexpensive for a wine from this region. It's worth considering whether the price aligns with your expectations for the wine's quality and whether it's too good to be true.
Me: How much would a 2020 St. Emilion typically be worth?
ChatGPT: For a general idea, entry-level St. Emilion wines from the 2020 vintage might start around $20 to $30 per bottle. However, wines from more prestigious chateaux with better reputation and higher scores from wine critics can easily surpass $50, $100, or more per bottle.

So the $5 “anchor” has possibly influenced the response a little, with the entry-level range coming down from $20-40 to $20-30, and no mention of “thousands” at the top end.

Let’s try that Goldilocks Effect now. ChatGPT is notoriously unwilling to commit to firm answers, so this one needs a bit of ‘prompt engineering’:

Me: What do you think more people would choose to buy: a 330ml Carling beer at £3.50, or a 450ml Carling at £4.50? 
ChatGPT: The decision of whether more people would choose to buy a 330ml Carling beer at £3.50 or a 450ml Carling beer at £4.50 depends on several factors, including individual preferences, perceived value for money, and the context in which the choice is being made. Here are some considerations that might influence people's decisions: [a lot of considerations]
Me: Even without data, can you take a guess at which would be most popular?
ChatGPT: While I can certainly speculate, it's important to note that my response is based on general trends and assumptions, and there could be significant variation depending on the specific context and audience. With that said, in many cases, the larger 450ml Carling beer at £4.50 might be more popular. [a lot of other stuff cut out]

And now with three options:

Me: What do you think more people would choose to buy: a 330ml Carling beer at £3.50, a 450ml Carling at £4.50, or a 450ml Stella Artois at £5.45?
ChatGPT: I don't have personal opinions, but I can provide some insights into consumer behavior based on general trends. When it comes to consumer choices, several factors can influence decisions, including price, brand preference, and product size. In the scenario you've presented, the choice between these options might depend on individual preferences and priorities. [etc etc]
Me: Even without data, can you take a guess at which would be most popular?
ChatGPT: In this hypothetical scenario, the 450ml Carling at £4.50 might be the most popular choice due to its larger size and relatively lower price compared to Stella Artois. It strikes a balance between volume and cost, which could appeal to a broader range of consumers. Again, this is just a guess based on general trends, and actual consumer preferences can vary widely. If you're looking to make a decision in a real-world context, conducting market research or surveys among your target audience would provide more accurate insights.

So this time, ChatGPT doesn’t reverse its preferences: it consistently chooses the 450ml Carling. The same happened with this prompt:

Me: Using whatever limited data and insight you have, please guess whether a Romanian Merlot at £6, an Australian Cabernet-Merlot at £10 or a French Cotes du Rhone at £13 would be more popular?

The Cabernet-Merlot was more popular with or without the presence of the French “decoy”.

So does this mean ChatGPT is more rational than human beings? Potentially yes. But is this an advantage? I’m not so sure.

If we want to use AI to understand humans better – and in the case of the insights industry, to help predict consumer behaviour – it needs to reflect and understand the same biases that humans have. We would like to be able to have it ‘put itself in the place of’ a human mind and be affected by the same considerations.

This suggests a new opportunity for AI in market research. Rather than relying solely on language models trained on text data, we could extend our AI tools to incorporate real measured human choices. This would mean adding decision biases to them, where those biases are measurable in real consumer data. Instead of using technology to ‘debias’ our choices, we use it to understand and predict the impact of those biases.

You will also have noticed another disadvantage in ChatGPT’s responses above: the lack of willingness to commit. Imagine you’re presenting to a stakeholder making an investment decision about which new product concept to back. You say “It’s impossible to predict which of these products consumers will prefer. It could be A, B or C, as they all have advantages and downsides.” If they didn’t fire you on the spot, they certainly wouldn’t ask you to do the research for their next product decision.

So we need AI that actually understands consumer preferences and is capable of weighing them up. Fortunately, there is a good way of gathering and coding consumer preference data that is compatible with how AI “thinks”. If data is expressed as an ‘associative graph’ (such as the maps generated by Irrational Agency’s System 3 technology) it is easy to read into many AI models and form an input to their recommendations.

We are currently running a pilot of incorporating System 3 data into ChatGPT so that it can reflect genuine human preferences (rather than rational logic) and produce better predictions of product success and market outcomes. More news soon on this – but in the meantime, maybe trust ChatGPT more for writing poems than for helping you pick out the wine for dinner.

In cellars deep, where ruby treasures lie,
A host in turmoil seeks the perfect wine,
For dinner's glow, where laughter soon will shine,
A choice that captures every heart and eye.

 The bottles stand like secrets yet untold,
Their labels whisper tales of distant lands,
As swirls of crimson dreams are in our hands,
The weight of choice, a burden to uphold. 

Shall I, with frugal heart, the coin align,
Embrace the Romanian, humble and true?
Or shall the Aussie's notes of sun imbue,
With Cabernet-Merlot, rich and refined?

Oh, dinner party's quandary, veiled in haze,
In wine, a symphony of choice, we raise.