A/B Message Testing for Alexa Smart Home on amazon.com

Background: Teams across Amazon Devices historically used Alexa utterances as a way to help customers better understand how to use their device, however, they also used them in float-all-boats messaging across marketing channels, including Amazon’s homepage.

Insight: Alexa utterances may cater to customers who are more familiar with Alexa, rather than customers who are less far along in their journey who are browsing Amazon’s homepage.

Hypothesis: If we use value propositions, rather than Alexa utterances, then customers will be more likely to click and purchase Amazon Smart Home devices from homepage.

Work:

Echo Glow A/B message test

Amazon Smart Thermostat A/B/C/D message test

Amazon Smart Plug A/B/C/D message test

Amazon Smart Plug A/B message test

Results and Impact:

Alternative headlines to Alexa utterances drove statistically significant positive results with a XX% win-rate, average +XX% increase in sales, and +XXXK incremental sales for Amazon Smart Home devices across US, CA, and MX. These results helped shape Amazon Smart Home’s messaging strategy across homepage, paid social media, and display advertising, and other offsite marketing channels.