Using your voice to shop
Inaudible Tones
Through the use of Alexa’s voice feature, we wanted to bring the ability to ingress to Amazon Live’s Immersive Player experience.
Product Design
New Feature
0 to 1
Timeline
Design to Development: Dec 2022 - Mar 2023
Launch: Aug 2023
Deprecation (Sunset): Oct 2023
Collaborators
Kari McMinn, Senior Product Manager
Ryder He, Software Development Engineer
Overall impact
Launched on Amazon.com for both iOS and Android for the 10-foot experience (TV)
My contributions
Sole designer in designing and creating an ingress from your Fire TV straight to your phone through Amazon Alexa within the Amazon App.
TL;DR context
As a first step in having to explore how to better connect customers to a shoppable livestream, my product manager and I developed a plan to help in implementing Alexa’s voice feature from the mobile app to carefully pick up any active video from Amazon Live’s livestream catalogue that was being displayed on the Fire TV to then connect and drop customers in to shop the full carousel of items that are being presented in that very moment.
Despite efforts being made, this experience ultimately had been sunsetted due to having very minimal impact to their shopping journey. However, it was able to introduce the opportunity in how we could better expand our ideas in a much more granular way through Shop-The-Show.
The context
New feature, who?
In July 2022, the Amazon Live (AL) team aligned with C&S organization under Lab126 to develop a unique shopping experience for AL customers, so they could shop & interact with content viewed on a living room device using a companion device.
Since then, the team has explored different transport mechanisms that can detect and decipher data embedded in streaming media, and have developed a feature in the ALFTV app that enables customers to bring up a related companion shopping experience in the Amazon Shopping app using inaudible tone transport technology.
What are the challenges?
Purchasing products discovered on TV was a frustrating experience.
Customers were primarily searching for products via Amazon or Google related to what they’ve seen on TV, but struggled to describe, find, and buy the product they want to shop.
There are many challenges that come with browsing products and completing checkout on TV including:
- Disruptions to the video streaming experience
- Difficulties using a remote control to navigate through many steps
- Consuming text and image based product information on a TV screen
Additionally, content streamed on living room devices are often communal viewing experiences, where only one viewer may want to engage with shoppable or other interactive content. With over 95% of customers using a second digital device while watching TV, there was an opportunity to create a companion mobile experience that eliminated these points of friction.
Disruptions to the video streaming experience
Difficulties using a remote control to navigate through many steps
Consuming text and image based product information on a TV screen
The designs
Ease your experience through sound.
To kickstart the concepts of shopping from your companion device, we’ve partnered up with Amazon Alexa team to help in ensuring that through livestream videos being played on your screen, the inaudible tones will help to pick up the correct video to be displayed on your mobile device.
Videos come with Inaudible Tones. Follow the instructions to access!
Tap the microphone icon on the home state to access this screen.
Or if you have the Inaudible Tones enabled, just type in “Shop the stream” to automatically detect!
Worried about privacy and doubting us? Feel free to disable.
(We’ll never hear you without your permission)
Results & Takeaways
Sigh. Didn’t work out, but a lot of learnings!
After launch, we’ve learned quite a few things, but to keep it short... it didn’t work out well.
We chose to deprecate and dial down this function in October 2023. When customers type "show the stream" in mShop app, they will no longer land on the Shop-the-Show experience. On TV, ALFTV app customers will no longer see the tutorial educating them about the companion experience and how to access. We have also completed dialing down the inaudible detection service in mShop, removed the FAQ from Fire TV FAQ’s, and have deprecated creator workflows to ingest tones.
What have we learned from Inaudible Tones on Amazon Live Fire TV App?One of the hot topics we debated prior to launch was whether there would be customer backlash around embedding inaudible tones (high frequency tones) into our ALFTV video content, given there was a known risk from our internal testing that a small percentage of customer’s might be able to hear them. To mitigate this risk, we aligned with legal, PR and business stakeholders to include messaging on the tutorial that says “This video includes inaudible tones to support this experience” with an option to turn tones off in settings. While we have received no customer service complaints or press hits on this issue, our engagement data does show that some customers are interested in turning Inaudible Tones off when they learn about the feature.
3.8% of customers who see the Inaudible Tones tutorial tap it to navigate to settings, and 2.6% of all customers watching previously live videos turn inaudible tones settings off, which is higher than the 0.05% guardrail threshold we set for that metric at launch.
What does this mean for customers?We believe that this will have minimal impact on the existing Amazon Live Fire TV app customer base.
In 2024, we’ve had 294 avg monthly visits to the ALFTV app second-screen experience, a 0.81% conversion rate of visits to the companion experience while watching a previously live video, and a 3.4% conversion rate of customers who visit the companion experience making a purchase. These companion visits generate a monthly average of 10 sales, $333.29 Direct OPS, and $33.32 avg price per unit sold.
Within the mShop inaudible tone detection experience, customers who opened the inaudible detection page abandoned the experience before seeing a success or failure 43% of the time.
For all customers who did not abandon the page, on average 66% of those visits were successful in leading customers to the companion experience and 34% of all visits failed, resulting in a customer reaching the re-try experience after 15 seconds.
For customers who were successful in being led to the companion experience, there was an average 4.8 second latency before a tone was successfully identified. These customers will continue to be able to shop the Amazon Live streams they’re watching using their remote and completing checkout within the Fire TV app or by using an on-screen QR code to navigate directly to a product detail page on mobile.
Why have we deprecated this feature?The key driver for this decision was the KTLO (Keep The Lights On) investments we needed to make to maintain this experience for customers. To accommodate QoS (Quality of Service) updates for the Amazon Live video ingestion stack, we would have had to invest resources in rebuilding the tone ingestion workflows to avoid disruptions to the customer experience. We can now dedicate those resources to focus on our goals to scale Shop the Show for Prime Video and Amazon Live FAST Channel. Additionally, our team is focusing on alternative technologies to support tv to phone synching, with no current plans to further invest in the inaudible tones tech stack.
What alternative paths did we consider?The team considered replacing the current CX with a new Shop-the-Show companion experience that uses account auth to sync the TV to phone so that ALFTV app customers could continue shopping via a companion experience. We decided against replacing given we have no reason to believe we would see greater engagement than the current CX, and we don’t believe the investment of building this experience is worth the >300 MAU, $333.29 Direct OPS.
Additionally, to replace we would need to do the work by Q3/Q4 to avoid KTLO investments in the inaudible tones experience. This is not a one-way door decision and we could decide to make this investment at any point after dialing down the IAT experience. By 8-31 we plan to present a document with a perspective on the ALFTV app’s ROI potential if continued KTLO and feature improvement investments are made.
Using your voice to shop
Inaudible Tones
Through the use of Alexa’s voice feature, we wanted to bring the ability to ingress to Amazon Live’s Immersive Player experience.
Product Design
New Feature
0 to 1
Timeline
Design to Development: Dec 2022 - Mar 2023
Launch: Aug 2023
Deprecation (Sunset): Oct 2023
Collaborators
Kari McMinn, Senior Product Manager
Ryder He, Software Development Engineer
Overall impact
Launched on Amazon.com for both iOS and Android for the 10-foot experience (TV)
My contributions
Sole designer in designing and creating an ingress from your Fire TV straight to your phone through Amazon Alexa within the Amazon App.
TL;DR context
As a first step in having to explore how to better connect customers to a shoppable livestream, my product manager and I developed a plan to help in implementing Alexa’s voice feature from the mobile app to carefully pick up any active video from Amazon Live’s livestream catalogue that was being displayed on the Fire TV to then connect and drop customers in to shop the full carousel of items that are being presented in that very moment.
Despite efforts being made, this experience ultimately had been sunsetted due to having very minimal impact to their shopping journey. However, it was able to introduce the opportunity in how we could better expand our ideas in a much more granular way through Shop-The-Show.
The context
New feature, who?
In July 2022, the Amazon Live (AL) team aligned with C&S organization under Lab126 to develop a unique shopping experience for AL customers, so they could shop & interact with content viewed on a living room device using a companion device.
Since then, the team has explored different transport mechanisms that can detect and decipher data embedded in streaming media, and have developed a feature in the ALFTV app that enables customers to bring up a related companion shopping experience in the Amazon Shopping app using inaudible tone transport technology.
What are the challenges?
Purchasing products discovered on TV was a frustrating experience.
Customers were primarily searching for products via Amazon or Google related to what they’ve seen on TV, but struggled to describe, find, and buy the product they want to shop.
There are many challenges that come with browsing products and completing checkout on TV including:
- Disruptions to the video streaming experience
- Difficulties using a remote control to navigate through many steps
- Consuming text and image based product information on a TV screen
Additionally, content streamed on living room devices are often communal viewing experiences, where only one viewer may want to engage with shoppable or other interactive content. With over 95% of customers using a second digital device while watching TV, there was an opportunity to create a companion mobile experience that eliminated these points of friction.
Disruptions to the video streaming experience
Difficulties using a remote control to navigate through many steps
Consuming text and image based product information on a TV screen
The designs
Ease your experience through sound.
To kickstart the concepts of shopping from your companion device, we’ve partnered up with Amazon Alexa team to help in ensuring that through livestream videos being played on your screen, the inaudible tones will help to pick up the correct video to be displayed on your mobile device.
Videos come with Inaudible Tones. Follow the instructions to access!
Tap the microphone icon on the home state to access this screen.
Or if you have the Inaudible Tones enabled, just type in “Shop the stream” to automatically detect!
Worried about privacy and doubting us? Feel free to disable.
(We’ll never hear you without your permission)
Results & Takeaways
Sigh. Didn’t work out, but a lot of learnings!
After launch, we’ve learned quite a few things, but to keep it short... it didn’t work out well.
We chose to deprecate and dial down this function in October 2023. When customers type "show the stream" in mShop app, they will no longer land on the Shop-the-Show experience. On TV, ALFTV app customers will no longer see the tutorial educating them about the companion experience and how to access. We have also completed dialing down the inaudible detection service in mShop, removed the FAQ from Fire TV FAQ’s, and have deprecated creator workflows to ingest tones.
What have we learned from Inaudible Tones on Amazon Live Fire TV App?One of the hot topics we debated prior to launch was whether there would be customer backlash around embedding inaudible tones (high frequency tones) into our ALFTV video content, given there was a known risk from our internal testing that a small percentage of customer’s might be able to hear them. To mitigate this risk, we aligned with legal, PR and business stakeholders to include messaging on the tutorial that says “This video includes inaudible tones to support this experience” with an option to turn tones off in settings. While we have received no customer service complaints or press hits on this issue, our engagement data does show that some customers are interested in turning Inaudible Tones off when they learn about the feature.
3.8% of customers who see the Inaudible Tones tutorial tap it to navigate to settings, and 2.6% of all customers watching previously live videos turn inaudible tones settings off, which is higher than the 0.05% guardrail threshold we set for that metric at launch.
What does this mean for customers?We believe that this will have minimal impact on the existing Amazon Live Fire TV app customer base.
In 2024, we’ve had 294 avg monthly visits to the ALFTV app second-screen experience, a 0.81% conversion rate of visits to the companion experience while watching a previously live video, and a 3.4% conversion rate of customers who visit the companion experience making a purchase. These companion visits generate a monthly average of 10 sales, $333.29 Direct OPS, and $33.32 avg price per unit sold.
Within the mShop inaudible tone detection experience, customers who opened the inaudible detection page abandoned the experience before seeing a success or failure 43% of the time.
For all customers who did not abandon the page, on average 66% of those visits were successful in leading customers to the companion experience and 34% of all visits failed, resulting in a customer reaching the re-try experience after 15 seconds.
For customers who were successful in being led to the companion experience, there was an average 4.8 second latency before a tone was successfully identified. These customers will continue to be able to shop the Amazon Live streams they’re watching using their remote and completing checkout within the Fire TV app or by using an on-screen QR code to navigate directly to a product detail page on mobile.
Why have we deprecated this feature?The key driver for this decision was the KTLO (Keep The Lights On) investments we needed to make to maintain this experience for customers. To accommodate QoS (Quality of Service) updates for the Amazon Live video ingestion stack, we would have had to invest resources in rebuilding the tone ingestion workflows to avoid disruptions to the customer experience. We can now dedicate those resources to focus on our goals to scale Shop the Show for Prime Video and Amazon Live FAST Channel. Additionally, our team is focusing on alternative technologies to support tv to phone synching, with no current plans to further invest in the inaudible tones tech stack.
What alternative paths did we consider?The team considered replacing the current CX with a new Shop-the-Show companion experience that uses account auth to sync the TV to phone so that ALFTV app customers could continue shopping via a companion experience. We decided against replacing given we have no reason to believe we would see greater engagement than the current CX, and we don’t believe the investment of building this experience is worth the >300 MAU, $333.29 Direct OPS.
Additionally, to replace we would need to do the work by Q3/Q4 to avoid KTLO investments in the inaudible tones experience. This is not a one-way door decision and we could decide to make this investment at any point after dialing down the IAT experience. By 8-31 we plan to present a document with a perspective on the ALFTV app’s ROI potential if continued KTLO and feature improvement investments are made.
Using your voice to shop
Inaudible Tones
Through the use of Alexa’s voice feature, we wanted to bring the ability to ingress to Amazon Live’s Immersive Player experience.
Product Design
New Feature
0 to 1
Timeline
Design to Development: Dec 2022 - Mar 2023
Launch: Aug 2023
Deprecation (Sunset): Oct 2023
Collaborators
Kari McMinn, Senior Product Manager
Ryder He, Software Development Engineer
Overall impact
Launched on Amazon.com for both iOS and Android for the 10-foot experience (TV)
My contributions
Sole designer in designing and creating an ingress from your Fire TV straight to your phone through Amazon Alexa within the Amazon App.
TL;DR context
As a first step in having to explore how to better connect customers to a shoppable livestream, my product manager and I developed a plan to help in implementing Alexa’s voice feature from the mobile app to carefully pick up any active video from Amazon Live’s livestream catalogue that was being displayed on the Fire TV to then connect and drop customers in to shop the full carousel of items that are being presented in that very moment.
Despite efforts being made, this experience ultimately had been sunsetted due to having very minimal impact to their shopping journey. However, it was able to introduce the opportunity in how we could better expand our ideas in a much more granular way through Shop-The-Show.
The context
New feature, who?
In July 2022, the Amazon Live (AL) team aligned with C&S organization under Lab126 to develop a unique shopping experience for AL customers, so they could shop & interact with content viewed on a living room device using a companion device.
Since then, the team has explored different transport mechanisms that can detect and decipher data embedded in streaming media, and have developed a feature in the ALFTV app that enables customers to bring up a related companion shopping experience in the Amazon Shopping app using inaudible tone transport technology.
What are the challenges?
Purchasing products discovered on TV was a frustrating experience.
Customers were primarily searching for products via Amazon or Google related to what they’ve seen on TV, but struggled to describe, find, and buy the product they want to shop.
There are many challenges that come with browsing products and completing checkout on TV including:
- Disruptions to the video streaming experience
- Difficulties using a remote control to navigate through many steps
- Consuming text and image based product information on a TV screen
Additionally, content streamed on living room devices are often communal viewing experiences, where only one viewer may want to engage with shoppable or other interactive content. With over 95% of customers using a second digital device while watching TV, there was an opportunity to create a companion mobile experience that eliminated these points of friction.
Disruptions to the video streaming experience
Difficulties using a remote control to navigate through many steps
Consuming text and image based product information on a TV screen
The designs
Ease your experience through sound.
To kickstart the concepts of shopping from your companion device, we’ve partnered up with Amazon Alexa team to help in ensuring that through livestream videos being played on your screen, the inaudible tones will help to pick up the correct video to be displayed on your mobile device.
Videos come with Inaudible Tones. Follow the instructions to access!
Tap the microphone icon on the home state to access this screen.
Or if you have the Inaudible Tones enabled, just type in “Shop the stream” to automatically detect!
Worried about privacy and doubting us? Feel free to disable.
(We’ll never hear you without your permission)
Results & Takeaways
Sigh. Didn’t work out, but a lot of learnings!
After launch, we’ve learned quite a few things, but to keep it short... it didn’t work out well.
We chose to deprecate and dial down this function in October 2023. When customers type "show the stream" in mShop app, they will no longer land on the Shop-the-Show experience. On TV, ALFTV app customers will no longer see the tutorial educating them about the companion experience and how to access. We have also completed dialing down the inaudible detection service in mShop, removed the FAQ from Fire TV FAQ’s, and have deprecated creator workflows to ingest tones.
What have we learned from Inaudible Tones on Amazon Live Fire TV App?One of the hot topics we debated prior to launch was whether there would be customer backlash around embedding inaudible tones (high frequency tones) into our ALFTV video content, given there was a known risk from our internal testing that a small percentage of customer’s might be able to hear them. To mitigate this risk, we aligned with legal, PR and business stakeholders to include messaging on the tutorial that says “This video includes inaudible tones to support this experience” with an option to turn tones off in settings. While we have received no customer service complaints or press hits on this issue, our engagement data does show that some customers are interested in turning Inaudible Tones off when they learn about the feature.
3.8% of customers who see the Inaudible Tones tutorial tap it to navigate to settings, and 2.6% of all customers watching previously live videos turn inaudible tones settings off, which is higher than the 0.05% guardrail threshold we set for that metric at launch.
What does this mean for customers?We believe that this will have minimal impact on the existing Amazon Live Fire TV app customer base.
In 2024, we’ve had 294 avg monthly visits to the ALFTV app second-screen experience, a 0.81% conversion rate of visits to the companion experience while watching a previously live video, and a 3.4% conversion rate of customers who visit the companion experience making a purchase. These companion visits generate a monthly average of 10 sales, $333.29 Direct OPS, and $33.32 avg price per unit sold.
Within the mShop inaudible tone detection experience, customers who opened the inaudible detection page abandoned the experience before seeing a success or failure 43% of the time.
For all customers who did not abandon the page, on average 66% of those visits were successful in leading customers to the companion experience and 34% of all visits failed, resulting in a customer reaching the re-try experience after 15 seconds.
For customers who were successful in being led to the companion experience, there was an average 4.8 second latency before a tone was successfully identified. These customers will continue to be able to shop the Amazon Live streams they’re watching using their remote and completing checkout within the Fire TV app or by using an on-screen QR code to navigate directly to a product detail page on mobile.
Why have we deprecated this feature?The key driver for this decision was the KTLO (Keep The Lights On) investments we needed to make to maintain this experience for customers. To accommodate QoS (Quality of Service) updates for the Amazon Live video ingestion stack, we would have had to invest resources in rebuilding the tone ingestion workflows to avoid disruptions to the customer experience. We can now dedicate those resources to focus on our goals to scale Shop the Show for Prime Video and Amazon Live FAST Channel. Additionally, our team is focusing on alternative technologies to support tv to phone synching, with no current plans to further invest in the inaudible tones tech stack.
What alternative paths did we consider?The team considered replacing the current CX with a new Shop-the-Show companion experience that uses account auth to sync the TV to phone so that ALFTV app customers could continue shopping via a companion experience. We decided against replacing given we have no reason to believe we would see greater engagement than the current CX, and we don’t believe the investment of building this experience is worth the >300 MAU, $333.29 Direct OPS.
Additionally, to replace we would need to do the work by Q3/Q4 to avoid KTLO investments in the inaudible tones experience. This is not a one-way door decision and we could decide to make this investment at any point after dialing down the IAT experience. By 8-31 we plan to present a document with a perspective on the ALFTV app’s ROI potential if continued KTLO and feature improvement investments are made.