From Speech to Data: Unraveling Google's Use of Voice Data for User Profiling (2403.05586v1)
Abstract: Smart home voice assistants enable users to conveniently interact with IoT devices and perform Internet searches; however, they also collect the voice input that can carry sensitive personal information about users. Previous papers investigated how information inferred from the contents of users' voice commands are shared or leaked for tracking and advertising purposes. In this paper, we systematically evaluate how voice itself is used for user profiling in the Google ecosystem. To do so, we simulate various user personas by engaging with specific categories of websites. We then use \textit{neutral voice commands}, which we define as voice commands that neither reveal personal interests nor require Google smart speakers to use the search APIs, to interact with these speakers. We also explore the effects of the non-neutral voice commands for user profiling. Notably, we employ voices that typically would not match the predefined personas. We then iteratively improve our experiments based on observations of profile changes to better simulate real-world user interactions with smart speakers. We find that Google uses these voice recordings for user profiling, and in some cases, up to 5 out of the 8 categories reported by Google for customizing advertisements are altered following the collection of the voice commands.
- Faqs on privacy: Google nest. https://support.google.com/googlenest/answer/9415830.
- Google assistant, your own personal Google. https://assistant.google.com/.
- Google My Ad Center. https://myadcenter.google.com/home.
- Market share of global smart speaker shipments from 3rd quarter 2016 to 1st quarter 2022, by vendor. https://www.statista.com/statistics/792604/worldwide-smart-speaker-market-share/#:~:text=Amazon%20is%20the%20leading%20vendor,percent%20in%20the%20same%20quarter.
- Nest smart speakers & home audio systems. https://store.google.com/us/category/nest_speakers?hl=en-US.
- Teach google assistant to recognize your voice with voice match. https://support.google.com/assistant/answer/9071681?hl=en&co=GENIE.Platform%3DAndroid&oco=0.
- Text to speech - openai api. https://platform.openai.com/docs/guides/text-to-speech.
- What is alexa voice id? https://www.amazon.com/gp/help/customer/display.html?nodeId=GYCXKY2AB2QWZT2X.
- Internet of things market analysis forecasts, 2020–2030. In 2020 Fourth World Conference on smart trends in systems, security and sustainability (WorldS4), pages 449–453. IEEE, 2020.
- Nancy Bonvillain. Language, culture, and communication: The meaning of messages. Rowman & Littlefield, 2019.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018.
- Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 19(4):788–798, 2010.
- When speakers are all ears: Characterizing misactivations of iot smart speakers. Proceedings on Privacy Enhancing Technologies, 2020(4), 2020.
- Online tracking: A 1-million-site measurement and analysis. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 1388–1401, 2016.
- Your echos are heard: Tracking, profiling, and ad targeting in the amazon smart speaker ecosystem. arXiv preprint arXiv:2204.10920, 2022.
- Fingerprinting the fingerprinters: Learning to detect browser fingerprinting behaviors. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1143–1161. IEEE, 2021.
- Voice-based determination of physical and emotional characteristics of users, October 9 2018. US Patent 10,096,319.
- Privacy implications of voice and speech analysis–information disclosure by inference. Privacy and Identity Management. Data for Better Living: AI and Privacy: 14th IFIP WG 9.2, 9.6/11.7, 11.6/SIG 9.2. 2 International Summer School, Windisch, Switzerland, August 19–23, 2019, Revised Selected Papers 14, pages 242–258, 2020.
- Alexa, are you listening? privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on human-computer interaction, 2(CSCW):1–31, 2018.
- Privacy attitudes of smart speaker users. Proceedings on Privacy Enhancing Technologies, 2019(4), 2019.
- Approaching automatic recognition of emotion from voice: A rough benchmark. In ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000.
- Recent trends in deep learning based personality detection. Artificial Intelligence Review, 53:2313–2339, 2020.
- Owning and sharing: Privacy perceptions of smart speaker users. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–29, 2021.
- Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022.
- Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017.
- Speech2face: Learning the face behind a voice. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7539–7548, 2019.
- Age recognition from voice. Journal of speech and hearing Research, 9(2):273–277, 1966.
- Healthline: Speech-based access to health information by low-literate users. In 2007 international conference on information and communication technologies and development, pages 1–9. IEEE, 2007.
- Rita Singh. Profiling humans from their voice, volume 41. Springer, 2019.
- X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5329–5333. IEEE, 2018.
- Joseph Turow. The voice catchers: How marketers listen in to exploit your feelings, your privacy, and your wallet. Yale University Press, 2021.
- Face reconstruction from voice using generative adversarial networks. Advances in neural information processing systems, 32, 2019.
- Sarah Myers West. Data capitalism: Redefining the logics of surveillance and privacy. Business & society, 58(1):20–41, 2019.