Revolutionizing Audio Content Navigation: AI-Enhanced Multimodality and Machine Learning for Speaker Diarization and Topic Segmentation
DOI:
https://doi.org/10.46243/jst.2025.v10.i01.pp10-25Keywords:
Speaker Diarization,, Topic Segmentation,, , Artificial Intelligence(AI),, Machine Learning, Multimodal Processing,, , Natural Language Processing(NLP),, Content Navigation, Deep Learning.Abstract
The rapid evolution of digital media has propelled an increased
consumption of audio content, ranging from podcasts to educational lectures.
Despite its growing popularity, the inherent unstructured nature of audio media
poses significant challenges in navigation and user interaction. Our paper
introduces an innovative AI-driven framework designed to fundamentally
transform audio content exploration. Utilizing cutting-edge machine learning and
deep learning technologies, the system applies precise speaker diarization and
topic segmentation to radically improve navigation and content discovery in
audio streams. Furthermore, the incorporation of an interactive chat feature
enriches user interaction, allowing listeners to effortlessly query and jump
directly to specific content via intuitive voice and text commands. This advanced
system not only streamlines the audio exploration process but also personalizes
the listener experience by integrating multimodal interfaces and sophisticated
content annotation techniques. By addressing critical navigational inefficiencies,
this framework sets a new paradigm in personalized, structured, and interactive
media consumption, catering to the evolving demands of modern audio content
users.
References
Anguera X, Bozonnet S, Evans N, Fredouille C, Friedland G, Vinyals O. Speaker diarization: A review of recent research. IEEE Transactions on Audio, Speech, and Language Processing. 2012;20(2):356-370. doi:10.1109/ TASL.2011.2125954
Mao HH, Li S, McAuley J, Cottrell G. Speech recognition and multi-speaker diarization of long conversations. arXiv preprint. 2020;arXiv:2005.08072
Frølund, J., Løkke, A., Jensen, H., & Farver-Vestergaard, I. (2024). Development of Podcasts in a Hospital Setting: A User-Centered Approach. Journal of Health Communication, 29(4), 244–
https://doi.org/10.1080/10810730.2024. 2321385
Vallet F, Essid S, Carrive J. A multimodal approach to speaker diarization on TV talk-shows. IEEE Transactions on Multimedia. 2013;15(3):509-
doi:10.1109/TMM.2012.2233724
Wang S, Ning Z, Truong A, et al. PodReels: Human-AI co-creation of video podcast teasers. Proceedings of the 2024 Designing Interactive Systems Conference (DIS ’24). ACM. 2024;17 pages. doi:10.1145/3643834.3661591
Austin A, Samuel A. Enhancing podcasting by leveraging AI technologies. Communications of the ACM. 2023;66(10):48-55. doi:10.1145/3626767.3625304
Edison Research. Podcast Consumer Report. Edison Research. 2023
Kinnunen T, Li H. An overview of textindependent speaker recognition: From features to supervectors. Speech Communication. 2010;52(1):12-40. doi:10.1016/j. specom.2009.08.009
Chan-Olmsted, S., & Wang, R. (2022). Understanding podcast users: Consumption motives and behaviors. New Media & Society, 24(3), 684-704. https://doi. org/10.1177/1461444820963776
Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ‘18). Association for Computing Machinery, New York, NY, USA, Paper 640, 1–12. https://doi.org/10.1145/3173574.3174214
Iacopo Ghinassi, Lin Wang, Chris Newell, and Matthew Purver. 2023. Multimodal Topic Segmentation of Podcast Shows with Pre-trained Neural Encoders. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval (ICMR ‘23). Association for Computing Machinery, New York, NY, USA, 602–606. https://doi.org/10.1145/3591106.3592270
A. Hajavi and A. Etemad, “Fine-grained Early Frequency Attention for Deep Speaker Recognition,” 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy, 2022, pp. 1-6, doi: 10.1109/ IJCNN55064.2022.9892054
U. S. Jha, “Efficient multimodal signal processing engine ease communication, computing, and multimedia convergence,” 2005 IEEE International Conference on Personal Wireless Communications, 2005. ICPWC 2005., New Delhi, India, 2005, pp. 348-352, doi: 10.1109/ ICPWC.2005.1431364
Galli, Carlo, et al. “Topic Modeling for Faster Literature Screening Using Transformer-Based Embeddings.” Metrics. Vol. 1. No 1. MDPI, 2024
Lie Lu, Hong-Jiang Zhang and Hao Jiang, “Content analysis for audio classification and segmentation,” in IEEE Transactions on Speech and Audio Processing, vol. 10, no. 7, pp. 504- 516, Oct. 2002, doi: 10.1109/TSA.2002.804546
A. Gomez, M. S. Pattichis and S. Celedón-Pattichis, “Speaker Diarization and Identification From Single Channel Classroom Audio Recordings Using Virtual Microphones,” in IEEE Access, vol. 10, pp. 56256-56266, 2022, doi: 10.1109/ACCESS.2022.3177584
Ghosh, R. et al. (2024). Topic Segmentation of Semi-structured and Unstructured Conversational Datasets Using Language Models. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2023. Lecture Notes in Networks and Systems, vol 825. Springer, Cham. https://doi.org/10.1007/978-3-031- 47718-8_7
Deepika Yadav, Mayank Gupta, Malolan Chetlur, and Pushpendra Singh. 2018. Automatic Annotation of Voice Forum Content for Rural Users and Evaluation of Relevance. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies (COMPASS ‘18). Association for Computing Machinery, New York, NY, USA, Article 12, 1–11. https://doi.org/10.1145/3209811.3209875
F. Landini et al., “Analysis of the but Diarization System for Voxconverse Challenge,” ICASSP Waseem Syed: Revolutionizing Audio Content Navigation: AI-Enhanced Multimodality and Machine Learning for Speaker Diarization and Topic Segmentation 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 2021, pp. 5819-5823, doi: 10.1109/ICASSP39728.2021.9414315
Caratozzolo, P., Alvarez-Delgado, A., Hosseini, S. (2022). Natural Language Processing for Video Essays and Podcasts in Engineering. In: Hosseini, S., Peluffo, D.H., Nganji, J., Arrona-Palacios, A. (eds) Technology-Enabled Innovations in Education. Transactions on Computer Systems and Networks. Springer, Singapore. https://doi. org/10.1007/978-981-19-3383-7_1
Soenksen LR, Ma Y, Zeng C, Boussioux L, Villalobos Carballo K, Na L, Wiberg HM, Li ML, Fuentes I, Bertsimas D. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ Digit Med. 2022 Sep 20;5(1):149. doi: 10.1038/s41746-022-00689-
PMID: 36127417; PMCID: PMC9489871
Y. Zhao, X. Xia and R. Togneri, “Applications of Deep Learning to Audio Generation,” in IEEE Circuits and Systems Magazine, vol. 19, no. 4, pp. 19-38, Fourthquarter 2019, doi: 10.1109/ MCAS.2019.2945210
M. R. Mahmud, A. Cordova and J. Quarles, “Multimodal Feedback Methods for Advancing the Accessibility of Immersive Virtual Reality for People With Balance Impairments Due to Multiple Sclerosis,” in IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 11, pp. 7193-7202, Nov. 2024, doi: 10.1109/ TVCG.2024.3456189
Edison Research: Sound-Data-The-State-of-Audio-in-50-Charts. https:// www.edisonresearch.com/wp-content/ uploads/2024/06/Sound-Data-The-State-of-Audio-in-50-Charts.pdf
Published
How to Cite
Issue
Section
