Analyzing YouTube comments reveals urban topics and sentiment challenges
Analyzing Public Discourse on Urbanism: Topic Clustering, Sentiment Analysis and Retrieval-Augmented Generation using YouTube Comments
Computation and LanguageArtificial Intelligence
Summary
Online conversations about city life cover many issues but are messy and hard to analyze. The authors built a system that organizes and studies YouTube talk about 309 cities to find common topics and feelings. They found some tools work better than others, especially when dealing with short, informal comments. Their work also shows that popular ways to check if generated summaries match original comments don’t work well with these short texts. These insights help improve tools that analyze and summarize short user posts in many areas.
What this means in practice
- •For city planners: Analyze public opinions from social media to identify local urban concerns and sentiment at city scale.
- •For social media platform engineers: Improve content relevance and summarization techniques for short, informal user comments using geographic and topic clues.
Authors
Jakob Morales, Monica Hegde, Fayeq Jeelani Syed
Abstract
Online discourse about urban issues - walkability, cycling infrastructure, public transit, housing density, and street safety - is voluminous but unstructured, and existing city-evaluation tools capture none of it. We present a pipeline and conversational system that combines geographic entity resolution, topic modeling, sentiment analysis, and Retrieval-Augmented Generation (RAG) over 22,788 chunks of YouTube transcripts and comments spanning 309 North American cities. Beyond the system itself, our contribution is a set of measurements about what happens when standard NLP components meet short, informal, geographically ambiguous text. A Twitter-tuned RoBERTa classifier outperforms a VADER lexicon baseline by 12.6 macro-F1 points (0.589 vs. 0.464; McNemar p = 0.0001), but both models collapse on the neutral class, which dominates urbanist comment traffic; annotators disagree on the same class (Cohen's kappa = 0.53). Dense retrieval beats a TF-IDF baseline at every cutoff (P@5 0.790 vs. 0.560), and video-level relevance proxies understate chunk-level precision by a wide margin (0.660 vs. 0.94 under human rating). For groundedness evaluation, we find BERTScore unusable when a multi-sentence generated summary is compared against a single short comment - scores are nearly flat regardless of relevance - and show that ROUGE-1-based groundedness is a paraphrase-driven lower bound rather than a hallucination rate. These findings generalize beyond the urbanist domain to any RAG system built over short user-generated documents.