Harnessing social media for innovative data collection: A Reddit scraping and data-cleaning pipeline

This paper introduces a novel approach to data collection from social media by presenting a fully automated pipeline that scrapes and cleans data from Reddit. Designed to complement traditional survey methods and administrative data, the workflow efficiently extracts large volumes of user-generated content, identifies relevant submissions via targeted keywords, and systematically cleans the resulting text for subsequent analysis. Through customized modules that remove duplicates, standardize linguistic features, and handle domain-specific stopwords, the pipeline produces a high-quality dataset readily applicable to sentiment analysis, topic modeling, or other statistical techniques. By capturing timely, organic discussions on rapidly evolving topics, this method offers a valuable supplement to conventional data-collection strategies for researchers, policymakers, and statistical agencies seeking richer, more immediate insights from online communities.