Data Science · Natural Language Processing · Public Policy Analytics · 2026

Circular Aspiration Architecture, a Social Media Pipeline for Monitoring Education Issues

A prototype data pipeline for collecting, verifying, and analyzing public aspirations on education access in Jakarta and Banten from social media. Twitter data goes through rule based verification, IndoBERT sentiment analysis, and LDA topic modeling, then gets reweighted to avoid domination by prolific accounts.

  • 3 classesSentiment analysisPositive, neutral, and negative
  • 2 regionsGeographic scopeJakarta and Banten
  • 1 platformPrototype sourceTwitter as the study boundary
Case study contents
Case study reading progress

Objective

Demonstrate how public social media conversations can be collected, verified, and processed transparently to monitor education access issues, without claiming to represent the entire population.

Results and limitations

Produced a pipeline that separates relevant data from noise, measures sentiment and the main topics of education discourse, and explicitly reports uncertainty and representation bias. Limitation, this is purely a prototype for essay and demonstration purposes, covers only one platform, and results should not be read as population opinion without larger scale validation and offline data.

Technical details

Open implementation details

Role and contribution

Designed and built the entire pipeline alone, from scraping and rule based verification to sentiment analysis, topic modeling, representation weighting, and the Streamlit dashboard.

Methodology

Collected public tweets using education related keywords via Apify. Verified the data through rule based classification to separate citizen accounts, communities, media, and institutions. Ran three class sentiment analysis with IndoBERT, including uncertainty auditing and manual validation. Performed LDA topic modeling with coherence evaluation, then reweighted the results for more balanced representation, and presented everything through an interactive dashboard.

Technologies

  • Python
  • Apify
  • IndoBERT
  • LDA
  • Streamlit
  • Pandas
  • Jupyter Notebook
← All projects