Political Finance Analytics Platform

Donor insight at scale

U.S. campaign-finance data is public, itemized, and large: millions of individual contribution records across candidates, committees, and cycles. That makes it a good stress test for a data-engineering question I wanted to answer for myself, which is where the real cost lives in a distributed analytics pipeline and how much of it you can engineer away.

Pipeline

The platform ingests raw itemized contribution records and runs them through a PySpark ETL pipeline: parsing and schema normalization, deduplication and standardization of noisy donor and employer fields, partitioning by state and cycle, then aggregation into the roll-ups that answer the actual questions (giving by state, by committee, by contributor occupation, over time).

The engineering focus was throughput, not just correctness. I profiled the stages that dominate wall-clock time, then worked the usual distributed-systems levers against them: partition sizing to avoid skew and small-file overhead, caching the reused intermediate frames, and choosing join strategies (broadcast versus shuffle) by the shape of each join rather than by default. The point of the exercise was to characterize the system, which stages are compute-bound, which are shuffle-bound, and where added parallelism actually helps versus where it just moves the bottleneck.

Takeaway

The analytics surface statewide donor patterns cleanly once the pipeline is fast and well-characterized. The durable lesson was the one that keeps showing up in my systems work: performance comes from understanding where the time goes, not from reaching for a bigger cluster.