YTChatX

System Architecture

Project Purpose

To help YouTube streamers and viewers analyze chat trends in real-time and understand communication patterns. This platform efficiently processes and visualizes massive chat data from large-scale live streams, enabling streamers to better understand their audience interactions and engagement patterns.

Tech Stack

Spring Boot·Java·Redis·Playwright·PostgreSQL·WebSocket·Rust·Chromiumoxide·React

Roles

• System Architecture Design
• Spring Boot API Development
• Playwright Crawling System
• Rust Scraper Development
• Redis Real-time Processing
• React Dashboard Development
• WebSocket Real-time Communication
• Performance Optimization

Key Achievements

• Large-scale Real-time Data Processing

  Stably collected up to 1,677 messages per second during a 30K-viewer livestream using Playwright automation. Built a reliable data collection pipeline that bypasses YouTube's dynamic loading and rate limiting.

• Performance Optimization with Rust

  Introduced lightweight Rust + Chromiumoxide scraper to overcome Java crawler limitations. Achieved 65% memory usage reduction and 3x scalability improvement with 50+ concurrent crawlers.

• Reliable Batch Processing System

  Implemented stable batch processing for 2,500 messages every 10 seconds using Redis Pipeline and RSemaphore. Prevented system overload and minimized data loss through resource control.

• Real-time Analysis & Visualization

  Developed real-time donation detection algorithm and keyword-based chat trend visualization. Enhanced user experience with real-time updates via WebSocket communication.

Screenshots

Click on images to view them in full size.

Reflection

What I Learned

I gained hands-on experience with the complexity of large-scale real-time data processing. Solving YouTube's dynamic UI and rate limiting challenges gave me deep insights into web crawling limitations and various solution approaches. Directly comparing Java and Rust performance helped me understand the characteristics of each language and their appropriate use cases.

Areas for Improvement

Having to rely on crawling for some features due to YouTube API constraints was challenging. I also regret not thoroughly reviewing and testing various scraper implementation approaches early on, which left room for performance improvements later in the project.

Future Plans

Instead of running Rust scrapers as individual executable binaries, I plan to transition to deploying a Rust Scraper Server in the user environment. This server will manage scrapers in separate threads and communicate with the Spring Boot server via gRPC. This approach will improve manageability, server-side resource efficiency, scalability, and maintainability. Additionally, I want to provide more value to streamers through NLP-based chat sentiment analysis, spam filtering, and automatic highlight extraction features.

Links