March 19, 2023 7:00 PM PDT
This document outlines the system design requirements and discussions for implementing a real-time commenting feature in a news feed. The focus is on scalability, message delivery methods, and architectural considerations necessary to support high user engagement and comment volume.
System Design Requirements
- Real-Time Commenting: The system must allow users to comment in real-time on posts in the news feed.
- Scalability: The design should accommodate a large number of users and comments, similar to platforms like TikTok and Facebook.
Key Discussions
Requirements
- The system should support real-time updates for comments.
- It should efficiently handle a high volume of comments, especially during peak usage times.
Scaling Requirements
- Considerations for scaling the system to handle millions of users and comments simultaneously.
Push vs Pull Models
- Push Model: Implementing a push model for real-time updates.
- Implementation: Utilize a Kafka message queue.
- Consideration: Kafka may face challenges with a large number of topics.
Industry References
- TikTok: 1 billion monthly active users, with posts receiving thousands of comments.
- Facebook: 3 billion monthly active users, with a unique approach to comment delivery:
- Each page load requires multiple writes for content display.
- The strategy is to "write locally, read globally" to optimize performance.
Database Considerations
- Comment Database: Necessary for retrieving old comments.
- Delivery Mechanism: Discussed methods for delivering messages in real-time.
Architectural Considerations
- Topic Creation: Create a topic for each post, though this may not scale well.
- Scrolling Window: Implement a scrolling window that handles comments for every 5-10 posts.
- Trade-offs:
- Using users as topics for scalability.
- Adding a frontend service to manage comment delivery.
Frontend and CDN Integration
- Fan Out to Receivers: Introduce a CDN service to distribute comments.
- Polling Support: Considerations for using CDN for polling.
- Comment Receiver Identification: The frontend server identifies comment receivers by sending post IDs.
- Caching: Utilize caching to store recent comments for each post.
- Trade-off Decision: Determine whether to send the comment directly or send a signal indicating a new comment is available.
Conclusion
The design of a real-time commenting system for a news feed involves careful consideration of scalability, message delivery methods, and architectural choices. The discussions highlight the importance of efficient data handling and user engagement strategies to create a seamless commenting experience.