September 11, 2022 7:00 PM PDT

This presentation covers the Google File System (GFS), a distributed file system designed to handle large amounts of data across multiple machines. It discusses the architecture, key components, and operational considerations of GFS, including its handling of metadata, chunk management, and fault tolerance.

System Design Presentation Summary

Key Components

Master Node:
- Acts as a single point of failure.
- Supports metadata management.
- Utilizes a write-ahead log for crash recovery, allowing restart within 1 minute.
Chunk Servers:
- Store the actual data.
- Sync with the master during boot.
Clients:
- Upload data to the closest replica.
- Interact with multiple replicas for data storage.

Data Management

Chunk Size:
- Set at 64MB to optimize communication and memory usage.
- Larger chunks reduce communication with the master but are not ideal for small data sizes.
Data Operations:
- Clients send data to the memory of three replicas before saving to disk.
- New chunks are created when appending data exceeds the current chunk size.
Lease Mechanism:
- A node can act as a master for a version for only 1 minute.
- Every request acquires a lease from the master.

Fault Tolerance

Shadow Master:
- Functions similarly to slaves in a master/slave architecture, allowing for read and write operations.
- Can recover from logs in case of master failure.
Handling Failures:
- Master failures require human intervention.
- Stale replicas may occur but are managed through versioning.

Trade-offs

Consistency Models:
- GFS uses "at least once" semantics which may result in duplicates but allows for faster writes.
- Not all master data is strongly consistent to enhance performance.

Audience Discussion