Distributed training system
A fault-tolerant distributed training system in Go, with a replicated parameter server.
- 22,807training steps per second through the replicated server, on a Rutgers iLab machine
A coordinator splits each training batch across workers over RPC and adds up their gradients in a fixed order, retrying a slow or crashed worker's share elsewhere.
The model lives in a key-value parameter server replicated across a primary and a backup, with every write acknowledged by the backup and each request applied at most once.
Coordinator
Splits batches, retries slow workers
Worker
Computes gradients
Worker
Computes gradients
Training loop
Reads and writes the model
Primary
Parameter server
Backup
A copy of every write
- Coordinator and Worker: slices
- Coordinator and Worker
- Training loop and Primary: get, put
- Primary and Backup: forward