Large scale embeddings on a single machine
Marius is a system under active development for training embeddings for large-scale graphs on a single machine.
Training on large scale graphs requires a large amount of data movement to get embedding parameters from storage to the computational device.
Marius is designed to mitigate/reduce data movement overheads using:
- Pipelined training and IO
- Partition caching and buffer-aware data orderings
Details on how Marius works can be found in our OSDI ’21 Paper, where experiment scripts and configurations can be found in the osdi2021 branch.
Requirements
(Other versions may work, but are untested)
- Ubuntu 18.04 or MacOS 10.15
- CUDA 10.1 or 10.2 (If using GPU training)
- CuDNN 7 (If using GPU training)
- pytorch >= 1.7
- python >= 3.6
- pip >= 21
- GCC >= 9 (On Linux) or Clang 12.0 (On