Recommendation is an important instance of deep learning. These ever-enlarging models consist of a sparse part with TBs of memory footprint and a dense part that demands PFLOPs of computing capability to train. Therefore, achiev- ing both memory efficiency and computation scalability is critical to distributed training. Existing systems can relieve memory pressure by partitioning embedding tables. Yet they do not perform well when scaling up due to the high sparse communication cost between different parallel strategies in different training stages. While such cost can be reduced by designing fine-grained strategies on the sparse part that utilize skewed access pattern of data, the performance tax of maintaining consistency must be paid.

We design P2Res, a two-fold system that replicates hot embedding vectors on all GPUs and stores the rest ones across hosts. Our parallel strategy guarantees consistency of all embedding vectors handled by both parts, keeping the system transparent to training algorithms. A performance model is created in selecting the replicated items for optimal communication latency. We reduce the overhead of accessing embedding vectors residing in different processes using a pipeline over decentralized indexing tables and a contention- avoiding schedule for data exchange. In our evaluation on 32 GPUs over real-world datasets, 2.16 − 16.8× end-to-end speedup is achieved over HugeCTR, TorchRec and TFDE.