Distributed large model inference is still in a dilemma where balancing cost and effect. The online scenarios demand intra-operator parallelism to achieve low job completion time (JCT) and intensive communications makes it costly. Conversely, the inter-operator parallelism can achieve high job processing capacity (JPC) with much fewer communications, but it fails to reduce the execution time.

In this paper, we present Liger, a distributed large model inference runtime system that is of capability to achieve low JCT at high JPC on the multi-GPU architecture. The key idea lies in the novel interleaved parallelism, which interleaves the computation and communication across requests. Liger enables this parallelism by carefully scheduling computation and communication kernels across requests onto multiple streams of multiple GPUs. It achieves precise control of kernel execution order efficiently by mixing use the CPU-GPU synchronization and the inter-stream synchronization. To prevent scheduling failures caused by resource contention, Liger introduces a contention factor strategy to anticipate the penalty of contention. It enables a higher degree of overlap by decomposing lengthy kernels into smaller, more manageable units at runtime.

Extensive evaluations show that Liger, in most cases, outperforms existing parallelism approaches across models and devices, presenting the best JCT and JPC results. In a 4-device case, Liger reduces the average JCT by 36.0% while maintaining the same JPC compared to the inter-operator approach. Meanwhile, it improves the JPC by 1.34× with improved average JCT compared to the intra-operator approach.