A preprint introduces Koi, a cluster planner for large language model inference across heterogeneous multi-cloud fleets. It considers accelerator choice, parallelization, serving engines, replica counts and changing workloads as interdependent operating decisions.

The authors describe a self-calibrating approach intended to update plans as resource contention changes. This is a research system; the paper’s results apply to its evaluated environments and are not a guarantee of cloud cost or latency savings.