More generally, reducing the communication problem to a minor obstacle allows new classes of co-designed architectures and training paradigms  maximizing available compute: we hope to see the training of modular constellations of small models loosely connected across heterogeneous devices, using compute arbitrage spread world-wide.