Why it matters
Google DeepMind’s Decoupled DiLoCo divides model training into asynchronous compute islands across data centers. Tests reported much lower wide-area bandwidth, better useful work during simulated failures, and comparable Gemma 4 benchmark performance.
My takeaway: Distributed training resilience is also a control-plane problem. Operators should test isolation and reintegration failures, verify checkpoint and data consistency across sites, and monitor whether heterogeneous infrastructure changes reproducibility or audit evidence.