AI Workload Placement: Borrowed Reversibility, Not Owned

A carbon-aware scheduler does the textbook-correct thing. It is 2pm, grid carbon intensity in the local region is high, a cheaper greener region is available 600 kilometres away, and it is holding a delay-tolerant training job. So it migrates the job to the low-carbon region to ride the afternoon solar. Clean decision, exactly what the carbon-aware scheduling literature says to do, and the dashboards light up green.

Except the job had checkpointing misconfigured - it wrote checkpoints to node-local storage that did not follow the migration. The scheduler preempted the job, moved it, and resumed it from the last checkpoint that actually survived the move, which was eleven hours old. The migration the scheduler treated as a cheap, reversible time-shift cost eleven hours of compute on a thousand-GPU job. The carbon saved was real and rounding error. The work lost was not.

Nothing in the scheduler was wrong. The placement logic was correct, the carbon math was correct, the region was genuinely greener. What was wrong was an assumption buried so deep nobody states it: that moving the job was reversible. It usually is. It was not this time - the checkpoint was recent but written to node-local storage that could not follow the move - and nothing in the scheduler's loop required it to check. The information was one metadata query away. The naive scheduler simply never asked.

This is Part 4 of the series. Part 1 set out the Operational Authority Gradient (OAG): the four-band model - Observe, Advise, Act-within-bounds, Closed-loop - where authority is set by irreversibility and blast radius, not model quality. Part 2 showed cooling at Act-within-bounds, where the envelope is the real engineering. Part 3 showed agentic SRE, where you gate on reversibility, never confidence. Workload placement is the use case that looks like the easy one - the actions seem reversible, so a path to bounded authority seems obvious. This article is about why that reversibility is borrowed, not owned, and what happens when the loan is called.

The thesis: placement's reversibility is borrowed, not owned

Placement is widely treated as the safe, near-reversible use case: you can always move the job back, reschedule it, try a different node, so the blast radius feels small and the path from Advise to Act-within-bounds feels short. Part 1 said as much - placement is reversible on the timescale of a scheduling window, and that reversibility is what earns the careful promotion.

Here is the claim this article owns, which sharpens that into something less comfortable: the reversibility that earns placement its authority is not a property of placement. It is a property of the checkpoint and migration layer underneath it - a layer the scheduling agent does not own, often cannot see, and must not assume. An agent that places work it believes is reversible, on top of infrastructure that has not actually made that work reversible, is not operating at Act-within-bounds. It is operating at Closed-loop while wearing an Act-within-bounds badge, and the badge is forged.

This is the same failure the whole series keeps finding, relocated one more time. In cooling, the danger was trusting the agent's judgement about thermal headroom it could not locally see. In SRE, it was gating action on the agent's confidence instead of the action's reversibility. Here it is assuming reversibility that lives in a different system entirely. The pattern: authority granted on a property the agent does not actually control is authority ungoverned.

Why this matters: placement is where the boom's flexibility actually lives

The reason placement is worth getting right is that it is the lever for the single biggest constraint on the boom - power - and the numbers are large enough to matter at grid scale.

The mechanism is workload flexibility. Data center loads split into two kinds: inflexible interactive workloads that must run now where the user is, and delay-tolerant flexible workloads that can be moved in time or space. AI training is the canonical flexible load - long-running, batch, interruptible, with weak latency SLAs. The flexibility is substantial: at hyperscalers, a significant fraction of jobs run in free and best-effort-batch tiers with weak SLAs, and at one large fleet roughly 87 percent of offline data-processing workloads carry completion SLOs greater than four hours, most of them 24-hour. That is a large pool of work an agent can move around without anyone noticing, if it moves it correctly.

Moving it buys two things. Carbon: shifting flexible jobs to times and places of low grid carbon intensity yields documented reductions in the range of 8 to 31 percent relative to carbon-unaware scheduling, more when temporal and spatial shifting are combined. And grid stability: because AI training maintains a relatively flat load at around 80 percent utilization, it leaves roughly 20 percent headroom, and flexible data centers that shift load from peak to off-peak flatten net demand, support renewable integration, and improve baseload utilization. Placement is not a tidy-up optimisation. It is the primary way the boom's enormous, growing load can be made a help to the grid rather than purely a threat to it - which is the subject of Part 5.

So the value is real and it scales. Which is exactly why the authority question matters: the temptation to let the agent act autonomously is strongest precisely where the stakes are highest.

Two axes of movement, two different reversibility stories

Placement has two independent degrees of freedom, and they do not carry the same risk - a distinction the band assignment has to respect.

Temporal shifting delays a job to a later, cleaner, or cheaper window in the same location. It exploits the fact that carbon intensity at one place varies across the day - solar floods the grid at midday, fades at night. The reversibility story here is comparatively clean for a queued or not-yet-started job: delaying it is close to free, because nothing is running to interrupt. Delaying an already-running job is not free - pausing it to resume later needs a working checkpoint, the same borrowed reversibility spatial migration depends on. So the deeper axis is not really temporal-versus-spatial; it is whether the action has to capture and relocate state. Temporal delay of a queued job captures nothing, delaying a running job borrows checkpoint capability, and spatial migration borrows capability plus portability - each step borrows more. The main risk on the cheap end is SLA: delay too long and you breach the completion deadline.

Spatial shifting migrates a job to a different region with lower carbon or cheaper power. It exploits the fact that carbon intensity varies across places at the same instant. This is where reversibility gets expensive, because migration is not free: it requires moving the job's state across regions, and that state can be large. Spatial shifting yields maximum utility for data centers located 300 to 400 kilometres apart - far enough for grid diversity, which is also far enough that moving terabytes of checkpoint between them is a real cost. The opening failure was a spatial shift. A temporal delay of a queued job could not have lost eleven hours, because nothing was running to interrupt - though delaying an already-running job is not free either, since pausing it to resume later borrows the same checkpoint the migration did.

The two axes carry different reversibility stories. Temporal delay is cheap and bounded by the SLA; spatial migration is expensive and must have its reversibility verified before it earns bounded authority.

The band assignment follows from this asymmetry. Temporal delay of a genuinely delay-tolerant job, inside its SLA, is a strong candidate for Act-within-bounds: the action is cheap, the worst case is a bounded delay, and the SLA is a natural envelope. Spatial migration is Advise by default and reaches Act-within-bounds only when the reversibility it depends on is verified to exist - which brings us to the thing the agent does not own.

The wrong way: assume the job can move

The naive scheduler treats placement as pure optimisation over carbon and cost, with movement as a free primitive.

code

# The seductive, wrong shape: movement is assumed free and always reversible.async def schedule(job: Job, regions: list[Region]) -> Placement:    best = min(regions, key=lambda r: carbon_intensity(r) * job.energy                                       + egress_cost(r, job))    if best is not job.current_region:        await migrate(job, best)        # assumed cheap, assumed reversible    return Placement(job, best)

The bug is not in the optimisation. The carbon-and-cost objective is fine. The bug is migrate(job, best) being treated as a primitive with no preconditions - as if every job is equally movable and every move equally cheap. The production scheduling literature is blunt about why this is false: most deep-learning training jobs today do not handle checkpointing or preemption, so a scheduler cannot rely on it; typically a low percentage of jobs have checkpointing enabled, which is unreliable from the perspective of performance a scheduler can provide. The scheduler that assumes movability is reasoning about a reversibility that, for most jobs, does not exist unless something else has gone to the trouble of creating it.

The right way: verify reversibility before you spend it

The correct scheduler does not assume movability. It treats reversibility as a precondition it must confirm, in the control plane, before the action's authority band is decided - exactly the cooling envelope and the SRE reversibility gate, applied to placement.

code

# Reversibility is a verified precondition, not an assumption. The band is# decided by the migration's real cost and the job's real movability.async def schedule(job: Job, regions: list[Region]) -> Placement:    target = best_region(job, regions)            # carbon + cost objective    if target is job.current_region:        return Placement(job, target)             # temporal-only, no move    # The move is a SPATIAL shift. Its reversibility is a property of the    # checkpoint layer, which the scheduler must verify, not assume.    move = MigrationPlan(        job=job,        checkpointed=checkpoint_status(job),       # is state actually captured?        checkpoint_age=checkpoint_age(job),         # how much work is at risk?        portable=checkpoint_reachable_from(job, target),  # does it survive the move?        state_size=state_size(job),                 # how expensive is the move?        sla_slack=sla_slack(job),                   # envelope for any delay    )    authority = authority_for(        action="spatial_migration",        # A recent checkpoint on node-local storage is the trap: it exists, it is        # fresh, and it does NOT follow the job across regions. Reversibility needs        # portability, not just existence and age.        reversible=move.checkpointed and move.portable and move.checkpoint_age < THRESHOLD,        blast_radius=move.state_size,        sla_envelope=move.sla_slack,    )    if authority is Band.ACT_WITHIN_BOUNDS:        # Reversibility verified: fresh checkpoint, movable state, SLA slack.        return await migrate_within_envelope(job, target, move)    # Reversibility not established. The agent proposes; a human disposes,    # or the move waits until the checkpoint layer makes it safe.    return advise_human(job, target, move)

The shape is the series' shape. migrate is no longer a free primitive; it is a plan whose properties are inspected before any authority is granted. The reversibility test is explicit and lives outside the agent, and it asks the three things that actually matter: is the job checkpointed, is the checkpoint reachable from the target region - a fresh checkpoint on node-local storage that cannot follow the move is the exact trap that opened this article - and is it fresh enough that resuming loses an acceptable amount of work? Blast radius is the state size - a small job is cheap to move wrong, a thousand-GPU job is not. And the SLA slack is the envelope on temporal delay, the same way setpoint floors were the envelope on cooling. An agent that cannot establish reversibility does not get Act-within-bounds. It drops to Advise, or it waits.

This is not theoretical. It is what planet-scale schedulers actually do. Systems like Singularity make preemption and migration transparent and the default for every job, with low-overhead on-demand checkpointing, precisely so that the scheduler can rely on reversibility instead of assuming it - so that lower-SLA jobs can opportunistically use spare capacity and be quickly preempted without lost work when higher-SLA jobs arrive. The lesson is the inverse of the naive code: the right to move work autonomously is earned by first building the layer that makes moving it reversible. The authority follows the infrastructure, not the other way around.

Borrowed Reversibility

The pattern deserves a name, because it generalises past scheduling to any agent whose safety rests on a property maintained by a system it does not control.

Call it Borrowed Reversibility: an agent is granted authority to act on the assumption that its actions can be undone, but the undo-ability is produced and maintained by a different layer, on a different team's roadmap, with its own failure modes - and the agent has no standing to verify it at decision time unless someone wired that verification in. The authority is real. The reversibility backing it is borrowed. And like any loan, it is fine until it is called - until the one job whose checkpointing silently broke is the one the agent chooses to move.

Borrowed Reversibility is dangerous specifically because it is usually true. If migration failed often, teams would never trust the scheduler and the problem would be visible. It succeeds 999 times, the dashboards stay green, confidence compounds, autonomy expands - and the 1000th job, the one with the stale checkpoint, takes the eleven-hour loss that all the prior green made invisible. The base rate of success is what makes the tail risk lethal, because it erodes exactly the caution that would have caught it.

The defence is to refuse to borrow silently. Make the reversibility precondition explicit and checked at decision time, so that when the loan cannot be honoured the agent finds out before it acts, not after. An agent that verifies the checkpoint before migrating is not borrowing reversibility - it is confirming it owns enough of it to act. That confirmation is the difference between Act-within-bounds and a forged badge.

Borrowed Reversibility: the agent's safety rests on a checkpoint layer it does not own. The same dependency resolves to a safe move or lost work depending on a checkpoint state the agent must verify, not assume.

Where each placement action sits

The band placement for the actions a scheduling agent actually takes, with the reasoning that pins each - and the reasoning, as always, is reversibility and blast radius, never the quality of the carbon model.

Temporal delay of a queued or checkpoint-capable delay-tolerant job within its SLA is Act-within-bounds. The action is cheap and the worst case is a bounded delay caught by the SLA envelope. The one caveat: pausing an already-running job to delay it still needs a working checkpoint, so a non-checkpointing running job is not the free case it looks like. For queued work, this is the safest autonomous placement action and the right place to start.

Spatial migration of a verified-checkpointed job with fresh state and SLA slack is Act-within-bounds, behind the reversibility check. The check is the envelope. Remove it and this action silently becomes Closed-loop on irreversible work.

Spatial migration of a job whose checkpoint status is unknown, stale, or absent is Advise at most - and arguably should simply be refused until the checkpoint layer is fixed, because there is no envelope that makes losing eleven hours of a thousand-GPU job acceptable.

Preempting a lower-priority job to place a higher-priority one is Act-within-bounds only when the preemption checkpoints the evicted job first - which is exactly what production schedulers do: before preempting a low-priority workload for a high-priority one, save a checkpoint for later restoration. The checkpoint-before-evict is the envelope. Without it, preemption is destruction.

Anything touching an interactive, latency-sensitive, tight-SLA workload stays at Advise regardless of carbon upside. The flexibility that makes placement safe is a property of the workload, and a tight-SLA interactive job simply does not have it. Confidence in the carbon savings does not create flexibility that the workload lacks.

The adoption sequence

The order follows the reversibility, not the carbon upside - the largest savings are often in spatial shifting, which is exactly the riskiest action, so chasing savings first inverts the safe sequence.

Start with temporal shifting of clearly delay-tolerant batch work inside generous SLA margins, where the worst case is a bounded, recoverable delay. Measure how often the agent's placement decisions would have breached an SLA if left unsupervised - that false-deadline rate is the placement equivalent of the SRE false-positive rate, and it is what tells you whether the agent's grasp of workload flexibility is trustworthy. Only once temporal shifting is proven, and only for jobs with verified fresh checkpoints, enable autonomous spatial migration - with the reversibility check mandatory and the move size bounded. Keep migration of large-state jobs behind a human gate longer than the carbon math says you need to, because the carbon math does not price the eleven-hour tail.

And invest in the checkpoint layer before you invest in the scheduler's autonomy, not after. This is the part that feels backwards and is the whole point: the scheduler's right to act is manufactured by the checkpointing infrastructure beneath it. Transparent, low-overhead, default-on checkpointing is not a feature adjacent to the placement agent. It is the thing that converts the agent's actions from irreversible to reversible, which is to say it is the thing that earns the agent its authority band. Build the reversibility first. Then grant the authority it supports.

Borrowed Reversibility checklist

Before a scheduling agent moves work on its own, confirm:

Reversibility is verified at decision time, not assumed. The agent checks checkpoint status and freshness before it migrates - it never treats migrate as a free primitive.
Temporal delay and spatial migration are separated. Delay inside the SLA is Act-within-bounds; cross-region migration is Advise until reversibility is confirmed.
Blast radius is the state size. A thousand-GPU job with stale checkpoints is not the same action as a small, movable one.
Preemption checkpoints the evicted job first. Checkpoint-before-evict is the envelope; without it, preemption is destruction.
The checkpoint layer is funded before the scheduler's autonomy. The agent's right to move work is manufactured by the infrastructure that makes moving it reversible.
You track the false-deadline rate. How often unsupervised placement would have breached an SLA tells you whether the agent's grasp of flexibility is trustworthy.

Where this lands

Placement looks like the easy use case because its actions look reversible, and most of the time they are. That appearance is the trap. The reversibility is real but borrowed - produced by a checkpoint and migration layer the scheduling agent does not own - and an agent granted authority on borrowed reversibility is ungoverned the moment the loan is called.

The fix is not to distrust placement. It is to stop treating reversibility as a free assumption and start treating it as a precondition the control plane verifies before it decides the band. Verify the checkpoint, bound the move, respect the SLA envelope, and refuse the migration you cannot establish is safe. Build the layer that makes the work movable before you let the agent move it on its own. The scheduler earns its authority from the infrastructure underneath it - and an honest scheduler knows exactly how much of that infrastructure is actually there before it acts.

Part 5 takes the load off the rack and onto the grid: power and energy management, where the actions do not just move work between regions but reach across the facility boundary into a regulated electrical system - and where the authority ceiling is set not by reversibility but by regulation.

References

Carbon-aware and grid-aware scheduling

"Carbon Explorer: A Holistic Approach for Designing Carbon Aware Datacenters" - Meta, ASPLOS 2023. Source for carbon-aware scheduling of delay-tolerant tiers, the Borg free/best-effort tiers, and the Meta 87% / 4-hour-SLO figures.
Sukprasert et al., "On the Limitations of Carbon-Aware Temporal and Spatial Workload Shifting in the Cloud" - EuroSys 2024. Source for the 8-31% carbon reduction range and the temporal-versus-spatial workload-suitability distinction.
"CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency". Source for temporal and spatial shifting and real-time carbon-intensity services (electricityMap, WattTime).
"Spatio-temporal load shifting for truly clean computing" - ScienceDirect (2024). Source for the 300-400 km spatial-shifting optimum and 24/7 carbon-free-energy matching.

Data center flexibility and the grid

"Flexible Data Centers and the Grid" - NBER working paper. Source for ~80% utilization, the ~20% flexibility headroom, and peak-to-off-peak load flattening.

Production scheduling, checkpointing, and migration

"Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads" - Microsoft. Source for transparent default-on preemption/migration, low-overhead checkpointing, and SLA-tiered opportunistic capacity use.
"MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters" - Meta, OSDI 2024. Source for checkpoint-before-preempt and continuous checkpoint-frequency reduction.
"SkyNomad: On Using Multi-Region Spot Instances to Minimize AI Batch Job Cost" (2026). Source for deadline-aware multi-region scheduling and two-stage checkpoint migration across regions.
"Deep Learning Workload Scheduling in GPU Datacenters: A Survey" - ACM Computing Surveys (2024). Source for preemption overhead, placement sensitivity, and checkpoint/migration mechanisms.

Agentic AI

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications: