Batch orchestrating image generation across two machines

The pipeline that ships every hero image on this site started as a question. I had a Mac that did its best work writing prompts and a Windows box ten feet away with an idle GPU. The question was how to make them cooperate without one becoming the bottleneck for the other. The answer was a queue, an SSH alias, and a JSON contract that has not changed in 200 renders.

This is the tutorial. By the end you will have an image generation batch queue running across two machines: prompts authored on one, renders produced on the other, results pulled back, converted, reviewed, and threaded forward into the next round. I am assuming you already picked your model and your hardware. The rest of the wider hybrid pipeline I run on this site covers those choices; this piece is the wiring.

Prerequisites for an image generation batch queue

Five things need to be in place before any of this works.

A Mac (or any prompt-authoring workstation) with shell access, git, Node, and Python. Anything from the last five years is fine. The Mac is doing text and disk work, not compute.

A Windows box on the same LAN with a discrete GPU and ComfyUI installed. Pinokio gives you ComfyUI in one click and is what I use; manual installs work too. The model you picked (Flux Schnell, Z-Image Turbo, Qwen Image, whatever) needs to be downloaded and resident on this machine. If you are still picking the GPU, the hardware selection log that picked the GPU on the other end walks through the math.

OpenSSH server enabled on Windows. It ships with Windows 10 and 11; turn it on under Optional Features. Make sure the service starts automatically.

A shared git repo accessible from both machines, or direct SCP between them. You can run this with one or the other; I run with both because the use cases differ. More on that in step 3.

Two scripts. A Python runner on the Windows side that reads the prompts JSON and submits each entry to ComfyUI. A Node converter on the Mac side that turns the returned PNGs into WebP and writes a manifest the review HTML can read. Both are short. Neither is novel.

The whole pipeline is shell, JSON files, and SSH. Anything heavier is overhead that does not earn its place at this volume.

Step 1: Wire the SSH alias both directions

Open ~/.ssh/config on the Mac and add a host block:

Host windows-render
  HostName 192.168.1.42
  User render
  IdentityFile ~/.ssh/windows_render_ed25519
  ServerAliveInterval 60

The hostname is your Windows box's LAN IP. The user is whatever Windows account you signed in as. The identity file is a dedicated key pair you will generate next. The keepalive matters: long render batches involve long-running SSH sessions and you do not want them dropping silently mid-queue.

Generate the key pair with no passphrase (you want this unattended) and copy the public half to the Windows box:

ssh-keygen -t ed25519 -f ~/.ssh/windows_render_ed25519 -N ""
ssh-copy-id -i ~/.ssh/windows_render_ed25519.pub windows-render

ssh-copy-id may not be available on every Mac; if not, just cat the pubkey and append it to C:\Users\render\.ssh\authorized_keys on the Windows side. Permissions matter: the file needs to be readable only by the user on Windows or OpenSSH refuses to use it.

Test the alias with one round trip:

ssh windows-render hostname

You should see the Windows hostname print, no password prompt. If you get a password prompt, the key did not land or the permissions are wrong; fix that before going further. A queue that survives 200 renders cannot survive an interactive prompt halfway through.

A note on the dedicated key. Use a separate key pair for this pipeline, not your daily SSH key. The whole point of automating an image queue is that you walk away from it. A rotation of your main SSH credentials should not be able to silently break the renders, and the key should stay out of your work agent, your dotfiles repo, and your laptop's keychain all at once. Keep it scoped.

Step 2: Define the JSON prompt list contract

The contract is the load-bearing piece of the whole queue. Get this right and everything downstream behaves.

One file per round at .image-work/prompts-round-<N>.json. The file is an array of entries; each entry is exactly five fields:

[
  {
    "slug": "batch-orchestration-mac-windows-queue",
    "seed": 250101,
    "width": 1344,
    "height": 768,
    "prompt": "Tack-sharp 4K UHD cinematic landscape photograph of..."
  },
  {
    "slug": "batch-orchestration-mac-windows-queue-cliff-macro",
    "seed": 250102,
    "width": 1024,
    "height": 1024,
    "prompt": "Macro of..."
  }
]

Five rules govern the contract:

1. The slug becomes the output filename. If you want a companion image for the same article, append a hyphenated descriptor (<slug>-cliff-macro).
2. Seeds must be deterministic and unique within the file. I use the round number as a prefix: round 25 means seeds in the 250xxx range. Two entries with the same seed collide on filename and the second one clobbers the first.
3. Width and height must be multiples of 64. Diffusion models pad or reject other sizes; the only safe path is to enforce 64-alignment in the contract.
4. Prompt is a single string per the model's preferred shape. Z-Image Turbo wants short declarative sentences. Flux tolerates layered direction. The contract does not care; the runner just passes the string through.
5. Order matters for review. Group related slugs in the JSON the way you want them to land in the review HTML.

Why an array, not a stream or a database. You want git history on what you asked for. Six rounds in, when image 47 of round 4 surprised you in a good way, you are going to want to read the prompt that produced it. JSON in git is the cheapest way to keep that record. A queue daemon would be more elegant and give you nothing useful you do not already have.

The seed-numbering convention earns its keep on rerolls. A reroll of round 24 starts a new file (round 24-rerun, or round 25, your call) with new seeds in a new range. The original PNGs stay on disk under their original seeds. You can compare the same prompt at two seeds without overwriting either.

Step 3: Push the round (git or SCP)

You have two options for getting the prompts JSON onto the Windows box, and you should know when to use each.

For finished rounds, commit the JSON to the shared repo:

git add .image-work/prompts-round-25.json
git commit -m "round 25: batch orchestration article batch"
git push

The Windows box pulls and runs. Git history captures the round, the prompt text, and any per-slug notes you stuck in a sibling file. Six months from now you can answer "what did I ask for in round 25" without thinking about it.

For exploratory rounds, where you are tuning a prompt template and you want ten-second feedback, skip git and SCP straight:

scp .image-work/prompts-round-25-test.json windows-render:~/render-jobs/

The Windows runner watches ~/render-jobs/ (or you trigger it manually over SSH; both work). You burn through three or four prompt iterations in the time a git push, pull, run cycle would take for one. When the prompt template stabilizes, you commit the final version.

Pick one default and stick to it for production runs. I run git as the default and only use SCP for the first hour of any new prompt direction. The mistake to avoid is mixing them on the same round; if half your prompts came from git and half from SCP, you will spend tomorrow wondering which version of the JSON the Windows box actually saw.

Step 4: Run the queue on the Windows box

The runner is a Python script that reads the JSON and walks the queue:

import json, sys, pathlib, subprocess
from comfy_workflow import build_workflow, submit

prompts = json.loads(pathlib.Path(sys.argv[1]).read_text())
out_dir = pathlib.Path(sys.argv[2])
out_dir.mkdir(parents=True, exist_ok=True)

for entry in prompts:
    target = out_dir / f"{entry['slug']}.png"
    if target.exists():
        print(f"skip {entry['slug']}: file exists")
        continue
    workflow = build_workflow(
        prompt=entry["prompt"],
        seed=entry["seed"],
        width=entry["width"],
        height=entry["height"],
    )
    submit(workflow, target)
    print(f"ok {entry['slug']}")

Sequential is fine. A ten-image batch is roughly three and a half minutes warm on a mid-tier GPU, give or take the model. You do not need parallelism for this volume. Trying to run two ComfyUI workflows simultaneously is the same shape of mistake as parallel mflux on the Mac: you fight your own VRAM and lose more time than you save.

Cold start adds about 30 seconds. It amortizes across the batch, so larger batches are more efficient. The trade is your patience for the review pass at the other end. I find 10 to 15 images is the sweet spot; bigger batches start to feel like you are reviewing yesterday's work.

The skip-if-exists check matters. Reroll passes lean on it. You write a fresh JSON with only the slugs that came back BAD or MAYBE; the runner sees the existing GOOD images on disk and walks past them without re-rendering. The renders directory becomes a stable cache; only the things you asked for get redone.

Failure handling is deliberately simple. If a generation errors, log the slug and skip to the next one. Do not retry on the same machine without a clean restart. The failure mode I have seen is carryover state in ComfyUI: a previous workflow leaves a node in a bad shape, the next workflow inherits the mess, every subsequent render is wrong in a subtle way. Restart the ComfyUI process between batches if any single generation in the previous batch failed. It is faster than chasing why image 12 came out grey.

Step 5: Pull renders and convert to WebP

When the runner finishes, the Windows box has a fresh renders/round-25/ directory full of PNGs. Get them onto the Mac:

git pull   # if Windows committed and pushed
# or
scp -r windows-render:~/render-jobs/renders/round-25 .image-work/renders/

Then convert in one pass:

node scripts/convert-renders-to-webp.js round-25

The script does three things. It walks .image-work/renders/round-25/, converts each PNG to WebP at the article-target resolution and quality, writes the WebPs to public/lab-cache/round-25/<slug>.webp, and updates a manifest at public/lab-cache/round-25/manifest.json that the review HTML reads.

WebP rather than JPEG is the only decision in this step that matters. At the resolutions used for article heroes (1344x768 down to 1024x1024), WebP saves roughly 40% on disk over JPEG at indistinguishable quality. The published public/images/writing/ tree is large enough that 40% is real money in build artifact size and in CDN bandwidth.

The conversion script also deletes the source PNGs after successful conversion. The PNGs are huge and you do not need them. If a conversion errors out, the PNG stays put so you can retry. A clean run leaves you with WebPs only.

The manifest is a flat JSON file the review HTML reads on load. It includes the slug, the path to the WebP, the original prompt (joined back from the prompts JSON for the round), the seed, and the dimensions. Do not skip the manifest write step; the review page is dumb without it.

Step 6: Open the review HTML and paste a verdict back

The review HTML is a single static page at .image-work/review-template.html that opens locally in a browser. It reads the manifest, lays out a grid of images in slug order, and gives you three buttons under each image: GOOD, MAYBE, BAD. Tap once per image. At the bottom is a paste block that updates as you tap, in this format:

- batch-orchestration-mac-windows-queue [GOOD] -- holds at strip aspect, atmosphere lands
- batch-orchestration-mac-windows-queue-cliff-macro [MAYBE] -- texture too even, want more crack detail
- ...

Two things to optimize for in the review page. Buttons rather than typed input, because half the time I am reviewing on my phone in the kitchen and typing kills the loop. And persistence on tap, because the worst review session is the one where you tap GOOD on image 14 and then realize tap 7 did not register and now you have to start over.

When the review pass is done, copy the paste block, switch to the Mac shell, and pipe it into the parser:

pbpaste | node scripts/ingest-verdicts.js round-25

The parser reads the block, updates the per-slug feedback ledger that feeds cues back into the next round, and runs the verdicts through a regex cue extractor that pulls reusable prompt fragments out of free text. Cues like "more crack detail" or "tighter macro" land back in the slug's record. The next time you regenerate that slug, the new prompts JSON inherits the cues automatically.

This is what makes the queue closed-loop instead of fire-and-forget. Without the ledger and the verdict ingest, every round is a fresh start. With them, round 25 is implicitly building on the lessons from rounds 1 through 24, even if you have not looked at any of them in a month.

Common mistakes

Five things that will break the queue at exactly the wrong moment.

Same seed across two entries in the same JSON. The runner writes the second one over the first because they share an output filename. Validate seeds with a one-line check before pushing the round: jq -r '.[].seed' prompts-round-25.json | sort | uniq -d should print nothing.

Non-64 dimensions. Some models silently pad to the nearest 64, producing an image that is not the size you asked for. Others reject. Either way the file is wrong, and you will not notice until you try to use it as a hero crop. Hard-code the allowed sizes in your prompt template.

Running the runner on the Windows box AND mflux on the Mac at the same time. Not because either crashes (both are fine in isolation; the local safety wrapper I still keep on the Mac prevents the mflux side from going sideways). Because you fight yourself for review attention. Two batches finishing at the same time means two review HTML pages and a doubled context-switch cost. Serial across both machines is the fastest because the human in the loop is the bottleneck, not the GPUs.

Letting the renders directory grow unbounded. Each round drops 10 to 30 PNGs in the staging area. After a month of work, you have a few thousand files you do not need. Tie the WebP conversion script to a cleanup step that removes converted PNGs older than two rounds. Or just run a find -mtime +14 -delete weekly. Either is fine.

Forgetting to rotate the dedicated SSH key. This one is on you; the queue does not know to remind you. Calendar reminder, twice a year, regenerate the key pair and push the new pubkey. Five minutes; saves you from finding out the hard way that the key has been on a disk image you forgot to encrypt.

What to try next

Wake-on-LAN before the SCP push if you turn the Windows box off between sessions. A ten-line script on the Mac that wakes the box, waits for the SSH port to open, then pushes the round. It cuts your idle power draw to near zero without making the queue any harder to use.

A webhook from the Windows runner to a phone notification when the round finishes. ntfy.sh, Pushover, anything cheap. The point is that you stop sitting at your desk waiting for renders; the phone tells you when to come back.

Move the review HTML to a shared local network address so any device on the LAN can review. A static-file server on the Mac, available at http://mac-prompt-desk.local:8080/review/round-25/, lets you review on the iPad in the living room while making dinner. Trivial to set up and it materially changes how often the queue actually closes its loop.

A staging directory between conversion and public/images/writing/. Right now the WebP path is public/lab-cache/round-N/<slug>.webp; only after Michael-review approval does a WebP get copied into the published public/images/writing/<slug>.webp. Treat the cache directory as quarantine. Do not promote rejects.

The full local-tooling pattern that wraps this queue (ledger, runners, review HTML, scripts) is part of the Operator's Stack that documents this and the rest of the local tooling, and the output is visible across every hero image across the case studies on this site. Same wiring; different image.

Frequently asked questions

Sources and specifics

• This pipeline shipped the bespoke hero image inventory for michaeldishmon.com across Q1 and Q2 2026, surviving roughly 200 batched renders without a queue-level failure.
• SSH alias windows-render resolves to the Windows render box's LAN IP via ~/.ssh/config. The dedicated key pair is windows_render_ed25519, kept out of the daily SSH keychain on purpose.
• JSON prompt list contract: each entry has exactly five fields (slug, seed, width, height, prompt). Dimensions are multiples of 64; seeds are deterministic and prefixed with the round number to prevent collisions across reroll passes.
• Conversion runs via node scripts/convert-renders-to-webp.js round-N, which walks .image-work/renders/round-N/, converts each PNG to WebP, writes both the converted file and a manifest under public/lab-cache/round-N/, and deletes the source PNG on successful conversion.
• The review HTML lives at .image-work/review-template.html. It is a single static page that reads the manifest, lays out a tap-driven verdict grid, and emits a paste block in the format - <slug> [GOOD|MAYBE|BAD] -- free text that the Mac-side ingest script parses into the feedback ledger.