Design efficient batch processing strategies

Move the pre-merge check to the batch API for the cost saving.

There is no latency guarantee and the window runs to many hours, so the gate that had to answer before a merge now blocks it indefinitely.

Batch suits work nobody is waiting on. A blocking check is not that.

Plan the schedule against the typical turnaround you observed.

Typical is not a commitment; the schedule has to hold when a batch takes the full window, which is the case that breaks the SLA.

Size submission frequency against the worst-case window, not the average.

Run the agentic extraction loop through batch to cut its cost.

Each batch request is self-contained, so your code can never return a client tool_result mid-request and the loop has nowhere to run. Server tools are the exception — web search and code execution run inside the request.

Batch is for self-contained requests. A client-tool loop stays synchronous; server tools batch fine.

Resubmit the whole batch when some documents fail.

It re-pays for everything that already succeeded and takes another full window to return the same results.

Identify the failures by their custom id and resubmit only those, modified for whatever caused the failure.

Submit the full volume and iterate on whatever comes back.

A prompt flaw is discovered at full scale after a full window, and every iteration costs both again.

Refine against a sample first; batch what you have already seen work.