Implementation

Follow the Python pipeline from scientific PDF processing to standalone records and named entity extraction. Start with process_document() for the main path. A run is one batch of papers and the folder holding its inputs, progress, and outputs.

Module map

Pipeline step	Where to look
Prepare page images	render.py creates the images and render.json, the list of rendered documents.
Start a run and follow the three passes	orchestrator.py: run() allows only one process to write to the run folder and checks inputs, code, and settings before resuming; process_document() handles one paper.
Read metadata and page references	grobid.py calls GROBID, reads marker coordinates to group bibliographic citations and internal references by page, and preserves the original fulltext.tei.xml. orchestrator.py merges those markers into retained page extractions. Saved GROBID output is reused on resume.
Tell the AI what to return, then read its answer	prompts.py holds the instructions (prompts); schemas.py defines the records and checks response formatting (parsing). These checks do not verify scientific truth.
Save the paper and the run summary	extract.py assembles document.json; reporting.py builds report.json. storage.py replaces JSON and XML files atomically, so a write does not leave a half-written destination file.

How progress is saved and resumed

Distils divides a paper into named steps. Examples include extracting page 1, extracting page 2, identifying the entities on a page, and selecting the document's main entities. RequestRunner in openai_client.py runs each AI step and records enough information to continue after an interruption.

Distils saves two kinds of progress for each paper:

• requests/<step>-<attempt>.json records one call to the model. It contains the prompt, the complete provider response when one was received, timing information, and the outcome of that attempt. Multiple files can exist for the same step when a request was retried.
• checkpoint.json records the result that Distils has already accepted for each step. An accepted result has been parsed into Distils records and has passed the required format checks. The checkpoint also lists any step that was left empty after using all of its format retries.

The request file is the detailed history of a model call. The checkpoint is the shorter progress map Distils uses to decide which steps are already finished.

What happens during a successful step

For a new AI step, Distils follows this order:

1. Create a request file and mark the attempt as started.
2. Call the model service.
3. Write the complete provider response to the request file.
4. Parse that response and check its record format.
5. Mark the request as accepted and save the parsed result in checkpoint.json.

The response is therefore saved before the checkpoint is updated. If the process stops between those two writes, the response that has already been received, and potentially paid for, is still available locally.

What happens when the same run is resumed

Run the same command again after resolving the interruption. Distils processes the paper in its normal order and makes the following decision for every step:

1. The checkpoint contains an accepted result: load that result and continue to the next step. No model call is made.
2. The checkpoint marks the step as missing after format retries: leave it missing and continue.
3. A response was saved but the checkpoint was not updated: parse the saved response locally. If it passes the checks, add its result to the checkpoint without calling the model again.
4. No reusable result or response exists: make a new model call using the configured retry rules.

This means that resuming a run does not repeat completed work. It can also recover a response saved just before an interruption, even when that response had not yet reached the checkpoint.

Responses that fail the format checks

A response can be received successfully but still be unusable because it does not follow Distils' record format. With the default settings, Distils makes one additional model request that includes the format error. If that retry also fails, the step is recorded under failed_steps in the checkpoint, the document remains incomplete, and processing continues where possible.

Connection failures and HTTP errors use a separate retry policy. Permanent API errors and exhausted API retries stop the run so it can be resumed after the service problem is fixed. See Failures and recovery for the complete error policy.