How it works
Distils combines scientific PDF processing, multimodal knowledge extraction, and named entity selection to prepare records for RAG and knowledge graph applications. The pipeline makes three kinds of model request; only the first sees page images.
The processing sequence
| Stage | Input | Result |
|---|---|---|
| Prepare | Original PDF | Page images from PDFium; metadata and reference markers grouped by page from GROBID |
| Read in order | Current page image and compact context | Standalone facts, equations, visual findings, and glossary definitions |
| Find entities | Each page's facts, document metadata, and glossary | Named concept candidates and missing definitions |
| Select and assemble | Candidates and evidence snippets | Main and secondary concepts, attached to their source pages in document.json |
A run is one batch of papers and the folder holding its inputs, progress, and outputs. Prepare the page images with distils-render, then start processing with distils-run.
1. Prepare the PDF
PDFium renders every page as an image. GROBID reads the original PDF for bibliographic structure: title, authors, abstract, bibliographic citation markers such as “[3]”, and internal references to figures, tables, and equations. Distils uses the markers' PDF coordinates to identify their source pages. The original response is saved as fulltext.tei.xml and reused on resume.
2. Read pages in order
The first model pass reads every page as an image. This multimodal extraction covers prose, equations, and scientific findings in figures and tables. On page 1, it also extracts visible metadata and the abstract.
For later pages, the request includes the current image, the document title and abstract, bounded earlier facts, and the shared glossary. The prompt asks for standalone claims while extracting only content visible on the current page.
The page-reading loop carries extracted knowledge forward without carrying previous pages in full. Accepted facts are appended and new glossary definitions are merged into the working context.
After each usable page extraction, Distils attaches GROBID's citations and internal_refs to that page's records. This preserves a shared page context for exploring references alongside facts, equations, and visuals. It does not attribute references to individual records; see citations and internal references.
For later pages, the LLM title and abstract are used when available, with GROBID values as fallback. Earlier facts and glossary entries are bounded by character budgets. See Configuration for those limits.
3. Extract named entities
After all pages have been read, a text-only pass identifies stable named methods, systems, datasets, objects, and scientific concepts in each page's facts. Pages without facts need no entity request. The pass also receives the document title and abstract and the glossary.
These named concepts are called entities in the code. Python deduplicates exact candidate names within the document and collects their source pages and fact snippets.
4. Select and assemble
The final text-only pass reviews candidate batches with evidence snippets, the title and abstract, and the glossary. Each candidate is classified as:
| Classification | Meaning |
|---|---|
main | A central subject of the paper |
secondary | Useful background context |
drop | A candidate that should be left out |
A paper's new method can be a main subject, while a dataset used only for comparison can be background. Kept concepts are attached to the pages where the entity pass identified them.
The final document keeps both metadata sources, standalone records, glossary definitions, source pages, selected entities, citation and internal-reference markers, and completion status together. Raw model responses are retained separately.
Ordering and concurrency
The three model passes happen in order: page extraction finishes before entity discovery starts, followed by document-level selection. workers controls how many papers can be processed at once; pages within one paper stay ordered.
If the first page is identified as a non-article, the remaining passes are skipped. Failed steps can leave an incomplete result; see Failures & recovery.
The implementation reference maps each stage to its Python module and explains how model requests are saved and resumed.