Output format

Distils converts scientific PDFs into structured JSON for downstream RAG, semantic search, and knowledge graph applications. Each retained page groups scientific records and named entities with GROBID's bibliographic citation markers and internal references. Inputs, model responses, and the original GROBID XML are also preserved.

Run directory

runs/quick/
  documents/<paper>/document.json
  artifacts/<paper>/
    fulltext.tei.xml
    checkpoint.json
    requests/<step>-<attempt>.json
  images/<paper>/<page>.jpg
  render.json
  run.json
  sessions.json
  report.json

Document records

document.json contains metadata, a shared glossary, and a pages array. The complete record definitions are in schemas.py.

Document field	Contents
document_id, source_pdf, page_count	Document identity, original PDF reference, and number of pages
metadata	Separate LLM and GROBID titles, authors, and abstracts; keywords, venue, date, and preprint status when available
glossary	Terms with a short display label and scientific meaning
pages	Page numbers, scientific records, selected entities, and reference markers
status, failed_steps	Execution state and any steps whose output remained unusable; failed_steps may be absent when empty

Each page groups the following arrays:

Page field	Record contents
facts	id and standalone scientific content
equations	id, optional printed label, scientific meaning, and latex
visuals	id, kind, subtype, optional printed label, and findings in meaning
main_entities, secondary_entities	Scientific entity name and kind
citations	Bibliographic citation markers detected by GROBID on this page, as visible text
internal_refs	Internal reference label and kind: figure, table, or formula, detected on this page

Here are selected fields from GQA page 2, DeepSeek T=0:

{
  "page": 2,
  "facts": [
    {
      "id": "p2f1",
      "content": "Grouped-query attention divides query heads into G groups, each of which shares a single key head and value head."
    }
  ],
  "main_entities": [
    {
      "name": "grouped-query attention",
      "kind": "method"
    }
  ],
  "citations": [
    "(Pope et al., 2022)"
  ],
  "internal_refs": [
    {
      "label": "2",
      "kind": "figure"
    }
  ]
}

The fact, citation marker, and figure reference above all belong to page 2. Their presence on the same page does not assert that the citation supports that fact or that the figure illustrates it.

main_entities name the paper's central subjects; secondary_entities provide background. They are attached to pages, not individual facts. Exact names are deduplicated within a document; cross-paper name resolution is not performed. Equations contain LaTeX and meaning; visuals contain a type, optional printed label, and findings.

Record IDs are local to each document. Visual findings are text descriptions, not figure crops or reconstructed tables.

Metadata retains LLM and GROBID titles, authors, and abstracts separately. Later passes use the LLM abstract when available and the GROBID abstract as a fallback.

For indexing these records or connecting papers through entities, see the RAG, semantic search, and knowledge graph use cases.

Citations and internal references

citations preserves bibliographic callouts such as “(Pope et al., 2022)” or “[3]”. internal_refs preserves mentions of figures, tables, and equations within the document, including the visible label and its type. The page number identifies where a marker occurs, not the page of the object it refers to. Footnote and section references are not included in these JSON fields.

Distils requests reference coordinates from GROBID and uses their page numbers to group markers. Markers without usable page coordinates are skipped. Repeated marker text is deduplicated within a page (with the type also considered for internal references), so these lists are not occurrence counts. They preserve GROBID's detected text, including partial labels or punctuation.

The JSON retains marker text and page membership. It does not include bounding boxes, reference target IDs, a structured bibliography, or links from citations to individual facts. Markers are merged into retained extraction pages; a page with no usable extraction result can be absent from document.json even when it has markers in the saved XML.

Saved GROBID XML

artifacts/<paper>/fulltext.tei.xml preserves the original response, including marker coordinates, reference target attributes, and bibliography entries when GROBID supplies them. In the archived GQA example, the page 2 citation “(Pope et al., 2022)” points to bibliography entry #b20, and the figure reference “2” points to #fig_1.

These XML links can support further reference resolution and navigation. Distils does not export them into the JSON or match cited works across documents. GROBID citation consolidation is disabled in the request. Building a citation graph therefore requires importing and resolving these targets; associating an individual extracted claim with a citation requires checking the source context as well.

See citation graphs and reference navigation for applications of these fields.

Completion status

Status	Meaning
complete	Every required step finished
incomplete	A step failed or processing stopped; available records are retained
skipped	The model identified a non-article on the first page

Request artifacts and reports

A request artifact preserves the prompt/settings, image path, original provider response, timestamps, duration, outcome, and parser warnings. Checkpoints contain accepted intermediate results and failed format steps. run.json records configuration and input/code hashes; sessions.json records invocations. report.json summarizes completion and measurements.

Prepare records for a retrieval index

Read pages[].facts[].content for scientific statements, pages[].equations[].meaning for equation interpretations, and pages[].visuals[].meaning for findings from figures and tables. Keep the equation's latex field when you need to display the mathematics.

An index entry you construct from the published GQA example could look like this. These fields are an application-side mapping, not a second Distils output format:

{
  "document_id": "2305.13245v3",
  "page": 2,
  "record_id": "p2f1",
  "record_type": "fact",
  "text": "Grouped-query attention divides query heads into G groups, each of which shares a single key head and value head."
}

Use the document identifier together with the record identifier as a key: a record ID such as p2f1 is local to its document. Store the source PDF reference from source_pdf as well, and map it to a URL or file your readers can open. Index content only after checking the document's completion state and reviewing extraction quality for your application.