From a scanned page and a recording to a searchable document.

2026-10-26 · story · documents · OCR · speech-to-text · search · v1.122.0 → v1.124.0 · ~11 min read

Office and PDF import, OCR, speech-to-text and filtered search in loomcycle v1.122 to v1.124 and on main

1. What lands on a reference system

We are building a document-based reference system on loomcycle. Its job sounds simple: hold the documents a group of people works from, and answer questions about them with a citation. Board files. Regulations, several of them in two or three editions. Decisions and the papers they rest on. Scanned pages someone photographed at a meeting. Recordings of the meetings themselves.

Very little of that arrives as text.

A DOCX is zipped XML. An XLSX is zipped XML with a shared-strings table and formulas. A PDF may have a text layer, or may be forty pages of scanned images with no text at all. A recording is an M4A from a phone or a WAV from a conference room. An agent that wants to read any of this has two bad options: ask the model to read raw bytes it cannot read, or shell out to a converter with whatever that converter does to the host.

The releases from v1.122 to v1.124, and the work now on main, close that gap end to end. A file arrives, the runtime reads it, a Document comes out, and search can be aimed at exactly the part of the corpus a question is about.

2. Reading a file the model never holds

Since v1.122, a file a tool returns lands in the run's inbox: an ephemeral volume of the run tree. The model sees one line instead of the bytes:

[file: volume:inbox:documents/report.pdf · application/pdf · 2.8 KiB · sha256:16d51716c3de… · from mcp:files/get_attachment]

The File tool works on that reference. File op=extract reads the file and writes extracted/<slug>.md and its pictures into the inbox, then returns counts, a characters-per-page yield and an excerpt of at most 30,000 characters. The excerpt is fenced as untrusted file content, the same way the runtime fences anything a stranger wrote.

The extraction is written in pure Go, inside the binary, with no converter process:

• DOCX: headings, lists, tables and pictures.
• XLSX: streamed, cached values and never formulas, the first max_rows and then [+N rows omitted].
• PPTX: slides in order, speaker notes and pictures.
• HTML: nothing fetched, scripts dropped. CSV, TSV, text, Markdown and images as well.
• PDF: the text layer becomes Markdown page by page, with ToUnicode CMaps, WinAnsi, MacRoman and Differences encodings and Cyrillic glyph names decoded. JPEG and JPEG 2000 pictures are copied byte for byte. Every encrypted PDF is refused.

A converter that reads a stranger's file is an attack surface, so every parser is bounded. One zip reader serves every zip format: at most 10,000 entries, duplicate names refused, total size checked on both declared and inflated bytes with a per-entry ratio guard, and entry names never used as paths. XML is parsed strictly, with no DTD entities, and with caps on elements, depth and text. A PDF is capped at 2,000 pages, with limits on characters per page, decoded content per page, form nesting and CMap size; a parser panic becomes an error rather than a crash. Document text is escaped so it cannot turn into Markdown structure, and in untrusted Markdown every image is defused and raw HTML escaped.

3. A document, not a blob

Extracted Markdown is still a file. File op=import turns it into a Document through the Document tool's own import_md, so the Document tool's grants, hooks and scope checks all hold. Each picture extract wrote becomes an image chunk under its own sub-heading, carrying the PDF page or PPTX slide it came from. Any other volume: reference in the text stays a link and is counted as skipped.

The imported document is tagged untrusted and source/<origin>, and on main its chunk bodies carry origin=import. That label can only lower trust. It keeps imported text out of fact recall: a clause someone imported is a passage to cite, not a fact the memory believes. On main the import also keeps a link back to where the file came from, with any credential-bearing query parameter in the download URL masked by the same rule the redactor applies to signed URLs at rest.

One small fix made a large difference in practice. A download tool often answers with bytes and no name, so a capture used to fall back to the tool's name, and a board paper became a document titled download-bytes. A capture now takes its name from the call's own fileName or path argument when the result gives none.

4. The page with no text layer

A scanned board paper or a photographed page is a PDF or an image with nothing in its text layer. Before v1.123, extract returned near-empty pages, and the agent could not read the bytes it must never hold.

File op=extract now takes ocr: never | auto | always, a language and dry_run. auto sends to OCR only the pages whose text yield falls below ocr_min_chars_per_page, so a document with three scanned appendices pays for three pages and not forty. The output keeps one ## Page N per page in order: text pages keep their text, OCR'd pages carry the transcript under an ocr engine=… provider=… model=… pages=… header.

OCR is a paid model call on material a stranger supplied, so it is off unless two people say yes:

• the operator configures a transcription.ocr block (engine, model tier or pinned alias, languages, page caps, an optional rasterise helper);
• the agent definition sets transcription.ocr to auto or always. That field is the ceiling and defaults to never. A call may ask for less, never for more.

Claude models, direct or on Bedrock, receive a page range as a PDF document block. Image-only models, such as local vision models on Ollama, receive one rasterised page at a time through the operator's helper. Pages are reserved on the run and the run tree before each call, and a cap stops the pass with what was already written kept. dry_run estimates pages, tokens and cost without calling a model at all.

Every call is a ledger row with purpose: ocr, its tokens and its cost. Every extract that ran or estimated OCR emits a transcription event, on SSE, gRPC, both adapters and the Web UI's run views, with pages, model, cost and status and never the text:

⟲ OCR scan.pdf · 12 pages · claude-haiku-5-5 · $0.04 · ok

5. What was said in the meeting?

The last kind of source is the one people forget until they need it: the recording. Before main, an M4A in the inbox answered unknown op "transcribe". Speech-to-text is on main now and goes into the next release.

File op=transcribe sends a recording to the operator's speech engine and writes the transcript to extracted/<name>.md, fenced as untrusted like any extract. The engine speaks the OpenAI audio endpoint shape, which means the same configuration works for OpenAI itself and for self-hosted whisper servers: speaches, whisper.cpp's server and LocalAI. For a reference system that holds recordings nobody wants to send out, the self-hosted path matters.

The model chooses only a few things: the file, one language from a closed list, a short vocabulary hint, whether to label speakers, whether to keep timestamps, and the output name. Everything else belongs to the operator:

• Cutting. A PCM WAV is cut by the runtime into pieces of chunk_minutes, each reaching back by an overlap, and the overlap's duplicates are dropped when the pieces are merged onto the recording's clock. Anything else is converted to WAV first by an optional ffmpeg helper that runs with no shell and a scrubbed environment. Nothing is ever truncated.
• Caps. The run's and the run tree's minute budgets are reserved for every piece before the first request. A WAV is measured from its samples, not from a header field an attacker can set. Any other container is counted at its declared length raised to its size at 8 kbit/s, so a header that understates a recording can only be corrected upward.
• Time. A whisper server on a CPU host answers only when a ten-minute piece is done, so stt.timeout_s (default 1800) bounds each request on its own clock rather than a chat-sized timeout.
• Speakers. On a diarising model, speaker labels become Speaker 1, Speaker 2 in order of first speech.

A recording does not always need an engine. A Teams meeting with transcription on already has a WebVTT file, and many other sources produce SRT. extract now reads both as a transcript, keeping each cue's start and turning a WebVTT voice tag into the speaker, in the same **Speaker** [hh:mm:ss] text shape the speech engine writes.

Each speech request is a ledger row with purpose: stt and its seconds, and GET /v1/_usage?group_by=purpose now reports chat, OCR and speech-to-text side by side, with audio seconds summed. For containers, a new loomcycle-media image carries ffmpeg and poppler beside the same binary; the default image stays distroless.

6. Why a filter after the search finds nothing

With extraction, OCR and transcription in place, one scope fills with several corpora at once: board files, regulations in several editions, decisions, and a quality gate's own reports. A question is usually about one of them. Before v1.123, Document op=search could not be aimed.

The obvious fix does not work. Each retrieval leg keeps 51 rows before fusion and rerank. For wording common to every corpus, those 51 rows can hold none of the corpus the question is about, and a filter applied afterwards then returns an empty list that looks like "no answer".

So the filters go inside. search and related take under_path(s), exclude_under_path(s), document_id(s), type(s) (widened to subtypes), status(es), tag, tag_prefix and as_of. They are resolved once per call and applied inside every leg before its limit, so the rerank pool holds only matching chunks. The response carries a filter echo only when a filter was named; a filter-free call is byte-identical to before. A match over half the scope is sent as its complement, and a list larger than the configured maximum is refused as filter_too_wide, never quietly post-filtered.

Access control holds inside the filter too. For a caller whose rules can deny part of the scope, the set is narrowed to what it may read before it is counted or sent, so no count moves when a denied document changes.

Two Postgres plans had to change to make this fast. A statement carrying an id list is now planned on each execution: a cached generic plan took 18.4 s where a custom one took 9 ms. An exclusion list became an anti-join, which brought a 20,000-id exclusion on 100,000 rows from 4.9 s to 0.26 s on the vector leg.

7. A clause and the period it was in force

Regulations change. A reference system that answers "what did the rule say in March" needs the March text, not today's.

Since v1.123, any ordinary chunk can carry its own valid_at and invalid_at, a half-open period, without becoming a fact. as_of on search, related and query_chunks requires a chunk's period to cover that moment, and for a fact its fact period as well. export_md, import_md, sync and diff_remote carry the dates, so an edition keeps its period wherever it travels.

graph_recall follows the same rule: it now returns up to 600 characters of an ordinary chunk's body by default, honours each chunk's own dates, and by default leaves out a chunk whose period has ended or not yet begun. Two editions of one clause can sit in the same document, and a question about a date finds the one that was in force.

8. Notes in one scope, references in another

An agent usually keeps its own notes in one scope and the reference corpus in another. It used to make two Memory calls and merge two ranked lists itself, which is exactly the kind of judgement a model does badly.

v1.124 adds documents_scope to Memory op=search and op=recall. When it differs from scope, that scope's document prose joins the call as a second leg inside the one search: the query is embedded once, both legs meet in the same fusion, rerank and single top_k trim, and every hit carries its scope. The document filters aim at that leg.

Two fixes made this safe. In v1.123, tenant document rows moved to the keyspace Memory reads, so a tenant-scope Memory search finds tenant documents at all. In v1.124, every document hit Memory returns passes the same access check Document search applies, on both legs: a caller denied a path can no longer read it through Memory.

9. What is not there yet

• Speech-to-text is on main and not in a tagged release. The command, http and gemini-audio engines are refused as not available.
• Encrypted PDFs are refused, not decrypted. Pictures other than JPEG and JPEG 2000 are counted, not re-encoded.
• Transcription cost is in the ledger and the usage report, not yet in the Web UI's run totals.
• OCR quality is the vision model's quality. The runtime bounds what it costs and records what it did; it does not grade the transcript.