Substrate

Context

In Pakistan, a renovation or a supply deal is negotiated on WhatsApp. Prices are agreed in voice notes, drawings arrive as PDFs, payments are confirmed with screenshots, and revisions happen in a message sent at midnight. When something goes wrong, the question “what did we actually agree?” has an answer, but it is buried across hundreds of messages in two scripts and a dozen file types.

The problem

Two findings shaped everything.

• The decisive commitments often exist only in audio. Across several real threads, payment terms and approvals for design revisions were given in Urdu voice notes, never in text. A system that reads only the text misses the parts that matter most.
• The documents and the conversation disagree. A quotation says one price, a voice note says another; a drawing shows one quantity, the chat confirms a different one.

So the goal became: put every message, document and voice note on one timeline, extract the claims each one makes, and surface where they conflict.

What I did

A pipeline in Python, built and validated one primitive at a time.

• Ingestion. WhatsApp exports (the chat text plus audio, PDFs and images) are loaded per thread. Speaker attribution comes from the export’s metadata, never from the audio, because transcription cannot reliably tell who is speaking.
• Transcription, locally. Urdu voice notes are transcribed on my own machine with Whisper large-v3 through faster-whisper (CPU, int8, batched, with voice activity detection and the language set to Urdu), at about twice real time.
• Documents. PDFs are rasterised by default rather than trusting their text layer: two of the first five had effectively no usable text.
• Search. Everything lands in a SQLite FTS5 full-text index on one timeline.
• Claims and contradictions. LLM APIs extract claims from each message, transcript and document, and compare them across the thread.
• A correction corpus. Every false positive is logged with its cause, so the system’s known failure modes are written down rather than rediscovered.

Validation before building. Work moves in numbered validation steps. Each step tests one primitive (extraction, transcription, contradiction detection) against real data before anything is built on top of it.

Key decisions and tradeoffs

What the data taught me

• Pick models for the language you actually have. A smaller, faster Whisper variant produced garbage on Urdu because it was trained for English only. Large-v3 worked.
• Proper nouns don’t survive Urdu transcription. Names and product codes need correction after the fact.
• Voice activity detection can cut the one sentence you need. On price notes, a number can be clipped mid-utterance, so those notes need special handling.
• Drawings lie about their versions. One PDF had inconsistent revision stamps across its own pages, and reading a CAD drawing’s text layer once concatenated label layers into a false claim about a change in quantity. I caught it, corrected it and logged it.
• Two scripts break search. The same thread mixes Urdu script and Roman Urdu, and full-text search does not match across them without an explicit strategy.
• Silent failures are the dangerous ones. Every WhatsApp export names its chat file the same way, so loading several into one folder silently overwrote earlier threads. A folder glob silently skipped one voice note, which turned out to sit right next to a revised drawing. Both are now explicit checks.
• A frontier model’s success says nothing about a local model. Results from a large hosted model cannot validate a pipeline meant to run locally.

Results

On four real threads from a live renovation project, Substrate surfaced:

• price gaps between quotations and what was said in chat and voice notes;
• wrong quantities between drawings and confirmations;
• conflicting dates around a design revision;
• an ambiguous possible duplicate payment.

It also produced false positives, and those taught as much as the hits. The clearest: the same number appearing in two unrelated threads, for different people, was flagged as a conflict. It now sits in the correction corpus.

What’s next

• Re-run transcription from explicit selection lists, prioritising voice notes close to unanswered questions such as an agreed price.
• Make rasterisation the standard document path and give transcription its own budget as a first class step.
• A deliberate strategy for mixed-script search.

What I would do differently

I’d treat voice notes as the core of the problem from day one instead of a later add-on, because that is where the decisions were. And I’d build explicit checks for silent data loss before anything else: the worst bugs in this project didn’t crash, they quietly dropped evidence.