- My role
- Problem definition, matching strategy, accept-rule thresholds, failure-mode handling, iteration from v1 to v4.1
- Status
- In production, feeding PulseOps daily
- Stack
- n8n · JavaScript code nodes · OpenRouter LLM · PulseOps webhook API
- Design rule
- The LLM is a fallback, not the engine — most emails never reach it
The problem
Every day, 50–80 partner reports arrive by email against a delivery schedule. Tracking them used to mean a person reading subjects and updating a tracker — error-prone, late, and unverifiable. Worse, subject lines are messy: abbreviations, aliases, forwarded threads, recalled messages, and "report will be delayed" emails that must not count as deliveries.
How it works
-
Ingest & deduplicate IMAP inbox trigger; duplicate messages dropped before any processing.
-
Fetch the day's catalog once The report catalog and matching profiles are pulled from the PulseOps API and cached per IST day — one fetch, not one per email.
-
Classify the email Regex patterns split report deliveries from recalls ("please ignore") and delay notices ("will share shortly"). The delivery timestamp uses the IMAP arrival time — not the sender's claimed time — so stamps can't be gamed.
-
Match deterministically TF-IDF cosine similarity between the subject and per-report profiles. Accept only if the top score clears a floor and beats the runner-up by a margin — a high score that's barely ahead of second place is not a confident match.
-
Fall back to the LLM only when unsure Low-confidence emails go to an OpenRouter model with the top-15 cosine-ranked candidates as context. Confident matches skip the LLM entirely — cheaper, faster, and auditable.
-
Post to PulseOps Matched deliveries are formatted and POSTed to the PulseOps webhook API, updating the report tracker in near-real-time.
Decisions worth explaining
- Score and margin, not just score
- Early versions matched on keywords and accepted the best hit. The failure mode was two similar report names ("Order Load — North" vs "Order Load — South") both scoring high. Requiring a gap between first and second place is what actually prevents wrong stamps.
- IMAP INTERNALDATE for the delivery stamp
- The email's Date header says when the sender claims to have sent it. The mailbox arrival time says when the report actually arrived. For an SLA metric, only the second is honest.
- The LLM is quarantined to the ambiguous remainder
- Deterministic code is testable and free; LLM calls are neither. Routing only low-confidence cases to the model keeps costs near zero and means a bad model day can't corrupt the easy 90%.
- Four versions to get here
- v1–v2 used hand-maintained keyword lists in Google Sheets; every new report meant editing keywords. v4.1 builds TF-IDF profiles from the PulseOps catalog itself, so adding a report in PulseOps is all it takes.
Outcome
- Report deliveries stamp themselves into PulseOps; the tracker no longer depends on a person reading an inbox.
- Recalls and delay notices are classified instead of miscounted as deliveries.
- Most emails match without any AI call; the LLM handles only the genuinely ambiguous tail.