Skip to content

Add a generic IMAP source - #188

Closed
constkolesnyak wants to merge 1 commit into
ClickHouse:mainfrom
constkolesnyak:feat/imap-source
Closed

Add a generic IMAP source#188
constkolesnyak wants to merge 1 commit into
ClickHouse:mainfrom
constkolesnyak:feat/imap-source

Conversation

@constkolesnyak

@constkolesnyak constkolesnyak commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Adds an IMAP mailbox source for providers the existing Gmail source cannot reach — that one goes through the gog CLI / Gmail API, so a plain IMAP mailbox (GMX, Fastmail, a company server) has no path in today. One source per mailbox, each with an independent cursor, registered from sync.imap.

Cursor is <UIDVALIDITY>:<max_uid>. IMAP UIDs are only stable within a UIDVALIDITY, so a server-side reset is detected and re-baselined from a SINCE window instead of trusted. imaplib is blocking, so the whole conversation runs in a worker thread.

Optional image pass

Some mail carries the part worth acting on only as an inline image — a scan, a photo, a rendered document — so the body text is useless to the inbox consumer. sync.imap.match singles those messages out by sender substring or by an image's Content-ID/filename, and sync.imap.vision runs a multimodal pass over the image at ingest time so what reaches the inbox is plain text. match.only_matched drops everything else at the source, turning the mailbox into a single-purpose notifier rather than a second inbox.

Both blocks are inert by default: with no match rules configured nothing is singled out, no model is ever called, and this is a plain mailbox reader. Nothing about any particular sender or language lives in the code.

Wording is configuration

vision.prompt and vision.answer_key are deliberately a pair. The prompt tells the model which label to emit; the parser reads the line after exactly that label. Changing one without the other silently yields unknown_answer every time — a failure that looks like the vision model degrading rather than a config mistake. Two tests pin the coupling: one moving both (works), one moving only the prompt (degrades as expected).

Templates accept {label} {answer} {vision} {subject} {sender} {date} {body}. An unknown placeholder logs a warning and leaves the template visible rather than raising mid-fetch and losing the batch.

Safety

Disabled by default. Passwords are read from sync.imap.passwords keyed by username, so no credential belongs in the tracked config. An account with no password is skipped with a warning rather than failing the whole sync. vision.enabled without a prompt is refused at registration with a warning instead of asking the model nothing.

25 tests. Documented in docs/sources.md and docs/config.md.

Adds an IMAP mailbox source for providers the Gmail source cannot reach
(the existing one goes through the gog CLI / Gmail API, so a plain IMAP
mailbox — GMX, Fastmail, a company server — has no path in today). One
source per mailbox, each with an independent cursor
(<UIDVALIDITY>:<max_uid>), registered from sync.imap.

Some mail carries the part worth acting on only as an inline image — a
scan, a photo, a rendered document — so the body text is useless to the
inbox consumer. sync.imap.match singles those messages out by sender or by
an image's Content-ID/filename, and sync.imap.vision runs a multimodal pass
over the image at ingest time so what lands in the inbox is plain text.
match.only_matched drops everything else at the source, turning the mailbox
into a single-purpose notifier rather than a second inbox.

Both blocks are inert by default: with no match rules configured nothing is
singled out, no model is ever called, and this is a plain mailbox reader.

The vision prompt and the answer key are deliberately configured as a pair:
the prompt tells the model which label to emit and the parser reads the line
after exactly that label, so changing one without the other would silently
yield unknown_answer every time — a failure that looks like the model
degrading rather than a config mistake. Two tests pin the coupling, one
moving both (works) and one moving only the prompt (degrades as expected).

Disabled by default; passwords are read from sync.imap.passwords keyed by
username so no credential belongs in the tracked config. An account with no
password is skipped with a warning instead of failing the whole sync.

25 tests pass.
@constkolesnyak constkolesnyak changed the title Add a generic IMAP source with Deutsche Post envelope enrichment Add a generic IMAP source Jul 26, 2026
@constkolesnyak

Copy link
Copy Markdown
Contributor Author

Closing stale branch; rebased on current upstream/main — new PR incoming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant