Missing or duplicate Message-ID: make reply correlation degrade safely instead of cancelling the wrong sequence

The identity or state under test

RFC 5322 says every message should have a Message-ID and that the identifier must be globally unique when generated. Replies should carry In-Reply-To and References as appropriate, which gives a normal thread a chain of identifiers.

RFC Message-ID is a strong cross-system correlation key when it is present and well formed, but production mail can still arrive without one or with broken identifiers from nonconforming software. A reply pipeline that treats Message-ID as a mandatory database primary key can therefore lose or duplicate events at the edge. Keep Message-ID as the preferred standards-based identifier, then define a fallback that is explicitly weaker: provider message ID where available, mailbox UID scoped by UIDVALIDITY, plus a bounded fingerprint of stable headers and body metadata. The fallback should be used for idempotency, not presented as proof that two independently generated messages are semantically the same.

Why the common workaround creates a second risk

“Must be unique” describes compliant generation, not a guarantee that every message your software receives will be compliant. Gateways, imports, bugs, or hand-crafted mail can still produce missing or duplicate values.

Generating a fake Message-ID during ingestion is especially dangerous because it erases the fact that the source message lacked one and can make later thread analysis look standards-compliant when it was not. Likewise, hashing the entire raw message can be brittle if gateways add headers or normalize line endings between copies. Preserve the original absence, store the raw provider identifiers, and record which correlation method was chosen. If References or In-Reply-To point to a known outbound Message-ID, those reply headers can still provide strong conversation linkage even when the inbound message itself is missing its own identifier.

Prove the failure mode before repairing it

Create fixtures with a normal ID chain, a missing Message-ID, a duplicated ID on unrelated messages, and a reply containing only partial References. Run each through the correlator and verify ambiguous cases do not auto-cancel unrelated leads.

Test with fixtures that include a normal reply, a duplicate delivery of that same reply, a message with no Message-ID, and two genuinely different messages whose visible subject/body happen to match. The normal duplicate should collapse to one processing event. The missing-ID fixture should use the fallback without inventing a source header, while the two similar messages must remain distinct. Re-run the fixture through a reconnect and a second folder view to make sure mailbox-local UIDs do not cause a duplicate. This is the kind of test that exposes a correlation scheme more reliably than reviewing database columns in isolation.

Correct the mechanism and retest the same question

Use a composite event identity such as account/mailbox UID for ingestion and a separate correlation layer for conversation mapping. When identifiers conflict, lower confidence and require additional evidence instead of forcing a match.

Keep a `correlation_method` field beside the chosen key so later investigations know whether identity came from RFC Message-ID, a provider extension, or a fallback fingerprint. Use idempotent downstream actions even with a strong identifier: canceling a follow-up twice should still be harmless. If a missing-ID rate suddenly rises, group it by sending software or source domain; that can reveal a broken upstream system rather than a bug in your parser. The goal is graceful handling of imperfect mail while preserving the distinction between authoritative identifiers and best-effort deduplication.

A concrete workflow for a solo operator

If two inbound messages both claim <abc@example>, the account UID proves they are distinct stored messages even though the header is duplicated. Address and References context can then decide whether either belongs to an outbound thread.

Fallback fingerprints should have a bounded purpose and lifetime. Combining normalized From/To, Date within a tolerance, subject, body length or a stable body hash can help collapse a duplicated delivery, but two automated notifications can legitimately share many of those fields. Record the components used and keep the collision consequence small by making downstream actions idempotent. If the provider offers a stronger opaque message identifier, prefer that inside the provider namespace. Also retain the raw RFC headers even when malformed; later parser improvements can re-evaluate them. A system that only stores its derived key loses the ability to learn from edge cases. The operational objective is not a mathematically perfect global identity but a layered scheme where strong identifiers are used when available and weaker fallbacks are visible, scoped, and safe to replay.

What to log so a second person can reproduce it

Store raw Message-ID, In-Reply-To, References, envelope/from addresses, mailbox UID, received time, matched outbound message, and a confidence/reason field for the correlation decision.

What this result does not prove

Subject text is a weak fallback. “Re: quick question” can collide across many prospects, so do not use a matching subject alone to cancel a sequence.

Field checklist

• Capture the raw evidence for missing duplicate Message-ID reply correlation email automation before editing DNS, queue state, or mailbox metadata.
• Confirm the cited mechanism using message-rfc, imap-rfc rather than a generic deliverability score.
• Keep a known-good control case so the failing layer can be compared without changing several variables at once.
• Apply the narrow repair described for Missing or duplicate Message-ID: make reply correlation degrade safely instead of cancelling the wrong sequence and preserve a rollback or retry path.
• Repeat the original failing test after the change; do not substitute a different checker as proof of recovery.
• Store timestamp, affected identity, raw result, action taken, and the post-change result in the operator log.