Summary
Deciding how to handle every email, like whether it needs a reply or can be ignored, is a hard task for personal email assistants. The authors built a system called Foyer to test and compare different ways to sort emails—including simple rules, traditional machine learning, typed decision models, and large language models. They found that simple keyword rules performed poorly and that a basic machine learning model often worked best with enough training data. They also discovered some information about the sender's relationship is hard to guess from just one message and is better tracked as part of the contacts' history. Finally, effectiveness isn’t just about accuracy—how the system acts on emails matters too, with trade-offs on missing important actions versus wrongly flagging noise.
What this means in practice
- •For email client developers: Improve email routing accuracy by choosing well-measured models and balancing detection thresholds to reduce missed urgent emails.
- •For crm software builders: Enhance contact relationship tracking by integrating contact history for better sender classification beyond single-message context.
Abstract
Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.