Email archiving tool
After seeing so many stories of people getting their email accounts hacked, locked, or unexpectedly suspended, I realized how fragile it is to rely on a single provider. And we rely too much on our web email to be always available. That gave me the idea to…
After seeing so many stories of people getting their email accounts hacked, locked, or unexpectedly suspended, I realized how fragile it is to rely on a single provider. And we rely too much on our web email to be always available. That gave me the idea to build an email archiving tool so i can always keep my own copy of my emails. So far is very simple and in early stages, but I plan to publish it as open source. The idea is, get all emails, store in maildir format, and store the metadata in a database. What do you guys think of this as a project?
已收录讨论
Expand the replies to this comment to learn how AI was used in this post/project.
Interesting, I didn’t know that tool. Yes, similar
Thanks! That’s exactly the idea
Fair, and those flags are the right way to get there with what already exists. mbsync's Expunge settings and imapsync's --noexpunge do cover the basic case, and you're right that deciding the sync policy isn't the tool's call to make. The distinction I'd hold onto is between not deleting by configuration and not being able to delete by design. A sync tool with expunge off is still a sync tool: the guarantee lasts exactly as long as every future invocation carries the right flags. One cron line edited in a hurry, one --delete2 pasted from an answer that solved a different problem, one config regenerated by a helpful script, and the archive is thinner with nothing recording that it happened. For a tool whose whole job is surviving mistakes, that's a fairly load-bearing assumption. What a purpose-built archiver can add on top, none of it exotic: An append-only operation log, so "never archived" and "archived and now missing" are distinguishable after the fact. With a mirror you generally can't tell those apart. Content addressed storage, so a message can't be quietly replaced by a different one carrying the same UID. The store writable only by the ingest process, read-only to everything else. For most people mbsync with the flags set correctly is genuinely the pragmatic answer, and I'd say so. But if someone's building the thing anyway, those three properties are what would make it worth more than a well-configured sync job.
It's sort of useful if it's POP because it doesn't delete them locally unless you do, but with IMAP/JMAP it just maintains a local cache of what's currently on the server.
You can! Just need to use a PST to MBox conversion tool, and then import the MBox.
Why?
Would that option (via bulk download from providers - like Google Takeout) also work there? I have advanced protection on my google so no app passwords (iirc)... Only did a brief check of this and OpenArchiver, but saw this said IMAP and app password...
Did you not google for existing tools before deciding to build your own?
https://github.com/LogicLabs-OU/OpenArchiver/tree/main - similar?
I love this tool, just a no brainer after first setup
Most sync tools (like imapsync, mbsync) let you specify if you'd like to mirror the live account exactly (including deletions) or not. I mean they're tools, it's not their job to decide how you'd like to sync.
The "your client already does this" objection is worth answering head on, because the difference is real and it's the thing that would make this worth building: a client mirrors, an archive is append-only. If the provider suspends the account, or something deletes a folder server side, the next sync propagates that deletion straight into your local copy. A store that never deletes is a genuinely different tool from a store that stays in sync, and that property is a design decision you have to make on day one, not bolt on later. A few things that will bite you, in rough order of how much pain they cause: Keep the raw RFC822 bytes exactly as received. Don't parse and re-serialize before writing to disk. The moment you rewrite or reorder headers you invalidate the DKIM signature, and that signature is the only thing that makes an archived message verifiable later instead of just a text file you're asserting is genuine. Parse into the database for indexing and search, leave the bytes on disk untouched. Don't use Message-ID as your unique key. It's optional in the spec and some senders reuse or omit it. Hash the raw message instead. Track UIDVALIDITY per folder, not just UIDs. When a server changes it, all UIDs are renumbered, and a tool that doesn't notice will either re-download the whole mailbox or silently skip messages. If you archive Gmail over IMAP, labels are exposed as folders and the same message appears in several of them. Without dedup by hash you'll store popular messages many times over. Maildir is a sound pick. Just know that one file per message gets unpleasant on some filesystems once you're into the millions, so plan the directory layout early.
Hmm. Looks very interesting! I have a Gmail account dating back to 2004, and this would certainly come in handy to archive it. This could fit into my homelab nicely.
https://github.com/s1t5/mail-archiver I use this, they just added support for personal Outlook accounts. I have it backing up 7 emails between google, microsoft and yahoo, works flawlessly. Only thing I wish it could do was ingest my old PSTs
How do you find the search?
I've extensively tested Bichon and Mail Archiver and settled on Mail Archiver. https://github.com/s1t5/mail-archiver
I'm maintaining a small mailcow server for 9-10 users, they rotate their passwords from time to time. How will I know the new password for various mailboxes in the first place for it to start backing up via IMAP?
Been researching this myself and I'm planning to give Bichon a try. Saw it in this reddit post which also mentioned some of the others in this thread and all have nearly same star counts.
It does not support hosting under a subpath, does not support redis/valkey acl, does not support sso, and it spawns a lot of containers and I'm not a big fan of docker compose.
FYI mail-archiver is the only solution that supports OIDC.
There's plenty of those around, here's another one https://thekoma.github.io/mailfallback/
Don't confuse syncing with backup. Sure, use a tool to periodically get a copy of all your fresh email. But you also need to do proper backups (3-2-1) with the local mail archive, and the tools for that are different. For example I use mbsync to copy all new messages over IMAP to the local server, but then use borg to take proper backups of the local messages.
I set up a Proton Mail account and all my Gmail is automatically forwarded to it - every month I use the Proton CLI tool to export all the mail and save it.
Your email client literally does this. It downloads emails from the server via POP, IMAP, or JMAP and stores them in plain text mbox files on your computer.