Mailbox re-processing & format conversions

Dovecot’s dsync is an alternative

RE-PROCESSING

dealing with duplicates

rm -f $HOME/msgid.cache

This rule uses formail to remove emails that have a message-id that has already passed through the system. It keeps an 8K log. “W” waits until it gets the exit code from formail and filters if appropriate. the ‘h’ means pipe the headers only. it uses a user-defined lockfile for this task.

# GET RID OF DUPLICATES
:0 Wh: msgid.lock
| formail -D 8192 $HOME/.msgid.cache

mbox re-processing

we need to split the messages with formail -s

mv -i mbox mbox.tmp
formail -s procmail < mbox.tmp && rm -f mbox.tmp

maildir re-processing

no need for Formail here

find folder/tmp/ -type f #should be empty
procmail < folder/new/message && rm -f folder/new/message
procmail < folder/cur/message && rm -f folder/cur/message
rm -rf folder/

FORMAT CONVERSIONS

maildir2mbox conversion

we need to add ^From and Formail does it so well compared to procmail -f -

find folder/tmp/ -type f #should be empty
formail < folder/new/message | procmail && rm -f folder/new/message
formail < folder/cur/message | procmail && rm -f folder/cur/message
rm -rf folder/

mbox2split conversion

mbox to Maildir or MH

formail -s procmail < mbox && rm -f mbox

SAVE THE WORLD

splitting files,

grep '^Return-Path: ' brokenmbox | wc -l 

that number minus 2 (think about it),

csplit -n 4 -s brokenmbox /^Return-Path:/ {901} 

find files about 10MB,

find . -type f -size +$((1024 * 1024 * 10))c             

move all matching files to another folder,

find .lists.* -type f -print0 | xargs -0 -I file mv file tosort/
rm -rf .lists.*

RESOURCES

re-processing / duplicates

\~/.procmailrc.html https://bethesignal.org/dotfiles/procmailrc.html

postfix/procmail - preventing duplicate email in procmailrc https://serverfault.com/questions/650023/postfix-procmail-preventing-duplicate-email-in-procmailrc

Procmail copies message instead of moving it https://stackoverflow.com/questions/28120828/procmail-copies-message-instead-of-moving-it

conversions

Faking “From ” header with procmail/formail https://www.depesz.com/2010/09/22/faking-from-header-with-procmailformail/

Procmail: Filter and Split Incoming Email https://www.togaware.com/linux/survivor/Procmail_Filter.html

Converting between mailbox formats https://wiki.dovecot.org/Migration/MailFormat

How can I apply my procmail filters to my maildir inbox? https://unix.stackexchange.com/questions/41971/how-can-i-apply-my-procmail-filters-to-my-maildir-inbox

How to convert maildir to mbox format https://mutt-users.mutt.narkive.com/14tojofq/how-to-convert-maildir-to-mbox-format

Convert Maildir to mbox https://stackoverflow.com/questions/2501182/convert-maildir-to-mbox

save the world


Nethence | Pub | Lab | Pbraun | SNE Russia | xhtml