Mailbox re-processing & format conversions

RE-DELIVER

re-deliver locally stored mail

check the destination

    head -20 /var/mail/root | grep To

re-deliver previously stored messages while checking the mail logs

    ls -ldF /var/mail/
    ls -lF /var/mail/
    formail -s sendmail -t < /var/mail/root && rm -f /var/mail/root
    mailq

RE-PROCESS

input is mbox

we need to split a single mailbox into several messages

to reprocess/convert some mbox see https://pub.nethence.com/bin/reprocess-mbox.txt

–or– in case you want to point to a specific procmailrc file e.g.

formail -s procmail -m /etc/virtual/nethence.com/pbraun.rc < mbox.tmp && rm -f mbox.tmp

–or– if you wanna use the default virtual domain delivery but against a specific user

formail -s procmail -m ER_DOMAIN=nethence.com ER_USER=pbraun /etc/procmailrc < mbox.tmp && rm -f mbox.tmp
chown -R vmail:vmail /var/spool/virtual/nethence.com/pbraun/

–or– against some already sorted folders you don’t wanna parse

formail -s procmail -m ER_DOMAIN=nethence.com ER_USER=pbraun /etc/procmailrc.direct < mbox.tmp && rm -f mbox.tmp
chown -R vmail:vmail /var/spool/virtual/nethence.com/pbraun/

mbox to Maildir or MH

formail -s procmail < mbox && rm -f mbox

input is maildir

we need to track the message files

to reprocess/convert some Maildir see https://pub.nethence.com/bin/reprocess-maildir.txt and https://pub.nethence.com/bin/reprocess-maildir-remove.txt

–or– no need for Formail here

find folder/tmp/ -type f #should be empty
procmail < folder/new/message && rm -f folder/new/message
procmail < folder/cur/message && rm -f folder/cur/message
rm -rf folder/

maildir to mbox

we need to add ^From and Formail does it well compared to procmail -f -

formail < folder/cur/message | procmail && rm -f folder/cur/message
formail < folder/new/message | procmail && rm -f folder/new/message
formail < folder/tmp/message | procmail && rm -f folder/tmp/message
rm -rf folder/

deal with duplicates

rm -f $HOME/msgid.cache

This rule uses formail to remove emails that have a message-id that has already passed through the system. It keeps an 8K log. “W” waits until it gets the exit code from formail and filters if appropriate. the ‘h’ means pipe the headers only. it uses a user-defined lockfile for this task.

# GET RID OF DUPLICATES
:0 Wh: msgid.lock
| formail -D 8192 $HOME/.msgid.cache

SAVE THE WORLD

splitting files,

grep '^Return-Path: ' brokenmbox | wc -l

that number minus 2 (think about it),

csplit -n 4 -s brokenmbox /^Return-Path:/ {901}

find files about 10MB,

find . -type f -size +$((1024 * 1024 * 10))c

move all matching files to another folder,

find .lists.* -type f -print0 | xargs -0 -I file mv file tosort/
rm -rf .lists.*

ALTERNATIVES

Dovecot’s dsync https://wiki2.dovecot.org/Migration/MailFormat

RESOURCES