Add option to compress incoming email

jandafields shared this idea 1 year ago
Completed

In Dovecot, the "mdbox" mailbox format is supported starting in cPanel v58.

In addition, the mdbox mailbox format supports email compression (within dovecot official), although that is not a settable option through cPanel. Dovecot does all the work, all you have to do is turn on the option in the dovecot config.

Using compression, you can save significant server space because most email is just text which is easily compressed.

I have been using this email compression on my cpanel server with v58, and it is working perfect. I only had to modify the dovecot local template file to add the compression options... and it just simply works. The end users see absolutely no change because the emails are compressed/uncompressed by the server on the fly as needed.

Making this an official option through the dovecot settings in the whm gui would be a great idea... and I cannot think of any disadvantages to it, because dovecot handles all of it automatically.

So, I'm not asking you to make it possible, because it already works great. I'm asking to make it an official option in the whm gui so I don't have to use a dovecot local template.

Best Answer
photo

The bug preventing this from working has been resolved, and this is now available in v66 in the CURRENT tier! Get a quick overview of all the improvements on the version 66 Release Site. If you have questions, feel free to join me for the next few hours in this public hipchat room:

https://www.hipchat.com/gQ7xUUD7i

Comments (20)

photo
1

This is something we'd be very interested in seeing as well - Dovecot's storage compression doesn't even rely on cpanel's introduction of mdbox as it works just as well with maildir.

Enabling it for new message deliveries only involves a couple of minor adjustments to Dovecot's config file (as with the OP, on systems where we manually use compression already we just do so via a .local template file currently) with literally 4 lines added and 1 more changed we typically see a 30-40% reduction in storage usage for mail, which is HUGE.

Enabling compression in Dovecot will cause newly-stored mail to be compressed (which is a great start) but doesn't compress existing stored mail, ideally we'd also like to see the feature include a mailbox conversion tool to compress existing stored mail (the process for doing so is again trivial, we use our own script for the purpose and the process is fully documented on Dovecot's wiki)..

photo
1

What would the CPU hit be for having all the email be compressed though?

photo
1

Revisiting this after a few months... The manual method of enabling this feature is still working well with cPanel v64 (although I had to modify the new .local dovecot file after the upgrade to v64 because that file changed and reverted back to the default .main file).

The CPU hit would depend on the level of compression you specify (on a scale of 1 to 10). I use the default of 6 and have no issues here. Again, this would be a very nice feature for cPanel to decrease disk space usage, and it would be very simple to officially add because all of the functionality is already there, it's just a matter of adding the zlib plugin to a few places in the dovecot file.

photo
1

This is currently being investigated. The work has been assigned case COBRA-4455

photo
1

Hi Nick,

Great to see this is now underway - Could I ask whether the intention here is for cPanel to handle mail compression only when mdbox is in use, or to also support it when using maildir? (We'd much prefer the latter, and Dovecot itself appears to essentially not care which mailbox format is in use for the purposes of its compression).

photo
1

This is feature is now available in EDGE under WHM, Service Configuration, Mailserver Configuration

040b4c5a802aff1e371319aa3bf90bd7

photo
1

Is there an automated option or script to compress existing messages?

photo
1

Same question.. existing messages be compressed when this feature is enable?

photo
1

If there is not an automatic way to compress old/existing emails, I'm pretty sure that converting to/from the maildir/mdbox will compress the old ones. (Whichever one you use, just convert to the other one and then back).

photo
1

This feature will only work with new incoming emails. The work to compress existing messages is a bit too much to be included in the existing system. We'll discuss internally if there are other (similar to the one @jadafields mentioned).

photo
1

Not sure if intended but it seems like this work has already been done in a roundabout way Benny - We've come to testing this now and while simply enabling the option to compress messages doesn't compress existing messages (or new messages when using maildir, we've raised a bugreport) jandafields is right that the mailbox conversion process does handle the compression.

So all existing mail can be compressed currently by enabling the compress messages option then converting all mailboxes from your current chosen storage method to the other, then back again.

photo
1

Neat! We'll call that an unintended feature, then. :)

photo
1

Ok, I just tried this... and it isn't working correctly. It can READ the (old/existing) compressed messages properly, but it is not compressing new incoming messages. Specifically, I sent myself an email, then I went into /home/jandafi/mail/mydomain.com/myuser/storage and I was able to plain-text read the newest file. The older files were compressed and obviously could not be plain-text read. The new file showed the incoming email that I sent myself and it was plain-text, not compressed. I'm using mdbox and the cpanel v66 edge 65.9999.192

photo
1

Yup, it's currently broken. It's being tracked internally as case CPANEL-14310. Sorry for that!

photo
1

It should be a very easy fix. There are simply some "zlib" lines missing from dovecot.conf. When you add those lines manually to "/var/cpanel/templates/dovecot2.2/main.local" and then run "/usr/local/cpanel/scripts/builddovecotconf", it works properly.

photo
1

Have we established this feature will work with both mdbox and maildir ?

And will it affect the cpanelsolr indexing for search ?

photo
1

Compression will only work with mdbox, and will work with the solr indexing for searching.

photo
1

Hi Benny, are you positive compression will only work with mdbox / is this something that's yet to happen / is there any solid reason for it?

From our testing the edge build a week or so ago both mdbox and maildir were compressible (though with a bug that new mail wasn't actually being compressed with maildir but that running a mailbox conversion would then compress existing maildir content, and our report has been accepted as a bug).

photo
1

Good catch! It looks like we don't have docs for this setting yet, but it seems like you might be correct! I'm watching the bug case that was opened (CPANEL-13086), and hopefully we'll see a clarification soon.

photo
1

The bug preventing this from working has been resolved, and this is now available in v66 in the CURRENT tier! Get a quick overview of all the improvements on the version 66 Release Site. If you have questions, feel free to join me for the next few hours in this public hipchat room:

https://www.hipchat.com/gQ7xUUD7i

Comments have been locked on this page!