cPanel & WHM Version 98 has been released, and brings a slew of great updates. Take a look at what is included, and then upgrade today!

Ability to compress existing email

cPanelMichael shared this idea 3 years ago
Open Discussion

As a web hosting provider and system administrator, I would like the Compress Messages option in WHM >> Mailserver Configuration expanded to include additional functionality that compresses all existing messages stored on the server in order to maximize disk space savings.


--------------------------------------------------------------------


The Compress Messages option in WHM >> Mailserver Configuration enables compression for recently created and delivered messages. I would like to see this option expanded to cover all existing messages stored on the server.

Forums thread: https://forums.cpanel.net/threads/mailbox-compression-and-maildir-mdbox-conversion.623391

Replies (7)

photo
1

As a workaround, you could convert from maildir to mdbox and back once compression is on.

photo
1

This is safe? I read some comments about loosing flags or something.


Thanks

photo
1

It sounds like it removes flags, but you would still have the messages, so I guess it would depend on your definition of "Safe" in this instance.

photo
1

You can do the conversion, but (see below) from my testing it messed up all the client-based flags in Mac Mail.

we experienced success using impacsync with no side-effects whatsoever (again see my comments below)

photo
photo
1

Yes, I read that some places also. But that is alot of work if you have hundreds of servers.

Would be much easier to run a script that does compress all old emails on the whole server:

/scripts/compressmail :-)

photo
3

Yes please add this feature.

Being able to compress all existing messages without having to convert the mailbox to and from would be so much better and dramatically reduce the possibility of errors and problems during the conversion.

photo
2

We have a few really huge mailboxes. like many GB is size. It would be fantastic to have this "compress old email" option, even if it were a script to run in the shell.

photo
2

This is a needed feature. Often we migrate cPanel accounts from other legacy servers or hosts who don't use compression and have huge 20-40gb mailboxes. Is this in the roadmap yet? Thanks!

photo
2

Nope, not yet. When it is, we’ll make sure to update folks here!

photo
2

Just to chime in -- please add this feature to cPanel, since the rest of the framework is already in place, it shouldn't be too great of an engineering challenge to implement. I manage several cPanel / WHM servers and this is a BADLY needed feature. One server (a VPS where I have root WHM access) has been running for 8 years and seen many cPanel upgrades, so now it has 250+ IMAP e-mail accounts that are ever growing larger. To compress them would represent HUGE disk savings. The other server is a reseller server with dozens of hosting accounts, and again over 5 years the mailboxes have ballooned in size. On that server, I only have limited WHM access, but will be migrating to a VPS this quarter. I really need to compress old mail messages in order to realize disk usage savings -- which directly translates to financial savings due to disk quota allotments -- $10 per 10GB/mo. I could be saving $30-50/mo if this feature was implement!

photo
1

I’m in the same situation with client’s who have extremely large email accounts. This would result in a massive storage space savings.

photo
1

@Benny, did this discussion ever progress further within cPanel? It could potentially make a huge disk space saving to many of our clients.

photo
1

We haven't made any changes with regard to this at this time. As was posted previously by Nick, as a workaround, you could convert from maildir to mdbox and back once compression is on.

photo
1

FWIW, we've had success approaching this a different way, using: http://imapsync.lamiral.info/#DOC_BASIC_UNIX

It's not without it's drawbacks though, namely, requiring a 2nd server and also requiring a notification to go out to the users of the mailbox you need to move informing them that due to system maintenance their mail account is being migrated to a more efficient format (which is true) and they will need to log into webmail to reset their password back to whatever it was after the migration.

You then perform a pkgacct and restorepkg of their account to a 2nd cpanel server (either temporarily setup for this or if you have a 2nd production cpanel server), reset their password via the webmail interface in source and destination to your temporary password, and run this script (with the appropriate flags) to migrate just the changes since the account backup and restore. You could then either leave them on your 2nd production server and change DNS records, or migrate them back from your temporary server after removing the account and recreating it to get the compression.

I don't believe this would work using the same domain (even if you used a different account) on the same server. A 2nd server is required and also outage to the client, which is not ideal. Depends upon your client I suppose, you may get some of them to accept this and get the savings.

I can also confirm the client-side flags are unharmed using this method because the scripts acts as an IMAP client.

photo
1

You don't have to have a second server if you have ssh root on that server.

I use IMAP sync all the time to get IMAP accounts into our servers and it's a bit slow, with message per message sync.

A proper conversion is still needed and I don't really understand why this features has so few votes.


Regarding performance, are the compressed mailboxes bringing to much load to the servers? Does anyone have feedback on this?


Another thing, is it possible to have just a few compressed mailboxes or we have to activate that at the server level and there is no other way to try compression?

I'd like to try with a couple of mailboxes before going balistic with a complete server.

photo
photo
1

+1 for this one. Due to the lack of this feature, disk usage is showing data for individual mail boxes, when old emails are not compressed. This is very misleading for end cPanel user.

photo
1

I thought I would leave another comment on this thread that I originally commented on ~5 months ago. I did some of my own testing with our top 6 accounts size-wise. I took the packaged up accounts (from pkgacct) restored them to a test server with email compression enabled, then ran the conversion from maildir -> mdbox -> maildir.

Prior to the conversion I verified each set of folders inside each mail box and the total message count for each.

I then validated the same post-conversion to mdbox and back.

Here's what I found:

0) We definitely saved tons of space doing this.

1) There seemed to be no data loss (only counted the folders and message counts for each so I cannot comment on the actual emails themselves as I'd still be checking them!)

2) I DID find however that when connected to a converted mailbox (I did my own to test it out too), my client-side flags were all borked up. I happen to use Mac Mail that gives me multi-colored flags (at least on the Desktop client) and those were all completely different. It appears the conversion process changes some "ID" in the email file that means the client has the wrong emails 'flagged'. From my research I think this was discussed already online - but I don't have the link for reference. The only other solution I saw was a client-side bit of code that would preserve (rewrite) the old ID so as to avoid that issue. That looked a little too intense at the time though so I did not pursue it.

I guess the net of my comment would be: Even if they "enabled" this feature, I would suspect they'd need to crack the nut of keeping the IDs the same or at least restoring them automatically after compression. So the "tools" may already be there, but there's some other magic to really make it work seamlessly to the end user.

At least these are my thoughts based on my experiences..

Cheers.

photo
2

Thank you Jay for this comment!

Your 2nd note is *extremely important* to me, and a total deal-breaker on any conversion...

My clients have 20-to-70 GB email accounts so the temptation to do something to limit disk space is immense. But 70GB worth of emails getting marked as unread, or dis-organized in any way, would definitely lose me my clients, permanently.

Also, for those hoping that compression is a panacea, please remember that compression usually does not apply to attachements (gzip won't save you any space on a jpeg, it might actually make it bigger). So, really, the aforementioned compression really only applies to a tiny portion of the usual space usage of emails..

photo
1

Another vote for this. It's not fun telling the owner of a business they're paying $30 in disk storage fees just because they have 50GB worth of company email that could be compressed if they're willing to lose all the flags.

photo
1

This would be a huge feature add and I'm surprised it hasn't gotten more attention here. Who doesn't want to save space?

photo
Leave a Comment
 
Attach a file