User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

Call IMAP folder compacting routines via the File|Compact Folders menu or Compact toolbar button (the Compact Folder context menu item is missing for some reason)

Actual results:

The deleted messages are expunged as expected. However, the folder index apparently gets corrupted, causing Thunderbird to (1) redownload ALL the messages in a folder, wasting significant time and network traffic, and (2) forget about the old message files but keep them on-disk, causing disk space duplication each time the folder was supposed to be compacted.
In fact, this problem has been present at least for three years (this is what prevented me from switching to Thunderbird back then).

Expected results:

(1) Compacting folders should not corrupt them and should retain the already downloaded messages
(2) Compact Folder context menu item should be available (it is mentioned in lots of online advice)
(3) Separate Expunge functionality should be available via context menu and toolbar button. I know we have "Expunge on exit" setting but (a) having to exit and relaunch Thunderbird just to expunge a few messages is pretty awkward, and (2) this feature ignores some folders with deleted messages.

What AV software are you using?
Pop account? or imap?

(In reply to Wayne Mery (:wsmwk) from comment #1)

What AV software are you using?

Dr. Web. They promise to to do the fully transparent filtering.

Pop account? or imap?

IMAP account over a secure connection.
Apparently the Maildir storage format is used by Thunderbird (one file per message).

However, the problem (either redownload or losing track of dangling stored messages) has never happened in Windows Live Mail. It also does not happen in Thunderbird due to Expunge itself, only when the folder is compacted.

I think this bug is a case of compaction being invoked even on maildir-backed folders, which causes the mbox-specific compactor code to blat over the maildir storeToken (which is a filename) with an mbox one (numeric). This breaks the connection between the msgDBHdr and the message file on disk.
The compaction code should be controlled entirely by the mbox nsIMsgPluggableStore implementation, but for now it's coordinated by the folder code, because:

1. on IMAP folders, it also issues an EXPUNGE.
2. File menu->"Compact Folders" uses the rather awkward nsIMsgFolder.compactAll() method, which issues a compact for all the folders on the same nsIMsgIncomingServer. The mbox store doesn't have the perspective for this at the moment.

• Adds nsIMsgPluggableStore.supportsCompaction checks to folder compact() and compactAll() implementations.
• Adds a last-ditch sanity check to nsMsgFolderCompactor to ensure it's not trying to compact folders which don't support compaction.
• Remove unused nsIMsgPluggableStore.compactFolder() implementation for now to reduce confusion (The compaction should eventually be handled by the msgStore, but we're not there yet).

Depends on D150923

Gene - Does this patch help fix the problem for you?
I tagged you as reviewer because you're down in the folder-compaction weeds with me at the moment :-) But Just let me know if you'd prefer to pass the review onto someone else!

Gene - Does this patch help fix the problem for you?

Fixes the problem for me and the code changes look good except for one minor issue where the window pointer check seemed to be left out.
I went ahead and marked the patch accepted. But yesterday I saw something that I might not be approved to review patches. However, a while back I thought Magnus told me I could but it's been a while since I was asked to.

Depends on D150923

Ok, I guess I need to approve that patch at bug 1776727 too since it appears they are cumulative.

(In reply to gene smith from comment #7)

Gene - Does this patch help fix the problem for you?

Fixes the problem for me and the code changes look good except for one minor issue where the window pointer check seemed to be left out.
I went ahead and marked the patch accepted. But yesterday I saw something that I might not be approved to review patches. However, a while back I thought Magnus told me I could but it's been a while since I was asked to.

Well, I value your review in any case! You understand the context of it all. I can get someone else to sign off on it too if need be.

Depends on D150923

Ok, I guess I need to approve that patch at bug 1776727 too since it appears they are cumulative.

Actually, those two patches are pretty independent - I use moz-phab to submit patches and that defaults to assuming dependencies if there's more than one changeset. I just forgot to tell it to submit independently. I'll probably leave it as is - don't think it matters too much.

try run here, just for a sanity-check:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=57c0b9520898b9be51a3fc4378afa595a5c61cff

This hasn't made it daily yet. Maybe still waiting for sancus review?

comm/mailnews/imap/test/unit/test_offlineStoreLocking.js X1 is failing on win debug in the try run - sound like it could be related

I added Magnus to double check these ones, as I really don't know anything about our C++ code at all.

Also that test failure may need addressing before this lands.

(In reply to Magnus Melin [:mkmelin] from comment #11)

comm/mailnews/imap/test/unit/test_offlineStoreLocking.js X1 is failing on win debug in the try run - sound like it could be related

I think you're right. Manually running ~/mozilla$ ./mach xpcshell-test comm/mailnews/imap/test/unit/test_offlineStoreLocking.js with only the changes from bug 1776727 passes consistently. Including the addtional changes from this bug causes the test to just hang and never reports pass or fail.

[:sancus] from comment #12)

I added Magnus to double check these ones, as I really don't know anything about our C++ code at all.

No problem - just wanted an extra sign-off while Magnus was away.

Magnus Melin [:mkmelin] from comment #11)

comm/mailnews/imap/test/unit/test_offlineStoreLocking.js X1 is failing on win debug in the try run - sound like it could be related

Yep - that test seems OK on mbox and hangs on maildir. Looking into it now.

BTW - make sure you run IMAP tests with the --sequential flag - the test IMAP server screws up with parallel instances and causes the tests to fail intermittently. You don't usually notice it when running the full suite of tests, but with just one test it'll always clash.
Also, Geoff added tags, so you can just run the tests in either 'mbox' or 'maildir' config (default is to run both).

so:

./mach xpcshell-test --tag mbox --sequential comm/mailnews/imap/test/unit/test_offlineStoreLocking.js

works, but:

./mach xpcshell-test --tag maildir --sequential comm/mailnews/imap/test/unit/test_offlineStoreLocking.js

is our failing test.

I'm see the test hang with BOTH methods above for mbox and maildir tags with the patches in place. After about 5m it times out and terminates.
When I ran without the patches or with just the first patch, as I said in comment 13, I saw consistently no errors without tags and sequential option.

:
 0:01.41 PASS testOfflineBodyCopy - [testOfflineBodyCopy : 205] 0 == 0
 0:01.41 INFO (xpcshell/head.js) | test run_next_test 7 finished (2)
 5:00.32 TEST_END: Test TIMEOUT, expected PASS. Subtests passed 2/2. Unexpected 0 - Test timed out
 5:00.32 INFO xpcshell return code: None
 5:00.32 ERROR xpcshell.ini:comm/mailnews/imap/test/unit/test_offlineStoreLocking.js | Process still running after test!
 5:00.33 INFO INFO | Result summary:
 5:00.33 INFO INFO | Passed: 0
 5:00.33 INFO INFO | Failed: 1
 5:00.33 INFO INFO | Todo: 0
 5:00.33 INFO INFO | Retried: 0
 5:00.33 SUITE_END

xpcshell
~~~~~~~~
Ran 3 checks (2 subtests, 1 tests)
Expected results: 2
Unexpected results: 1
  test: 1 (1 timeout)

Unexpected Results
------------------
xpcshell.ini:comm/mailnews/imap/test/unit/test_offlineStoreLocking.js
  TIMEOUT xpcshell.ini:comm/mailnews/imap/test/unit/test_offlineStoreLocking.js - Test timed out
ERROR xpcshell.ini:comm/mailnews/imap/test/unit/test_offlineStoreLocking.js | Process still running after test!
 5:00.33 INFO Node moz-http2 server shutting down ...
 5:00.33 INFO http3Server server shutting down ...

As discussed at bug 1776727 comment 25, the cause of the problem is that "storetoken" for maildir (the filename) gets overwritten by an unneeded file offset in the compactor code. So my idea is to just change the patch so that "storetoken" is not "blatted" when the maildir folder is run through the compactor. So even though maildir theoretically doesn't need compacting, apparently there is something that the compactor code still does to it to make test_offlineStoreLocking.js happy and if compacting is skipped for maildir the test fails.

With this change test_offlineStoreLocking.js passes for maildir and mbox.

Edit: I'm only suggesting this as a possible fix if a better solution can't be found.

Here's a proposed change set that fixes several things:

• Passes the unit test test_offlineStoreLocking.js
• Compact occurs when triggered and no longer on the 2nd trigger (Re: bug 1776727 comment 24)
• Add a "compact" right-click selection for imap maildir folders
• Compact of maildir now works ("compact" meaning maildir file for expunged messages removed).
• Tested with mbox and maildir both with all 3 delete models.
• Tested some with condstore enabled to check for any flag state issues due to "partial" flag fetches.

Here's the change summary:

• Adds the "compact" right-click to imap maildir folders as also suggested in bug 1130277
• Stops "storetoken" (maildir filename) from being overwritten with garbage in compactor
• No "sanity check" in compactor since allowing maildir to be "compacted"
• Save new array of expunged keys (UIDs) in flag state, fUidsExpunged
• In nsImapMailFolder.cpp let Compact() and CompactAll() handle mbox and maildir.
• In nsImapMailFolder.cpp change position of DeleteStoreMessages() call depending on storage type.
• In nsImapMailFolder.cpp use new flagState->GetExpungedUids() to determine keys to delete.
• In nsImapMailFolder::DeleteStoreMessages() add call to compactor for mbox.
• (Temporary) "printf("gds: ...\n") scattered around for tracing and debugging.

Comment on attachment 9285781 [details] [diff] [review]
compact-right-click-plus-others.diff

Ben, please take a look.

Comment on attachment 9285781 [details] [diff] [review]
compact-right-click-plus-others.diff

Thanks Gene! I've had a good think about this now and I think we need to bite the bullet and disentangle the separate concepts of compacting and expunging - at least on the C++ side even if we decide a generic "compact folder" is good enough for both on the GUI side.
I really want to hide the (currently separate) folder compaction inside the mbox store functionality because:
a) it's mbox specific, after all!
b) it needs to coordinate closely with the mbox store to manage file access. Without that, it's so easy for compaction to screw up the mbox (see Bug 1777454).

Also compact and expunge, while separate operations, do need to loosely coordinate - in the sense that expunge should really complete first. As the confirmation comes back from the server, it marks messages in the mbox as "Expunged". Then a compact can run and do it's stuff.

Anyway. Doesn't help us for this bug... So here's what I propose:

1. I'll fix up my patch here to exclude maildir from compaction (and fix test_offlineStoreLocking.js!)
2. we move over to Bug 1130277 to properly separate out Expunge and Compact and to sort out the GUI.

Pushed by [email protected]:
https://hg.mozilla.org/comm-central/rev/cb0907e81744
Don't run folder compaction on folders which don't support it. r=mkmelin

not too scary to take on next beta?

I think we should take it.

Yes, I agree. It's a big-ish patch, but fixes a big-ish issue.

Comment on attachment 9285781 [details] [diff] [review]
compact-right-click-plus-others.diff

[Triage Comment]
Approved for beta

Thunderbird 104.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/7978c5335d00

Good for esr?

I'm not concerned that it land quickly - in fact I'm not sure we want it for 102.2.2 considering everything else going on - just don't want to forget about it.

(In reply to Wayne Mery (:wsmwk) from comment #26)

Good for esr?

I'm not concerned that it land quickly - in fact I'm not sure we want it for 102.2.2 considering everything else going on - just don't want to forget about it.

Hmm... I think it's an improvement... but then I would say that! :- )
It does fix the bug, so I guess it really comes down to how serious an issue it is on 102.2.2. I don't really have any intuitive feel for that.

Comment on attachment 9285781 [details] [diff] [review]
compact-right-click-plus-others.diff

[Triage Comment]
Approved for esr102

Thunderbird 102.2.2:
https://hg.mozilla.org/releases/comm-esr102/rev/df6bb8392530