Movemail: Message body empty in target folder when moving message using filters before junk classification

[PhLinuX-Lab]

Hello,
I installed BB yesterday to replace TDB due to the lack of movemail.
All was Ok but this morning I can notice that the body of all the local emails was empty.
They appear as arrived but there is no more message body.

I apply a filter with this box to class any email into a local directory. All was properly classified.

It seems that the message body disappear only when the message is filtered by my BB filters.

A snapshot of what I got into the local box with no filter word in the subject:

The message body is OK

A snapshot of what I got into the local directory with a filter word in the subject:

This is exactly the same message but a filter word [PhlSys] was added in the subject.
The message body is empty

A snapshot of the filter I used:

Thanks for your assistance.

[Betterbird]

Now we have a problem ☹️

We're basically developing BB on Windows. We've brought back the movemail feature for Linux by popular demand without actually testing it, just by putting back the code from TB 78 and tweaking the movemail account setup a bit. A volunteer tested it for us and confirmed that it was working, apparently they didn't try to run filters.

To test this, we would have to set up Unix mailspool. How is that done, can you assist?

Another question: The code we restored in the C++ backend is the code that was contained in TB 78. Was that working in TB 78, have you tested it in TB 78? Should be easy on Linux to just get a version from
http://ftp.mozilla.org/pub/thunderbird/releases/78.9.1/ and test it.

[Betterbird]

Can you run this with filter logging, see https://wiki.mozilla.org/MailNews:Logging, it's the Filters setting.

Also enable the filter log in the "Message Filter" panel.

[PhLinuX-Lab]

Ok, I run the same email (suject test 2) with log enabled

But same result : the message body is empty.

By the way, I had to create by hand the missing file filterlog.html into .../.thunderbird/8i4pv3di.default-default/Mail/localhost and apply access rights 600 pour my user.

[Betterbird]

So can you attach the log file? More important is the Mailnews:Logging referenced above.

[PhLinuX-Lab]

Ok,
I can see that the message Test 2 was finaly recorded in the log. It just take time to do it.
I did again with Test 3 to confirm and they are both recorded in the log.

html file cannot be attached :
`

<style type="text/css">body{font-family:Consolas,"Lucida Console",Monaco,"Courier New",Courier,monospace;font-size:small}</style>

[24/11/2022 13:52:59 UTC+1] Applied filter "PhlSys" to message from PhLinuX <phl@phlsys> - [PhlSys] Test 2 at 24/11/2022 13:52:47 UTC+1 moved message id = [email protected] to mailbox://nobody@Local%20Folders/PhlSys

[24/11/2022 14:38:48 UTC+1] Applied filter "PhlSys" to message from PhLinuX <phl@phlsys> - [PhlSys] Test 3 at 24/11/2022 14:38:12 UTC+1 moved message id = [email protected] to mailbox://nobody@Local%20Folders/PhlSys

`

[PhLinuX-Lab]

I forget to tell what I used:
$ export MOZ_LOG_FILE=/tmp/bb.log
$ export MOZ_LOG=5
$ /opt/betterbird/betterbird-bin

and the result is empty
-rw-rw---- 1 phl phl 0 24 nov. 15:12 bb.log.moz_log
-rw-rw---- 1 phl phl 0 24 nov. 15:12 bb.log.child-1.moz_log
-rw-rw---- 1 phl phl 0 24 nov. 15:12 bb.log.child-2.moz_log

[Betterbird]

The child logs are typically empty, but the main one isn't. You need to set Filters setting as I mentioned above:
$ export MOZ_LOG=Filters:5,timestamp

[PhLinuX-Lab]

Ok
bb.log.moz_log.gz

[Betterbird]

OK, so this is the relevant bit:

2022-11-24 14:51:07.723979 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Detected new local messages on account 'Local'
2022-11-24 14:51:07.723996 UTC - [Parent 15874: Main Thread]: I/Filters Reading filter list for account 'Local'
2022-11-24 14:51:07.724001 UTC - [Parent 15874: Main Thread]: D/Filters Reading filter list from file '/home/phl/.thunderbird/8i4pv3di.default-default/Mail/localhost/msgFilterRules.dat'
2022-11-24 14:51:07.724014 UTC - [Parent 15874: Main Thread]: I/Filters Creating a new filter list with id=List15
2022-11-24 14:51:07.724497 UTC - [Parent 15874: Main Thread]: I/Filters Read 1 filters
2022-11-24 14:51:07.724500 UTC - [Parent 15874: Main Thread]: I/Filters Filter list stored as List15
2022-11-24 14:51:07.753361 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Running filters on 1 message (159902)
2022-11-24 14:51:07.753371 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Using filters from the original account
2022-11-24 14:51:07.753374 UTC - [Parent 15874: Main Thread]: D/Filters (Auto) nsMsgFilterList::ApplyFiltersToHdr
2022-11-24 14:51:07.753387 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Filter run initiated, trigger=InboxRule (1)
2022-11-24 14:51:07.753391 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Running 1 filters from List15 on message with key 8 in folder 'Inbox'
2022-11-24 14:51:07.753395 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Running filter 0
2022-11-24 14:51:07.753441 UTC - [Parent 15874: Main Thread]: D/Filters (Auto) Filter name: PhlSys
2022-11-24 14:51:07.753575 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Filter matched message with key 8
2022-11-24 14:51:07.753579 UTC - [Parent 15874: Main Thread]: D/Filters (Auto) Matched message ID: [email protected]
2022-11-24 14:51:07.753583 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Applying 1 filter actions on message with key 8
2022-11-24 14:51:07.753586 UTC - [Parent 15874: Main Thread]: D/Filters (Local) Message ID: [email protected]
2022-11-24 14:51:07.753589 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Running filter action at index 0, action type = 1
2022-11-24 14:51:07.775527 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Action execution succeeded
2022-11-24 14:51:07.775537 UTC - [Parent 15874: Main Thread]: I/Filters (Local) Finished executing actions
2022-11-24 14:51:07.775543 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Applying filter actions succeeded
2022-11-24 14:51:07.775547 UTC - [Parent 15874: Main Thread]: I/Filters (Auto) Stopping further filter execution on this message

The account is called "Local" an the filter "PhlSys". According to the log all when well, but the body didn't get transferred. Did you have a chance to test with TB 78? See comment #104 (comment).

[PhLinuX-Lab]

Unfortunately, I'm afraid no. I only have one box and this is my main server: all my life is in !!
Otherwise I should have to build a VM , install a Deb 11 ... and so on.
Possible but it would takes me a lot of time.

[Betterbird]

No, you can just download http://ftp.mozilla.org/pub/thunderbird/releases/78.9.1/linux-x86_64/en-US/thunderbird-78.9.1.tar.bz2, unpack it, start it with thunderbird -p -allow-downgrade and select the existing profile with the Movemail account. It's however risky to run old versions on a new profile. To not run this risk, start with -p, create a new profile, set up the Movemail account and the filter and test it. Anyway, that will just give an indication of whether Movemail + filter worked when it last shipped in TB 78. Would be good to know though whether we're looking at an old problem or one introduced in BB when re-establishing Movemail.

[PhLinuX-Lab]

Ok, thanks for these precisions.
I'm going to do that and let you know the result ... probably tomorrow end of morning (Paris time)

[PhLinuX-Lab]

Finaly I was able to do it quickly.
It works fine with TDB78:

and got into the directory

[Betterbird]

OK, thanks for testing. Meanwhile set up movemail ourselves, just a matter of copying a mbox file to /var/spool/mail/user. One can do that repeatedly to create more (fake) incoming messages. Moving a message to another folder in the Movemail account works, and moving it to a subfolder of Local Folders works, too. What doesn't work is to move from the Movemail account to an IMAP folder. No filter action, the message remains in the Movemail inbox.

So your move target is PhlSys in a local folder according to #104 (comment):
14:38:12 UTC+1 moved message id = [email protected] to mailbox://nobody@Local%20Folders/PhlSys.

Is it possible that the target folder is broken? What happens if you repair it?

[Betterbird]

E-mail received into Movemail inbox with no filtering:

The same e-mail received after applying a filter that moves messages with "Folder" in the subject to a filter folder, you can see that the body isn't empty:

And here, move to another folder in Local Folders:

As you can see, the messages moved by this filter doesn't produce empty bodies in the target folder:

[Betterbird]

We did our testing with mbox format for the all accounts involved. Maybe you used maildir on the target folder? If you set up TB 78 on a new profile, you would have defaulted to mbox.

[PhLinuX-Lab]

#104 (comment)
I'm sorry. After run repair on PhlSys directory, nothing is better, the message body is removed.
Nevertheless an observation : If I do my filter disabled and manually run the filter with Local dir it works, the message body is OK. If I do the filter enabled, the message is automatically moved but without body.

[PhLinuX-Lab]

#104 (comment)
Yes, I did a new profile. Anyway, I did a check this morning with TB 78 and it works fine. The message is automatically moved to the directory by the filter and its body is not empty.

[PhLinuX-Lab]

#104 (comment)
In my case, it is a "classic" unix installation (mbox) concerning local emails. I mean, any local email is added into a single file /var/spool/mail/user for each local user.
(I use imap for other external emails)

[Betterbird]

Thanks for testing. It's not a surprise that running the filter manually will work; the integration of running the filter into the process that incorporates the new messages into the Inbox (from /var/spool/mail/user) causes the issue. At a guess, if you run the filter after Junk classification, it will work, too, Please try this as a workaround.

That said, as per comment #104 (comment), it works for us. We've asked another user to do some testing in the area.

[Betterbird]

Please answer #104 (comment). Are you using mbox or maildir for the Movemail account?

[PhLinuX-Lab]

I'm afraid I didn't understood something (Sorry for my poor English)
With TB 78 I have no problem at all. All goes well and yes I use a new profile with a local mbox dir (I also use mbox for the BB 102.5 patched). #104 (comment)

[Betterbird]

OK, mbox for both the TB 78 profile and the BB 102.5+movemail profile.

Now please try running the filters after Junk classification, I mean this:

[PhLinuX-Lab]

If it's a traceable lib/procedure/function/method, I can possibly compile it and trace it with GDB if that helps. But probably with your assistance (makefile, compile options...) .... it's been a while since I put my fingers in either C or gdb :-D

[Betterbird]

Well, the issue is that we need to reproduce the problem before we can fix it. Please try running the filters after Junk classification.

[PhLinuX-Lab]

I just check ... this is what I already do:

[PhLinuX-Lab]

OOPS ! AFTER ....

[PhLinuX-Lab]

It's ok with Filter AFTER Junk Classification (sorry !)
The message body is there !

[Betterbird]

Please use the workaround of filtering after junk classification for now.

[Betterbird]

This hasn't dropped off the radar. We've asked another user to test the filters and then we'll see how to reproduce the issue and fix it. It won't be quick though.

[Betterbird]

Let's see what happens when TB bring back movemail, if they ever will. We can't reproduce the issue and neither could a person who is using movemail in production and whom we asked to test this. Likely a problem in the original TB code we re-established. You might want to test this with the last TB version that still had movemail, ie. TB 78.

[ThiloteE]

The one bug that comes closest to this issue here (and is reliably reproducible) is [TB Bug 1781792] When a filter copies multiple messages into local subfolders, the copied messages are empty!!, but since you also encounter empty bodies, when only a single message is moved, there must be an additional variable at play.

Please try the workaround I found for the other bug.

I found a fix using the "Move later" filter action provided by the Filtaquilla Addon

Following actions in the given order work without dataloss:

1. "Move later" the matching message to Imap Sub-Folder
2. Copy the matching message to Local Folder

[Betterbird]

https://bugzilla.mozilla.org/show_bug.cgi?id=1781792 is for copying from an IMAP folder. This ticket here is about Movemail. Movemail is local mail, messages already on the client, so quite a different codepath from the IMAP issue. Besides, neither myself nor another Movemail user I asked to test this can reproduce the Movemail issue.

Please don't mention Betterbird on Bugzilla, they don't like that at all.