K9 mail not showing all emails (google, imap)

[BenoitM-I2BC]

Hi,

I've been using K9-mail for 9years now and always been satisfied.
Recently, I noticed that some emails (from my Google account illustrated here, but also from my professional account) are not displayed, even if they appear when using the webmail.

This is most probably linked to #1422, but I use a version >5.7xx in which this bug should have been fixed.

Description of the bug:

• K9mail version: 5.802
• See corresponding screenshots below: Some emails are visible using the webmail (left, bottom). For example, the Easyjet email, but this is also the case for "Google Team" message and the "Esp@ace Client" message.
• But they remains not visible in K9-mail (left, top)
• The local search (from K9 mail) does not find he Easyjet mail. The server search from K9 does find it. Once it has been found by the server search, the email does appear in the K9 mailbox (right).

I indicate:

• mails are not group into conversations
• I haven't found (yet) a pattern in the email that are not displayed (no matters when the email was sent, whether I'm or not the only recipient, ...).
• The fact that it is shown at the right place when the server search has been done suggests that K9-mail ingnore it to start with.
• I haven't seen anything obvious in the header of the email that are ignored and not sync-ed. Yet, I've posted the header of the Easyjet email below.
• I tried to recreate my mailbox from scratch (recreate a second time my gmail account): the emails that were ignored are still not fetched.

I welcome any ideas. Thanks !

Screenshots

Environment (please complete the following information):

• K-9 Mail version: 5.802
• Android version: 9
• Device: Samsung Galaxy A8
• Account type: IMAP

** Header of the Easyjet email ** (my firstname.lastname replace by xxxx.xxxx)
Delivered-To: xxxx.xxxxx@gmail.com
Received: by 2002:adf:ce92:0:0:0:0:0 with SMTP id r18csp1442794wrn;
Fri, 3 Sep 2021 03:34:04 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJypy6wCwkyASpJPgbUFuLd/GD8DAXUl2rLIovDb95XRWfChGsfeLhx3C4hqo+fmRP4w9GqK
X-Received: by 2002:adf:c54f:: with SMTP id s15mr3269427wrf.222.1630665244695;
Fri, 03 Sep 2021 03:34:04 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1630665244; cv=none;
d=google.com; s=arc-20160816;
b=mlWn5zO9+bsT1c4wXyqb9eLr935saW40a+rVhPKRHpJClp6xQoKJpxBPTedYDks12b
/foPu9kPb9nuMzrpieoW5p1OzbGWy+qLpndElX7lUMmSYXCmAJJZPxaAINq/ZmaxuL7G
RUUZpMWX+3Sb5lwCygiMtvfJkQkOEQDGIhM+y5mC28OpjwMCUktNBgqy2CiBJXX4kbUF
nMTlKHh5RtXPf7UDC1u325GwdghsOtM0BQCFxl9SHxPxRX4pjOiq9jAVnZrZ9/GA7hw8
JLjAQ1XZVjhX+FKmtDXe/4cuOjDPFYl5iRL3VlL10HmYRxfQ1ZdiLU9/bZhw/XzxKFGr
lnjg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=date:message-id:list-unsubscribe-post:list-unsubscribe:list-id
:mime-version:from:subject:to:dkim-signature:dkim-signature;
bh=ebIWJacdzrNviXT7lzUwoWBsZvnjbXPspEqOPT/luAo=;
b=errRLUsa8lTX6Wu5v+VuZLAXnVMcljepFZNy/4fgpptjnKunirH7XpWAuqZZqExyB+
Yn6awMTvv6GnSLcK/RQmvYiwPKJaQdYMjQTrQqdk+3mNMaslfn5SC50c+HBn0eddm/IN
3OPywJQR8k1q+ZpWAfYr4LXEzcoCBw12yZiPepdJJ7C7FUGM5uLwZLBG70I57ADlOw5A
tOHMAyzM2WfBP/8dQS0nsRGl0tlSJjTBJOMRY5Cyk3vxXhUjSG8PlTnK/IH94AHaxI9M
QFxcdmoIo9Vx8UcE0aeLM3g7dChWzPZbb07mzcULxKP+Ce1CxrCIhITrujTTeBm30LPA
WMSA==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@travel.easyjet.com header.s=key2 header.b=cw+8TfDF;
dkim=pass header.i=@emarsys.net header.s=key2 header.b=ADrIRAUm;
spf=pass (google.com: domain of suite37@xpressus.emsmtp.us designates 45.86.116.32 as permitted sender) smtp.mailfrom=suite37@xpressus.emsmtp.us;
dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=travel.easyjet.com
Return-Path: suite37@xpressus.emsmtp.us
Received: from mta-2d567420.ip4.emsmtp.us (mta-2d567420.ip4.emsmtp.us. [45.86.116.32])
by mx.google.com with ESMTPS id f1si4666207wrr.138.2021.09.03.03.34.04
for xxxx.xxxxx@gmail.com
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Fri, 03 Sep 2021 03:34:04 -0700 (PDT)
Received-SPF: pass (google.com: domain of suite37@xpressus.emsmtp.us designates 45.86.116.32 as permitted sender) client-ip=45.86.116.32;
Authentication-Results: mx.google.com;
dkim=pass header.i=@travel.easyjet.com header.s=key2 header.b=cw+8TfDF;
dkim=pass header.i=@emarsys.net header.s=key2 header.b=ADrIRAUm;
spf=pass (google.com: domain of suite37@xpressus.emsmtp.us designates 45.86.116.32 as permitted sender) smtp.mailfrom=suite37@xpressus.emsmtp.us;
dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=travel.easyjet.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=key2; d=travel.easyjet.com;
h=To:Subject:From:MIME-Version:List-Id:List-Unsubscribe:List-Unsubscribe-Post:
Content-Type:Message-ID:Date; i=inspiration@travel.easyjet.com;
bh=ebIWJacdzrNviXT7lzUwoWBsZvnjbXPspEqOPT/luAo=;
b=cw+8TfDF6TCs7r3b2WLV5varnM1Qct1jXLes4dITSME4gX/WwQInNCLzruAECLwxtUnTKNxO9LBG
6xWxrRmM2eK+qvl5umaLdQW4pVmQkN5E8lVDa5KemgX12MSiWtFXaFhYa4M8JITVs2W25n0ynAdH
q9gteCp+mf1DyX5CyjU=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=key2; d=emarsys.net;
h=To:Subject:From:MIME-Version:List-Id:List-Unsubscribe:List-Unsubscribe-Post:
Content-Type:Message-ID:Date;
bh=ebIWJacdzrNviXT7lzUwoWBsZvnjbXPspEqOPT/luAo=;
b=ADrIRAUmMdFDHUv9z6rP0NHRlWL/P47HRly4AJ+/hgU/mdlPqzznlG5PHzIQW4Oh4eK25BQVjXek
xOftBYwK5u8DylZZlZEGXZLnsSUiEV5DVP+nBJHKrhaEEfGDAvMiNlMvZC7ZGHwOKAadfXPgqXG9
uZu54mm81Uj4fNzJAxQ=
To: xxxx.xxxxx@gmail.com
Subject: XXXX, your winter is warming up =?UTF-8?Q?=F0=9F=8C=A1=EF=B8=8F?=
=?UTF-8?Q?=E2=98=80=EF=B8=8F?=
X-Mailer: class SMTPMail
From: "easyJet" inspiration@travel.easyjet.com
MIME-Version: 1.0
List-Id: 798153599
X-EMarSys-Identify: 798153599_14354_1768140
X-EMarSys-Environment: suite37
X-Report-Abuse: Please report abuse here: abuse-report@emarsys.com
List-Unsubscribe: mailto:XXX@XXX, https:XXX
List-Unsubscribe-Post: List-Unsubscribe=One-Click
Content-Type: multipart/alternative;
boundary="--=---NextPart--=-BLjJqwsfKS"
Message-ID: 0.0.92.574.1D7A0AEF85077A8.0@mta-2d567420.ip4.emsmtp.us
Date: Fri, 3 Sep 2021 12:32:18 +0200

[BenoitM-I2BC]

Hi again. Am I the only one having this issue?
Thanks

[jeroenbell]

You are not the only one 🙂 I have the same problem.

I found out that (in my case) this is because K-9 does not correctly respect the "Sync messages from" setting. It seems like it is ordering messages by id rather than date when syncing. So when you have a low "Local folder size" set (such as the default 25), it will only sync the last 25 messages with the highest id (which are not per-se the most recent ones).

Can you try setting the "Local folder size" to "all messages" to see if that solves your problem?

[BenoitM-I2BC]

Hi Jeroen. Thanks for your reply. I'm happy to hear that I'm not the only one

I haven't tried to set the "Local folder size" to "all message", but I'm sure this will indeed solve the problem, given that all messages are going to be sync-ed. Because I use K9 mail for a 3 email adresses, and storage is limited on my phone, I'd rather not sync all my messages from my 3 email adresses (but I increased the "Local folder size to 100, I'll see if it helps)

If indeed K9 is not anymore ordering by date when syncing, this is worrying. Shouldn't this be the default behavior?
In 2020, I already had this issue, which then was smoothly solved by using a beta version of K9.

[parkerfath]

Seems related to #1642

[getzze]

I am also having this problem, increasing the Local folder size to 100 didn't solve it.
By making a server search, I managed to import some more messages but not even all of them.
Any idea what is the problem? What can we do to try to solve it?

[jjakob]

Version 6.707 still has this bug. Threads disabled. The problem folder is one that I sort messages into with a filter on the server. K-9 doesn't show any messages in the folder even though there are 800 messages in it visible with other clients. No amount of refreshing with pull-down shows them. Searching for them doesn't find them client-side, but does find them when I click on the server-side search button. But that does not make them show up in the folder as others have said, the folder is still empty.

[Shallrath]

I have this same problem using the mail provider web.de via IMAP on K9 version 8.1.
One more oddity: I have K9 on my phone [1] and on my tablet [2]. I received a mail that was shown (but not deleted from the server!) on the phone. Later, I checked the mails on my tablet and that one mail was missing. The tablet has the same version of K9 with the same configuration and it received other mails in the same inbox. I set the number of downloaded messages to "all" as this was mentioned as a workaround but it didn't help.

Later, I checked the mails on the PC using Thunderbird and all mails were there. The PC is configured to delete all checked mails from the server but my mobile devices do not do this.

So, some addional information:

• This behavior is not limited to gmail
• It can behave differently on different devices using the same K9 with the same configuration

[1] Samsung Galaxy S7, Android 8
[2] Huawei MediaPad M5, Android 9

[jjakob]

@Shallrath I switched to Thunderbird 8.1 from F-Droid (Android 13) becase OAuth2 wasn't working in K-9 (Google not allowing the app, it's a known issue). I've only been using it for a couple weeks but it seems to work fine now, those problem folders now work.

[Shallrath]

@jjakob This seems odd, as far as I know, Thunderbird and K9 are identical. However, you seem right, Gmail authentication is not working anymore on my phone. I didn't notice because I don't actively use this address.

Did you migrate (export / import) your settings from K9 to Thunderbird? Maybe not changing the app but starting over from a clean installation was what fixed it for you?

[jjakob]

@jjakob This seems odd, as far as I know, Thunderbird and K9 are identical. However, you seem right, Gmail authentication is not working anymore on my phone. I didn't notice because I don't actively use this address.

Yup, it's #8598 . Google will stop allowing "app password" logins soon so everyone will have to switch to Oauth2, even though existing app password logins still seem to work.

Did you migrate (export / import) your settings from K9 to Thunderbird? Maybe not changing the app but starting over from a clean installation was what fixed it for you?

Clean install on an entirely new device.

[dl8dtl]

Thanks for the idea, @jjakob

I've been facing the same problem of not seeing some mails, no clear pattern which mails are affected. When it happens, it's quite annoying – you now, the mail must be there, but it isn't displayed.

Likewise, I installed Thunderbird from F-Droid now, and it indeed shows up the mail in question.

Strange, indeed, but I'm happy to have a workaround that way (whithout needing to resort to Webmail).

(Mailserver is a private one, using Dovecot for IMAP. The inbox is actually quite full, but Thunderbird/K9-Mail only fetches the most recent messages anyway.)

[firewire10000]

I'm having the same problem with the BETA version v17.0b4 downloaded from the Play Store whereby I will search a keyword and only get a few results back, even if I click on the Search messages on server button. However, on the Gmail app I get all emails using the same keyword. I'm using OAuth to connect my Google account.