What version of K-9 are you using? 2.403

Is your email account a POP account or an IMAP account? IMAP (dovecot)

When configuring folders for PUSH mail (IMAP IDLE) in K-9 Mail, it keeps one IMAP connection open for every folder. So for someone who has 45 folders configured for PUSH, K-9 Mail constantly has 45 connections open to the server. Can this be improved? Thunderbird with IDLE support only keeps (per default) five connections open.

Furthermore, sometimes we see multiples of this amount of connections - so instead of 45, we see 90, or even 270 connections (with netstat -ntp on the server - the state is shown as "ESTABLISHED"). Have a look at the attached graph of IMAP connections over time. Why does this happen? This is only one user, having Thunderbird (with max. 5 connections) and K-9 Mail open.

IMAP IDLE is limited to watching only one mailbox/folder per connection. That is just the way it works. If Thunderbird is using only 5 connections for IMAP IDLE, then it is only active for 5 folders. I'm not sure how it decides which 5 folders to watch. If you want to watch 45 folders with IMAP IDLE, you need to have 45 open connections.

K-9 Mail has a configurable limit as to how many push (IMAP IDLE) folders/connections will be allowed, in each Account's configuration. The default is 10. Did you change that?

IMAP NOTIFY, which I do not believe to be commonly available yet, supports monitoring multiple folders per connection. I do not have any plans to implement it, but it could be good for the future. It would be a lot more resource efficient.

The disturbing report here is that sometimes you wind up with multiple connections for each watched folder. I cannot tell you for sure why that happens. K-9 Mail is supposed to close the connections before establishing new ones. I cannot think of a process in K-9 Mail that would cause the sharp jump from 45 to 270 connections that is shown in the graph.

Can you correlate a surge in connections with changing the Account Settings? When settings are changed, K-9 Mail disconnects and reconnects its IMAP IDLE connections. If that process is failing, leaving the old connections open, then you'd see the behavior you describe.

One possibility is that your wireless coverage was interrupted, causing K-9 Mail to disconnect the connections, but due to limitations in the network, your IMAP server was not notified of the connection disconnect. K-9 Mail would create new connections upon restoration of service, but it would take up to the "read timeout" on the server's sockets for the server to detect the loss of connectivity. There's not anything that can be done about that in K-9 Mail.

What about when switching from 3G to WiFi and back? Do those changes correlate with the behavior you see? K-9 Mail doesn't get warning of the impending change, so does not close the existing connections first. (Perhaps there's a way to make this happen.) So, just like the possibility that the wireless connection changed, there may not be anything to be done about this problem.

K-9 Mail also has a logging mode, which might help with diagnosing the problem: http://code.google.com/p/k9mail/wiki/LoggingErrors

Thanks for the detailed answer!

Ok, with the current IMAP IDLE specification, there is no way we can have less connections than folders to watch. I didn't know that.

About the second part - I will investigate further and then report back.

Any more info? I have not found a flaw in K-9 Mail that could cause this. I believe it is simply due to the fact that the network between the mobile device and the IMAP server does not inform both ends that the connection has been severed. So, K-9 Mail creates new IDLE connections but the server is not aware that the previous ones are no longer valid. I don't know of anything that can be done in K-9 Mail to prevent this.

Sorry for not having answered earlier.

You are right, the problem is not in K-9 Mail itself - the TCP connections are severed by Android switching between different physical connections (WLAN, 3G, ...), and neither K-9 Mail nor the dovecot server get informed.

Maybe there is some kind of "just-before-physical-connection-gets-disconnected hook" in Android, where the TCP connections could be correctly terminated (or at least there could be a simple RESET packet sent to both the K-9 Mail process and the dovecot server)?

This would not help in cases where the disconnection is caused from the outside (like leaving the WLAN reception area, entering a building where there is no mobile network reception, etc.). But it would help in cases where Android "decides" that it "wants" to switch the physical connection, to save power, or because the user disabled WLAN, etc.

I don't see any facility in ConnectivityManager to find out that a network is about to be dropped. An application only finds out after the network has been dropped.

I think you're best bet is to make sure that the number of connections allowed by the IMAP server is a multiple of the number of folders which you want to check with IMAP IDLE. A multiplier of 2x or 3x is probably sufficient.

Perhaps you could modify dovecot to detect loss of connectivity sooner, by adjusting TCP/IP keepalive settings connections from the dovecot side.

Also, the imap_idle_notify_interval might be useful. It should be set to 120 seconds, and perhaps having it send the "OK Still here" might cause the lower levels to detect the loss of connectivity sooner.

Thanks for the advice and thanks for K-9 Mail! :)

I went on to the Dovecot mailing list, and made two proposals on how the situation around this problem could be improved (see http://dovecot.org/list/dovecot/2010-April/048184.html).

I see somebody already pointed out IMAP NOTIFY (and see my comment above from 22 Feb). I've been interested in implementing IMAP NOTIFY in K-9 Mail, but will only do it once dovecot supports it.

Oh well... I should have read comment #2 more carefully, and researched IMAP NOTIFY at the time ;) But better late than never.

While IMAP IDLE only watches a single folder, in general, people are only time-sensitive interested in changes to their "Inbox" folder. One could posit an environment where folders were constantly changing and the changes needed to be communicated, but this will not be the general or typical case. In other words, you need only IDLE on the "Inbox" folder. IDLEing on other folders is, generally, a waste of time and does no good (and, as we see, consumes excessive resources.).
I would suggest that IDLE only "IDLE" on the "Inbox" unless some other specific configuration is set to look at more than one folder per IMAP account. (Joel Snyder/jms@opus1.com)

Joel, more and more people set up sophisticated filter rules right on their mail server, which sort mails into folders based on criteria like sender or receiver address, keywords in subject, spam filter result headers, mailing list headers, etc., as they come in. If you regularly use multiple mail clients this is also the only sane way to filter/sort your mail. Once you have server-side filtering, incoming mails pop up in many places, not only the inbox.

Comment deleted

Dovecot IMAP NOTIFY is coming. :) How about K-9's IMAP NOTIFY?

it's not, as the only server i know that supports it is archiveopteryx, but i'm not aware of its user base. dovecot will likely be the first mainstream server to implement this. if someone gives me an account with NOTIFY support (either dovecot-dev or archiveopteryx, preferably both) it would make development of k9 supporting it much easier, and perhaps inspire me (or someone else, if they have the accounts) to implement it.

i've also considered implementing something that basically has the same effect for gmail accounts, since everything (except trash and spam) shows up in "All Mail".

Hello, I'm observing same problem. I'm serving multiple (tens of them) IMAP folders, which are all quite important, and I want be notified immediately when new email is delivered any of these folders so i want IMAP PUSH notication for them.

Moving from wifi -> cellular network zones and back causes, that it opens multiple connections and after several times I hit connection limit for imap connections (which admin don't want to increase, obviously).

So would be fine if this situasion would be solved somehow.

One possible solution is to support IMAP NOTIFY extension as proposed in comments above, which can maintain one control IMAP IDDLE connection with INBOX, and through NOTIFY extension it will creates fetch requests for other folders on-demand. As we now dovecot already supports this feature.

Thx, for your reevaluated perspective.

Issue 4469 has been merged into this issue.

hi i don't find any information that dovecot already supports this. Please link documentation.

Hi bernhard, the dovecot v2.2 status http://www.dovecot.org/list/dovecot/2012-August/067725.html

2.2 is not stable yet. I guess somebody has to provide an instance of an current 2.2 instance. No way to implement such a feature without an testinstance.

Hi! Re: the number of concurrent connections in K9 that is set to 10 by default and can be configured (c.f. comment #2). I'd like to -reduce- that number but I cannot enter numbers, only choose from a drop-down list -- and the lowest number is 10. :-) Would it be possible to change that? I am running into difficulties with my ISP as K9 creates quite a lot of dovecot/imap processes on their side related to IMAP folders. Thanks!

the various bugs noted here need to be fixed. sounds like there are two bugs.

1) keeping connections open in some zombie state after connection status change. shouldn't you be able to set up a connection change receiver, and blow away ALL cached connections when there's a change event relevant to you?

2) the important one to me, that #20 references: there's the ability to control the number of connections, but the lowest number is >>10<

Thanks for listening.

Noticing this issue on both official Play store version and the latest APK from here, as well. On an htc thunderbolt. I had been running a rooted ICS AOSP ROM until the official ICS release came out just a few weeks back. Was not noticing this issue, at least not nearly to this degree while on the AOSP ROM. now that I'm back on a Sense based ROM, it is broken again.

Dovecot 2.2 has been released.

http://www.dovecot.org/list/dovecot-news/2013-April/000251.html

I can provide a Test server if necessary for the implementation.

I would also like to see NOTIFY implemented in K-9 Mail, which would lower the amount of IMAP connections on servers a lot!

created a feature request for IMAP NOTIFY: https://code.google.com/p/k9mail/issues/detail?id=6264

Workaround to #20:

It is possible to limit push to "1st Class". This feature combined with setting folders which are not time sensitive to "2nd Class" and thus making sure that you have less than 10 "1st Class" push folders should solve the limitation.

See https://github.com/k9mail/k-9/wiki/WorkingWithClasses

Comment deleted

This is not Priority-Medium. Assuming one uses K9+Thunderbird, this issue renders Gmail unusable.

I cant use Thunderbird, because it shows a message that google refused the connection. K9 does not even give feedback on errors. If I close (kill) K9 on my phone or shutdown my phone, thunderbird is working fine.

Imho conceptually more than one connection is not necessary. Please fix.

Outlook+K9+Gmail causing the same issue as described in #28. Especially when switching between WiFi and mobile connection.