Change the message polling loop in com.fsck.k9.mail.store.ImapStore change to use IDLE for new message and flag change notifications.

I'd love to see IDLE support happen. If any enterprising hacker is interested, just say the word and I'll cut you a commit bit.

+1000

I'm a Palm developer coming over to Android. I will be seriously looking into this though I'm guessing I won't get far..but who knows.

ChatterEmail for the Palm Treo was the best email app I've ever seen. In addition to easy switching between accounts and full IMAP support, it had a summary view that had color coded inbox with email from all accounts. It supported IMAP idle, and full attachment send/receive support. I hope to see all those features ported over...and hopefully I will be able to contribute.

turbo2ltr - I loved ChatterEmail. I'd consider chatter-parity to be a pretty wonderful goal.

Comment deleted

Me 3. I want it BAD... .BAD... BAD

I am a Palm Treo several generation converter here. I love ChatterEmail and think that it is the best email app out there on Mobile phone.

I love K9 be able to do similar to chatter and more. here are the list: 1. summary view of mail from various account and color code ( as mentioned here) 2. short cut key for delete email, mark as read, unread, select one or group or multiple emails for deletion or operation 3. quickly to switch between email accounts without cumbersome menu select account list thing like now 4. OPTIONAL: encrypt of email message so corporate email can be "secured" 5. Some ability to filter or rule on mail coming in (Chatter has this)

Additional feature: 6. Ability to sync with Contacts and calendar on corporate email (EXCHANGE) via IDLE 7. Search for contacts on Corporate email.

It is a long list and I am willing to help. I'll try to download your current K9 code and look for some open source code on IMAP IDLE or just port the IDLE code open source from chatter over.

I am already working on this support, and in fact have a "working" version now, however significant changes are necessary before this is stable and efficient. I have been discussing the details of the redesign with the original application's author and have begun work on a design that I think will make it into the upstream git tree.

In particular, I am changing the MailService to takeover all folder and message handling and separating it to run its own separate process. I am also extending the main MessagingController work queue to use a separate queue for each account, this way multiple can be waiting in IDLE at once, with minimal overhead for other cases.

In the meantime, I'm interested in releasing these enhancements through your project, but I do want to stress that I will be pushing hard to get my changes submitted upstream as I think long term a fork is not the right way to go unless Google proves unwilling to accept our changes.

P.S. I have written a script which swaps package namespaces back and forth so that I can load my changes onto my phone while maintaining a clean diff against Android's git tree.

jasta00: That's a good point. Should we somehow try to merge this back in with Google's version before it gets too far removed? It seems like they either (a) didn't much care about its quality or (b) were rushed (more likely).

Jesse?

jasta00 - I'd love to get everything we're doing merged upstream into the core application as we prove that it's reasonable and works well. Is there a chance we could talk the original author into joining us on the k9 list to talk through collaboration strategies?

I consider collaboration with upstream to be my #2 priority, coming in a close second after "build a better email application that I can ship today for myself and others."

Would you be up for checking your namespace switcher into the K-9 tree?

Yes yes yes! I, too, am a former ChatterEmail user and would love to see similar functionality here! I was considering writing my own Chatter-clone from scratch until I learned about K-9 -- now I'd like to help make K-9 as close to Chatter as possible!

Thanks

Hello, folks. This is Marc, author of ChatterEmail. I'm no longer at Palm, and I'm thinking about doing some magic for Android... It's nice to see some old users around!

Marc

Hallelujah! ChatterEmail was my absolute favorite mobile email program and I miss those great features now that I've switched from Palm to Android. I was slowly resigning myself to having to write my own Android mail app to recreate all my favorite Chatter features, but now that Marc is here... ;-)

I am dying for the summary view. And the ability to move messages, flag messages and view only certain folders...

• m

That's excellent news! It seems like we've all been missing Chatter since switching to Android.

Oh, man - if that really IS Marc, things are looking greater than ... well, Android has always looked good, but now? Jeez. Let's invite Steuart Dewar (Datebk3/4/5/6) over as well - and maybe Marshall from PDA Performance (Contacts5) and I will be in PDA heaven.

Sorry, this was really OT. Back OT, yes, please, bring IDLE support to K9, and maybe let us set our own mail checking interval?

Just another "me too"... Please support IMAP IDLE soon!

I'm told that another Android hacker has a patch to add IDLE support. We've got some housekeeping to do before I ask for permission to apply it.

Any of you folks who have expressed interest in hacking on K-9 or improving the state of internet email on android should drop me/us a note and get a commit bit. I'm a pretty benevolent dictator and would love more collaborators.

-jesse

If we're talking about the same patch (and I suspect we are) note that a lot more work is needed to really properly support IDLE without a great deal of unnecessary reloading of entire message lists from scratch. See RFC 4549 for the details ( http://www.rfc-archive.org/getrfc.php?rfc=4549 ). I would probably be willing to help with this as time permits.

Any word?

I really don't see how an email client on Android can use IMAP IDLE. Once the phone goes to sleep, how will it keep the TCP/IP socket open to receive the notification that new mail is available? Unless there is a facility to have the radio wake the phone when communication is received on the socket (which I don't think there is, otherwise folder synching would never fall asleep in the middle of communication), IMAP IDLE seems doomed.

Researching "network tickles" and how to send them would probably be a worthwhile direction.

Actually, if implemented as a Service, there should not be any issue with implementing IMAP IDLE.

There was already a working patch floating around.

Well, my interest is piqued. Did the patch work when the phone went to sleep? Do you have a pointer to the patch?

http://devtcg.blogspot.com/2009/01/push-services-implementing-persistent.html

I know this issue is about IMAP IDLE, but I think an proposal of alternative instant mail notification is appropriate. This is just an idea - I'm sure you can judge whether it is useful.

Instead of keeping open a socket (which would probably, even if it works, kill the battery), have the mail server notify the phone of new mail, e.g. via a phone call.

ToDo server-side: Upon receipt of a new mail have the mail delivery agent (e.g. maildrop) use a SIP provider to call the phone.

ToDo phone-side: Whenever a call from the SIP account's number arrives, reject it and check for new mail.

Advantages: - does not cost any extra money since the incoming call is rejected - does not drain battery since the phone is ready to accept calls anyway - nearly instant notification

Disadvantages: - need to have access to maildroprc (or similar) on the mail server - need a server-side interface to a SIP provider to make calls (SMS is easy, but they arent free) - is it even possible on android to handle incoming calls as described above?

I like the idea of using SMS. The only "problem" would be to flag the message so that it can be handled (somehow) in the background so that the user does not experience two notifications. I prefer SMS over a phone call because I think its less obstrusive: a phone call brings up the "Incoming Call" screen and activates the "Phone Call" activity (or whatever its called). An SMS, if notifications are turned off, happens almost silently, except for the message in the notification bar. Also, I think waking up and killing the Phone app would murder the battery life.

Does anybody know now the Gmail app works? Is it a non-public API?

Gmail uses its own network system to get the notifications. They are called "network tickles".

IMAP IDLE and IMAP NOTIFY are the right way to do push for IMAP accounts.

So what's the status on this?

Thanks,

Andreas

r718 | danapple0 | 2009-07-25 11:02:54 -0500 (Sat, 25 Jul 2009) | 2 lines Changed paths: A /k9mail/branches/issue4-1.X (from /k9mail/trunk:717)

For working on Issue 4, IMAP IDLE

Issue 581 has been merged into this issue.

Any progress?

r733 | danapple0 | 2009-09-10 02:14:42 -0500 (Thu, 10 Sep 2009) | 13 lines Changed paths: M /k9mail/branches/issue4-1.X/src/com/android/email/Email.java M /k9mail/branches/issue4-1.X/src/com/android/email/MessagingController.java A /k9mail/branches/issue4-1.X/src/com/android/email/mail/PushReceiver.java A /k9mail/branches/issue4-1.X/src/com/android/email/mail/Pusher.java M /k9mail/branches/issue4-1.X/src/com/android/email/mail/Store.java M /k9mail/branches/issue4-1.X/src/com/android/email/mail/store/ImapStore.java M /k9mail/branches/issue4-1.X/src/com/android/email/service/MailService.java

Issue 4

Operational implementation of Push mail using IMAP IDLE. Fetches new mail from the Inbox of any IMAP account which is set with an automatic sync interval of greater than one minute.

Needs to be enhanced to use a "Folder Push Class" for determining which folders will participate in Push mail.

Does delete messages or update the flags of messages based on push notifications.

Push mail using IMAP IDLE is available K-9 build 1.100 and later. 1.100 is currently in the Android Market as K-9 Mail Alpha Dev Release.

Hello, I installed your latest BETA K-9 on my G1Dev phone yesterday, but cannot get the idle function to work, it always seems like I need to go back to .114, it's the best so far.

Gary Brant in NY

Gary,

Can you provide a bit more information? Are you using an IMAP account? Which IMAP server or ISP? Can you get the logs from K-9 using ddms or logcat?

Dan.

Trying to new beta, love the idle. But the "Syncing and sending suspended..." is completely annoying when you're transitioning back and forth between 3G to WiFi. K-9 should at least try to reconnect every # minutes, no?

bengrimm,

I haven't tried switching between networks. I mainly intended the notification for when losing radio coverage. Isn't the notification automatically cleared once the new network connection is established? There may be a way to delay the notification until it is certain that no connectivity will be established soon.

Dan.

Dan,

It doesn't seem to clear itself until the original network connection is established again. If you're on wifi and have ip address 1.2.3.4 when K-9 connects to the server, then the phone goes to sleep and falls back to the EDGE or 3G connection with a different IP address 5.6.7.8. K-9 thinks it's lost connectivity until the phone wakes up and reestablishes the wifi connection with ip 1.2.3.4 again.

Also, shouldn't it be Ongoing instead of a Notification?

Ben

Ben,

Thanks for the feedback! I'll experiment with switching between WiFi and 3G/Edge and learn more how that works. You may be interested in Issue 657, a request to make these notifications optional.

Also, I'll check out adding the ongoing flag to the notification and see if that makes it more sensible.

Dan.

Dan,

Sure thing. Network flapping is an issue with so many android programs, it makes one question the value of using the WiFi connection.

I do have the ongoing flag set, and that works for the "Checking..." messages, but this message doesn't seem to honor that preference.

Ben

Hi Dan,

I have been trying the new 1.100 beta in an attempt to utilize push mail on my Yahoo E-mail account. No problems with IMAP access but apparently Yahoo doesn't provide the IDLE capability on their IMAP servers and instead chose to implement the push mail function using a form of SMS message to notify the client that a new message is available. Not sure if you or any of the K-9 development team are familiar with this form of push mail configuration or how difficult it would be to implement in K-9, but no doubt there's plenty of Yahoo mail users who would be grateful for it.

Thanks for your efforts to bring this product to the Android community.

-Ed Rico

I tried it yesterday but I always have 'idle not supported error'. I hoep it's because all of my mail provider really don't support this.

Just to be sure : at wich frequency the idle imap is used ? What if I choose 'every minute' or 'every 5 minutes' ?

(I'm not sue of the correct settings because i'm not really fluent in English).

Correct me if I'm wrong, but isn't the checking frequency ignored when IDLE IMAP is used?

geoffrey.metais,

If you are seeing "idle not supported" then your IMAP server does not advertize the "IDLE" capability, so will not support push mail using IMAP IDLE.

geoffrey.metais & dbriskin, IDLE is supported independent of the polling frequency. The polling will continue to work as before, even when IDLE is in operation. This is useful because the updating of flags (Deleted, Seen, Flag) is not 100% reliable through IDLE. Polling and IDLE are independent. Neither, either, or both, can be used. Using both offers the best combination of reliability and fast message delivery.

I switched from polling @ 15 minutes to polling at 1 hour, and IDLE on three folders, but poll on about 12 folders.

Dan.

Thanks for the explanation, Dan. It makes better sense now.

edrico1,

I'm sure it would be possible to integrate Yahoo's proprietary SMS system into K-9, and somebody else may choose to do it, but it is not something in which I am interested. If it is a published standard, there would be more chance of it getting done. It would be better for Yahoo to adopt the industry standard IDLE protocol.

Dan.

My isp for mail is fastmail.fm in Australia, per Dan's question.

Gary (galstg),

I configured my fastmail.fm account per the instructions on http://www.fastmail.fm/help/remote_email_access_server_names_and_ports.html and IMAP IDLE is working fine for me for that account.

If you can provide any more information, I could look into it further. As log file from ddms or logcat is always most helpful.

Dan.

OK, thank you for the explanation.

So I managed to get it working with Gmail. I seems to work very weel, it's event faster than the Gmail app push according to my only 2 tests ;).

I'm still looking for mail provider supporting idle to do more testing

Ben (bengrimm),

I have tried switching back and forth between WiFi, Edge and 3G, and did not experience the problem you described. Sometimes, K-9 did not even suffer a temporary lack of connectivity. Even when it did, as soon as the new network was up, the new connectivity was noted and syncing/sending/polling was resumed.

I can no longer reproduce it -- although I did reboot my phone between when the problem occurred and when I just tested, so whatever was causing the network errors may have gone away. Thanks for looking into it, I'll let you know if I can isolate the conditions if it occurs again.

Hi Dan,

I agree that it would be much better for users as well as client software developers if Yahoo were to implement the industry-standard Imap IDLE capability. The only reason I can imagine for their non-standard implementation is to arbitrarily restrict this function to iPhone users who apparently have this capability built into their OEM E-mail application. Perhaps one day Yahoo will see the light, but I'm not holding my breath.

Thanks for your reply, -Ed

Comment deleted

Comment deleted

My email provider is fusemail.com IMAP IDLE works ok, but not always. I sometimes get immediate notification with push, but sometimes it doesn't work and I have to poll the emails to get them. Any ideas why push does not work every time?

fusemail.com imap.fusemail.net

smtp.fusemail.net

And please make the automatic polling on connectivity change optional because it slows down the phone a lot after each call typically.

fusemail does not provide free accounts (at least, I could not find a reference to any), so I cannot test on fusemail.

Interesting point on the automatic polling on connectivity restoration. I never tried making a phone call. I also didn't realize that a phone call would cause a connectivity change.

Comment deleted

You get actually get a test account for 14 days free here: http://www.fusemail.com/order/order.php

I would greatly appreciate if you could help me out with fusemail. I have been dreaming to be able to get push functionality. Let me know if I can assist you in any ways.

I'd prefer not to give my credit card info, knowing that I just want the trial. Do you any spare email accounts in your hosting package to which you could grant me access for a few days? I won't be able to get to this until the upcoming weekend, however.

It looks like 3G networks don't disconnect data connectivity during a phone call, but Edge does (at least on T-Mobile.)

I think I'll just eliminate the polling on reconnect, but leave the automatic sending of pending outgoing messages (which is almost zero load if there are no pending messages), and the automatic reestablishment of IMAP IDLE connections.

dan,

your comment about 3G and Edge is afaik correct. That's what I experience on my G1 as well.

Since I've got the same problem as christian - unreliable push: works, but not always - I'd be happy to provide you with accounts on one of my mailservers. They are standard courier imap mailservers as packaged by Debian (etch) employing ssl. Just send a mail to my google address.

This is probably related to Issue 153. K9 seems to be kicked out of memory by Android? As soon as I start K9 again, it reconnects to the IMAP servers (the phone lost its connection before) and finds the new mail.

@danapple: I will see if I can register a new account for you. I'll let you know asap.

Code is committed to trunk. Expect in K-9 build 2.000.

r854 | danapple0 | 2009-10-21 19:41:06 -0500 (Wed, 21 Oct 2009) | 19 lines Changed paths: M /k9mail/trunk/AndroidManifest.xml M /k9mail/trunk/res/menu/folder_list_option.xml M /k9mail/trunk/res/menu/message_list_option.xml M /k9mail/trunk/res/values/arrays.xml M /k9mail/trunk/res/values/strings.xml M /k9mail/trunk/res/values-de-rDE/strings.xml M /k9mail/trunk/res/xml/account_settings_preferences.xml M /k9mail/trunk/res/xml/folder_settings_preferences.xml M /k9mail/trunk/src/com/android/email/Account.java M /k9mail/trunk/src/com/android/email/Email.java A /k9mail/trunk/src/com/android/email/EmailReceivedIntent.java M /k9mail/trunk/src/com/android/email/MessagingController.java M /k9mail/trunk/src/com/android/email/MessagingListener.java M /k9mail/trunk/src/com/android/email/activity/Accounts.java M /k9mail/trunk/src/com/android/email/activity/ChooseFolder.java M /k9mail/trunk/src/com/android/email/activity/FolderList.java M /k9mail/trunk/src/com/android/email/activity/MessageCompose.java M /k9mail/trunk/src/com/android/email/activity/MessageList.java M /k9mail/trunk/src/com/android/email/activity/MessageView.java M /k9mail/trunk/src/com/android/email/activity/setup/AccountSettings.java M /k9mail/trunk/src/com/android/email/activity/setup/AccountSetupCheckSettings.java M /k9mail/trunk/src/com/android/email/activity/setup/FolderSettings.java M /k9mail/trunk/src/com/android/email/mail/Folder.java M /k9mail/trunk/src/com/android/email/mail/MessageRetrievalListener.java M /k9mail/trunk/src/com/android/email/mail/MessagingException.java A /k9mail/trunk/src/com/android/email/mail/PushReceiver.java A /k9mail/trunk/src/com/android/email/mail/Pusher.java M /k9mail/trunk/src/com/android/email/mail/Store.java M /k9mail/trunk/src/com/android/email/mail/store/ImapResponseParser.java M /k9mail/trunk/src/com/android/email/mail/store/ImapStore.java M /k9mail/trunk/src/com/android/email/mail/store/LocalStore.java M /k9mail/trunk/src/com/android/email/service/BootReceiver.java M /k9mail/trunk/src/com/android/email/service/MailService.java

Issue 4: Implements Push Mail for IMAP accounts using IMAP IDLE

This commit contains the entirety of the changes performed in the issue4-1.X branch from revision 718 through revision 851. Because the issue4-1.X branch was up-to-date with trunk revision 847 at revision 849, the source of this commit was not an "svn merge". Instead, it is merely a copy of all changed files from the issue4-1.X branch to my trunk working copy and a straight commit.

Also: Issue 551 Issue 628 Issue 650 Issue 654 Issue 656 Issue 682 Issue 696

Sorry I couldn't set up an account for you at fusemail. Thanks for having removed email check upon connection change, it is much better now.

Push functionality reliability is the same after the last updates: only around 70% of the new emails are pushed.

The remaining issue may be related to how IMAP idle works (correct me if I'm wrong):

Should the internet connection fail at a given point of time s, K9 will reconnect as soon as possible (t). However, messages received server-side during the downtime were unable to be pushed and will not be "re-pushed" by the server. Thus, K9 has missed some e-mails.

Possible solution: Upon reconnection, do a "mini-refresh" of the involved folders.

You can simulate the described behaviour by emulating network outage using the background sync switch, which is respected in its newest beta version (yay!)

K-9 does do a "mini-refresh", as you put it, every time the push connection is re-established. I will test to make sure that functionality is still working. It works by comparing the highest loading UID against the UIDNEXT supplied by the server upon connection establishment and SELECT, and loading all of the UIDs in-between those values.

Comment deleted

I got a pretty good reliability of push lately. I might have identified a reason why some emails were not beeing pushed earlier: all the folders where push is activated are market with (Push) when all the folders are listed. I noticed that sometimes the mention (Push) was not present anymore. It was sufficient to go into 'Account settings' and coming back without making changes for the folders to be marked again with the mention (Push) and accordingly they started properly pushing again.

Hello, To use the IDLE command, do you turn off the Email Check Frequency and enable Background Synchronization?? I am only wanting my Inbox on one account to be checked and I check my other account when I want to. I am using 1.991

I had the client initially set to check every 1 min but I presume this will drastically drain my battery??

Cheers Neil

Neil,

To use IMAP IDLE, make sure the "Folder push mode" or "Folders to check for push" is set to something other than "None" I recommend leaving the "Check Frequency" on an hour so, because it serves as a backup. Some types of changes can be missed by IMAP IDLE. And yes, be sure the Background synchronization is not set to Never.

Question: Why is the folder default class setting for push as "2nd class", rather than "Same as sync"??? Is each folder a separate connection?

I've read the "class" explanation in the wiki, but there is no "push" discussion.

By the way, either I don't understand the "class" explanation AT ALL, or there are a bunch of errors in the examples (ie, in most of them).

Why is the folder default class setting for push as "2nd class"

Push can be (but isn't always) a much heavier burden on K-9, the device, and the server than regular sync, so only the Inbox defaults to having push enabled.

Is each folder a separate connection?

Yes

I've read the "class" explanation in the wiki, but there is no "push" discussion.

It was written well before the push feature was created, but the principles are the same.

or there are a bunch of errors in the examples (ie, in most of them).

I'd be happy to correct the documentation if you can point me to the errors.

Dan.