IssueTracker

You don't need all that other behavior... just a "hanging get" would let you
implement the other stuff on top of it. I.e., let a http get request be serialized to
the database in some way, so some other process can pick that get back up and respond
to it.

But yeah comet in app engine would be great! There is already a third party offering
this service, which is yet more evidence that it's useful:
http://popcnt.org/2008/05/missing-services-for-google-app-engine.html

Yes indeed , this will be the most sought feature in google apps.
if not can you document the maximum processing time allowed for an request.
by this we can atleast start of with some alternatives.

Regards
Dave

I'm really hoping that this will be added together with the promised XMPP support...
That'd be the logical place for it.

Fingers crossed the Google guys realize that this has more value than "support Java" etc. in terms of facilitating
the development of really interesting applications...

I can't fully run my app without it.  Without comet type support (I prefer to use
DWR), I'll have to do a mashup and send requests to another server for processing there.

It'll be awkward to have the app and data on appengine but conversations (chats)
about the data going to another server and being pushed out from there.

ICEfaces can also make use of this feature, compare:

http://icefacesauction.appspot.com/

http://auctionMonitor.icefaces.org/

The demo is a lot more fun with Ajax Push.  This can be implemented with either Servlet 3.0 or (since the App
Engine is using Jetty) Jetty Continuations.

The jetty team would certainly be interested in working with google to make our
continuations available.

Even if behind the scenes they we implemented as a blocking wait/notify, this would
at least allow code to be deployed with the asynchronous semantic.  Google could then
do their own calculation if it is best to make this truly asynchronous or just throw
more memory/threads/machines at the problem.

Note that the continuation API in jetty-7 from http://eclipse.org/jetty has been
somewhat refined and the thrown exception has been removed.   This updated API is
intended to be a bridge to servlet 3.0 and applications written against it can be
deployed on any 2.5 or 3.0 servlet container, but will only be truly asynchronous
when running on jetty or a standard 3.0 container.
 

I really like this feature.
Server initiated rendering is very cool.

I vote Yes!

I would also love this feature. I'm looking to develop a multi-player AJAX game and
was seriously considering using appengine as the host, but due to the lack of
official comet that makes it near impossible. Still, I'll be watching this issue as
well as possible third party appengine comet support offerings.

I vote this too! Without comet support ,I have to remove refresh function of my app:
http://geo-twitter.appspot.com/ 

I too vote yes

If you'd like to see this implemented, star the issue, do not leave a comment. Every
time a "me too" comment is left, 153 emails get sent to people like me, which makes
my phone beep. Additionally it makes the ticket less readable :)

yes

[Comment deleted]

[Comment deleted]

this issue is discuss for so long. may i know will this feature be available for next
app engine release?

Can't wait for this feature to get it working with pubsubhubbub protocol.. I'll guess
that this feature becomes available when google launches the open beta for wave.
Anyway, xmpp is on the roadmap. I would like to hear some dates :)

ya. we all want to hear dates .hehe=)

m.e.siersema brings up an interesting point. To truly make Wave succeed, having wave protocol built into google
app engine would be a huge boost for wave development! Doing that would also fulfill at least in part, all the
comet type requests.

See http://docs.google.com/View?id=dg7nkx7d_3239mc4rs4g for a simple and secure proposed solution.

And for those who were really quick: I've added a bit about how to use my CallHome without the need for
browser support: http://docs.google.com/View?id=dg7nkx7d_3239mc4rs4g.

[Comment deleted]

Wouldn't generic support for CallHome allow users to instruct a web application to
make requests to any server they desire?  That doesn't sound like the best security to
me.

Isn't this the stuff you've been waiting for: http://pubsubhubbub.appspot.com ?

I see now what is wrong with pubsubhub... its a server-to-server protocol. still no long polling on GAE :(

Couldn't you add a client endpoint to a small app and have the publisher (via post) in the same app too?
Think "chat engine"?

Hey there,

Please remember that 262 people get e-mailed every time a comment is left on this
ticket, it causes phones to beep, including in the middle of the night, all over the
world. :)

In the absence of official comment from the AppEngine team on the status this
particular feature request, I think a much better forum for discussion of
alternatives would be the appengine groups themselves.

Thanks!

If I am to use an external comet server I already have many great open source options
(http://cometdaily.com/maturity.html), Orbited for example is brilliant.

The point is to be able to serve comet connections from within the GAE
infrastructure, thus taking advantage of Google's reliability, smart pricing model,
and auto scaling capabilities. It would also remove cross-domain scripting issues of
having separate comet and "regular" web servers.

I don't see comet capabilities being shoehorned onto the current GAE request cycle
because, from an engineering point of view, GAE is diametrically opposite to the
requisites of comet. It's designed to handle massive ammounts of short-lived http
requests, scaling automatically. To accomplish this no simple feat it has to
compromise on every aspect that makes comet possible today (most notably, the server
ability to hold an http request).

Google could and should, though, offer a separate but integrated comet server
companion to GAE. Bonus points if they used an open source solution like Orbited
(which is written in python BTW) that would not lock my apps into GAE.

Sorry in advance for the beeping phones ... :)

I clicked to support this issue a while back, the ticket itself was posted over a
year ago, there are over 250 developers supporting it, and still I have yet to see an
actual reaction from a google employee on this issue (apart from the label changes
and acknowledgement status).

So hi Google people! May we have some feedback on this?

Specifically:
- Has any decision been reached with respect to comet-type functionality?

If yes, and it's a go:
- Could you give us a rough idea on when we might expect a release?
- Do you have an overview on how your version of such functionality will look like?

If yes, and it's a no-go:
- Could you give us a short explanation why this is not going to happen for GAE?
- Is HTML 5 (event-source, Web Sockets) what we should wait for?
 
If no:
- No more questions, I'll try to be patient :) (and check alternatives, not including
cross-domain scripting)

Thanks for any reply, regards,
Sam

1. The CallHome Server solution only ever tells the the client to do its normal poll sooner. It can never make the
client call any URL other than the one it normally polls.
2. I agree that I'd like Google to provide a solution that is complimentary to GAE. I think a CallHome service is
the right and minimal way for them to do that. If done in the general way that I specify then it could also be used
by non-GAE apps. Then if it becomes popular it could migrate closer to the user. Actually it only needs to be
just outside the users firewall/NAT box, and it could easily be incorporated into that box [remember: doesn't
need to keep any state across reboots].
3. I'm trying to get up the energy to write an IETF InternetDraft, which I haven't done for 10 years.

Supposedly the XMPP support just released is the solution to this problem (issue 231). I haven't worked out how
to use it. How does the javascript on the web side of your application receive the sent XMPP message? Note that I
posted this question in issue 231 as well and I apologize to the people watching both issues.

XMPP support in GAE is not a solution to this problem. GAE doesn't provide an XMPP client for users of your app.
I'm still waiting for some information about when long-polling will be supported in some way by GAE.

Please Google give us some idea as to the way you see GAE used for real-time apps.

can anyone share what workaround that you folk doing right now? anyone doing "relay
to external" for comet in java?

A minimal external comet server just tells the browser to poll. I have a plan (and some code) for this at
http://code.google.com/p/callhome/. An alternative that doesn't require an external server is to have a
separate local program that talks xmpp/jabber/IM to gtalk (or other), and which delivers xmpp messages
using the same comet technique as in CallHome from a localhost URL. This looks pretty easy (anyone want to
start a project to do that?). But we don't want users installing programs. We like the web because it avoids
that. However I think the way google have done xmpp points the way to their solution. IM addresses can be
something@yourapp.appspot.com. So I think we can see that they are going to let browser clients connect
with BOSH or some other comet system to a virtual IM service at yourapp.appspot.com and receive xmpp
messages, without violating same origin in the browser. Just guessing...

Robert, can you eleborate how to run your demo callhome? it's in "C" ? only 1 file?

[Comment deleted]

Compile/link against GNU libmicrohttpd. Only arg is port number. Server pokes it with http://.../idcreate?
id=number Then client pokes it with http://.../toclient?id=number&httpDomain=url to set up comet stream.
Then server pokes from time to time with http://.../fromserver?id=number&extra=number (which causes stuff to
be sent to the client on the comet stream). The client javascript expects postMessage messages to arrive in top
window. I don't have a javascript+gae test yet: just testing by entering urls by hand.

Is xmpp + emite (http://code.google.com/p/emite/) the solution for this issue at
least for GWT developers?

Does emite run on GAE and support bosh?

[Comment deleted]

My program wouldn't scale because it uses thread-per-comet-connection. I might use it as a stopgap if google
doesn't come out with something in the next few months. By the way to alessandro: the problem isn't a lack of
suitable software, rather it is a missing bit of google infrastructure. When we know how they're going to fill that
gap, then we'll know what software we need. Its pretty obvious that it will be the same as for wave: xmpp.

[Comment deleted]

[Comment deleted]

I vote too

I vote too!
Please support DWR2 or DWR3 for Reverse Ajax!

I vote also.

Like Ted I ask for this to get a working ICEfaces AJAX Push.

To reply #40 problem with browser client like emite are the restrictions for javascript
in browsers. Especially the one that denies cross-domain requests. Otherwise it would
be a great solution for all of this. Probably there could be some sort of XMPP gateway
on each GAE domain (account) created by Google directing the traffic to GTalk servers.
But it would rise the XMPP traffic for GTalk servers heavily, as everybody would use
this, so I think its a no-go.

In comment 36 above I predicted that they would go with XMPP for real time. However since then we've seen  
websocket software from google: Pywebsocket, then Go came out with a websocket library. So now it looks more
like they'll provide websocket in some way. We notice that Go has Erlang-like concurrency and should support a
lot of simultaneous websocket connections. We also note that though Go seems C-like it is actually a safe
language that could be sandboxed like python or java. Still websockets and xmpp are overkill for most
applications, with long-running tcp connections snaking across the Internet. A CallHome server could be a much
cheaper option that would satisfy most needs. I'm trying to get up the energy to rewrite CallHome in Go, which
would scale.

can we close this in favor of Issue 2535:     Web Sockets API ?

I think that this is still needed, even when Web Sockets is supported. Many people
don't have the latest versions of a web browser, or use IE, and many times you want
your site to support those users as well.

I don't understand 2535 at all. There is nothing now to stop the browser app talking websocket to some other
server. We just don't want to have to set that up. And if we're talking about GAE supporting long running GAE
processes talking continuously to long running browser applications then it is overkill for most applications.
Certainly mine. We just need to get a real time message to the javascript in the browser. If the only message you
could get was "poll the server now" that would be good enough [and avoids a lot of security issues].

David, I agree with you.
Comet is nearly dead. Now it's time for web sockets.

Comet may be dead in the long run but it will be a long time until we can assume
that all our users have a websocket compatible browser... There is what, one
development one so far?
This product: http://www.kaazing.com/products/kaazing-websocket-gateway from
a quick glance looks like the sort of thing I would hope for Google to provide with
App Engine... Obviously we need a Google solution as it needs to be served from our
domains.

Given that XMPP is already supported on App Engine what would make the most
sense to me is an XMPP (<)-> websocket proxy wit fallback to Comet alternatives
where websocket isn't supported.

Let's not forget that WebSocket support can be emulated in a browser by Flash.
http://github.com/gimite/web-socket-js
With this, you can cover most browsers, and let the newest browsers use their native
implementation.

Google Appengine could support JETTY's reference implementation of the HTML5 WebSocket.
http://blogs.webtide.com/gregw/entry/jetty_websocket_server

The trouble is the reason JETTY's implementation is so efficient is they use
suspended requests that allow HTTP/TCP requests to be "frozen" without sleeping a
thread instead they use a thread-pool and non-blocking IO sockets.

So if google were to adopt JETTY's websockets they'd most likely have to adopt their
non-blocking "suspendable" requests or Continuations in the servlet 3.0 API.

I think the only real "all-browser" compatible support would be to do something like
Adobe's BlazeDS or Java Atmosphere, and support 3 "COMET" strategies.

1. Normal poll and respond if no messages (expensive on the server because clients
would constantly be pinging them, or clients would have to tone down their polling
and would have response latency increased).

2. Long polling COMET, this is the strategy listed in this ticket of connecting a
request and holding on to it on the server side until a response comes through (for
this to be efficient there needs to be an implementation that can "suspend" the
request so that it does use up thread/CPU resources).

3. Streaming COMET some browsers (see iframe AJAX) can stream messages without having
to close the request, the downside is you need 2 HTTP requests available so you can
send on one and just receive on the other.

4. WebSockets HTML5 support this is the holy grail, but probably 3 - 8 years before a
majority of users have full support for this (unless it can be baked into a
ubiquitous plugin like Flash)

It's a "Java" application engine.  Why not support the standard Java API in Servlet 3.0 for asynchronous requests
and avoid all the problems with which nonstandard API or undefined protocol to support?

Re: Comment 59

Still need something for Python though, and it would be nice for the environments to have a consistent
implementation.  Although I agree you could use the same underlying architecture and just wrap the API calls
different(ly).

Originally this was only a Python application engine :P
Servlet 3.0 is the best choice beside the fact that it's JSR is not ready yet.

Using a servlet based architecture like that wouldn't fit in with the App Engine
architecture anyway...
It's already structured into front-end servers which dispatch incoming requests to the
appropriate app server, so the logical step is to support long-running connections at
this level (which avoids the issues with long-running java processes, Google already has
tech for supporting low-cost long-running connections like Go) and interface with the
app through the XMPP API that is already there.

The front-end dispatch could easily be adapted to handle asynchronous requests, and the whole point of AsyncContext is that you avoid blocked threads on the server.

Jetty's websocket implementation is quiet separate from it's continuations or async
3.0 support.

Googles problem is that their app engine is actually remote from their HTTP servers
(and eventually websocket servers I'd assume).  So for these sorts of features, they
have 2 problems:  what API to support (in asyncs case servlet 3.0 is now the obvious
choice, not so clear for websockets)   and then how to actually implement it in their
architecture and what limits they will apply to it.

Atmosphere already supports GAE (servlet 3.0 is more low level and unlikely to). Take a look.

@rasputnik, can you elaborate more... cant find info or demo on this

[Comment deleted]

Thank you rasputnik for the information. Atmosphere looks very promising.

https://atmosphere.dev.java.net/

For those interested, check this out:
http://n2.nabble.com/ANN-Google-App-Engine-now-supported-
tt3936201.html#a3936201
tiny: http://tinyurl.com/ycss4sc

If I am not mistaken, that guy is saying that long-polling works on GAE with the
Atmosphere Framework now. Great!!!

Jett-7 Continuations and cometd based long polling also work.... they just consume
threads and there is a maximum time that you can hold the long poll open (30s I think).
I'm sure atmosphere also consumes threads in this way.

@gregory is right, the 30s timeout needs to be handled (either by your framework or your application).

Fom what I've seen atmosphere is a nice abstraction layer on top of $YOUR_COMET_FRAMEWORK_HERE
(it takes advantage of Jetty continuations if they're available in the servlet environment, for example).  
Also seems to play nice with others (Scala and JRuby support) and handles browser-specific 'quirks' , etc.

WebSockets is going to be niche for the next few years or so, but Atmosphere will take advantage of it
if the browser supports it - best of both worlds really..

Some actual code at:

http://weblogs.java.net/blog/jfarcand/archive/2009/11/06/servlet-30-async-or-atmosphere-you-decide

Only downside is you'd have to write your webapp in a JVM language :(
We still need a Python equivalent.

afaik multiple threads are not allowed in python on GAE so this kind of approach
would be a non-starter.

Another question is whether we get charged for CPU time for the sleep period? In
that case polling would end up being cheaper.

But I wonder if even for Java this would work if your code ends up running on
multiple physical servers... You would surely need some kind of XMPP / pubsub
backend architecture anyway. It would make a lot more sense for Google to provide
this kind of infrastructure rather than what amounts to a pretty ugly hack like this.

gregory.j.wilkins, jetty's continuations are setup so that when you initiate the
continuation jetty lets go of the thread, so that threads are not held up.  the time
limit is actually about 60 seconds before browsers tend to time out, I personally use
50 seconds right now.

Long polling every 30 secs is better than just polling, I guess. Do I need a separate thread in python? Can't I just
loop sleeping, checking memcache every sec (or less) to see if there is something to respond. Client needs to
cancel that waiting request and initiate a new one when there is user interaction to send to the server?

long polling saves on bandwidth but can be a pain to setup, and often ties up a thread
in the web server if you dont do it right (such as using jetty's continuations in
java).  the way that long polling works, the browser makes a request using an ajax
request and the server just doesnt respond to the request until it has something to
tell the browser or until the browsers timeout limit is approaching.

Some browsers cant even do this, such as IE6 I think, and so you have to use an odd
iframe methodology instead for those.

Wednesday,
Jetty continuation do try to give up the thread if they can.  But if they are not
running on a platform that provides continuations, then a waiting continuation is
provided, which blocks the thread and waits for the resume.  Jetty-7 continuations
can be used in any 2.5 or 3.0 compliant container and will wait without threads on
jetty-6, jetty-7, jetty-8 or any servlet 3.0 container.  On other containers they
hold the thread while suspended - which should work on GAE (but might cost a bit).

 

See the new roadmap: http://code.google.com/appengine/docs/roadmap.html.

good!!! who will take this issue?

i saw the roadmap. may i know for the push comet. will that be additional GAE api to
use comet or we can use existing DWR/atmosphere ?  the current atmosphere is using
long-pooling and not push?

this would be a wonderful feature! thanks BigG!

Looks like this feature will be out very soon.. Its called the "Channel API". This links to a session from last Google I|O 2010 which describe exactly how it works ..
http://code.google.com/events/io/2010/sessions/building-real-time-apps-app-engine-feed-api.html

Have fun watching ;)

i need this featrue toooo

Any news on if this will make the next release? ("Channel API" that is)

We're trying to get this out as quickly as possible, though we have no ETA on when it will be out.

Channel API shipped in 1.4.0!

Hey guys what is the alternative to this now? Firestore takes 40 minutes to compile for iOS and is a mess.