My favorites | Sign in
Project Home Downloads Issues
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 158: Adding support for getting body of requests or responses
35 people starred this issue and may be notified of changes. Back to list
Status:  Migrated
Owner:  ----
Closed:  Mar 2013

Blocking:
issue 238


Sign in to add a comment
 
Reported by saleh...@gmail.com, Jul 3, 2011
Adding support for accessing body of requests (useful for http post requests) in the onResourceRequested call back.

Which version of PhantomJS are you using? 1.2




Jul 4, 2011
Project Member #1 ariya.hi...@gmail.com
I'm not sure how we would encode the content. Typed array comes to mind, but that's not widely supported yet.
Jul 15, 2011
Project Member #2 ariya.hi...@gmail.com
(No comment was entered for this change.)
Summary: Adding support for getting body of requests or responses
Jul 15, 2011
#3 jgorn...@gmail.com
It seems the body would be part of the "text" property in the content object of the HAR response object: http://www.softwareishard.com/blog/har-12-spec/#content
Jul 21, 2011
#4 jgorn...@gmail.com
After doing some more research, wouldn't it be feasible to add the content of the response to a text property in the data emitted when listening to the NetworkAccessManager::handleFinished and performing a readAll() on the reply argument?  Then taking that byte array and converting to a QTString?

Currently, the signal is coming from: http://doc.qt.nokia.com/4.6/qnetworkaccessmanager.html#finished
Then the reply object is: http://doc.qt.nokia.com/4.6/qnetworkreply.html#finished
Where we could call readAll to : http://doc.qt.nokia.com/4.6/qiodevice.html#readAll

Just wanted to check if I am on the right path here.

Jul 22, 2011
Project Member #5 ariya.hi...@gmail.com
I'm not sure readAll is wise there. Doing that definitely leaves WebKit with data at all?
Jul 22, 2011
#6 jgorn...@gmail.com
Are you saying that if you call readAll on the response object, it won't leave that data available for WebKit to read?
Jul 25, 2011
Project Member #7 ariya.hi...@gmail.com
Yes, I am quite sure calling readAll will eat all the received data and leave nothing.
Aug 4, 2011
#8 jgorn...@gmail.com
Off the top of your head, would there then be a way to read a copy of response data?  If you can point me in the right direction, I can start experimenting with it.
Aug 7, 2011
#9 jgorn...@gmail.com
I did a little more research and found out that it might be possible to write a proxy for the QNAM and the NetworkReply.  This way you could capture the data received and still have it accessible after the read has finished.  An example I found pointed me to: http://gitorious.org/qtwebkit/performance/blobs/master/host-tools/mirror/main.cpp

Am I on the right track here?

Thanks!
Aug 7, 2011
Project Member #10 ariya.hi...@gmail.com
Yes, proxying both the network access manager and network reply is the right approach.
Aug 7, 2011
#11 jgorn...@gmail.com
I'm on my way to figuring this out but have run into a snag.  Keep in mind I'm not that familiar with C++ but after adding in the reply proxy, I get a compile error of: 

Undefined symbols:
  "vtable for NetworkReplyProxy", referenced from:
      NetworkReplyProxy::NetworkReplyProxy(QObject*, QNetworkReply*)in networkaccessmanager.o

Here is my latest gist of the NetworkAccessManager:
https://gist.github.com/1131280

Any help would be greatly appreciated!
Aug 14, 2011
#12 jgorn...@gmail.com
Pull request: https://github.com/ariya/phantomjs/pull/124

We'll definitely need to go through a code review on this one :)
Aug 21, 2011
Project Member #13 ariya.hi...@gmail.com
See https://github.com/ariya/phantomjs/commit/578aa6c2.

We need an option to enable and disable (by default) this data gathering (since it introduces some performance penalty).
Status: Accepted
Labels: Milestone-Release1.3
Sep 14, 2011
Project Member #14 ariya.hi...@gmail.com
Hmm, some ownership issue and/or race condition in the proxy might provoke a crash.
Sep 15, 2011
#15 jgorn...@gmail.com
Ariya, I'm sorry, but I'm not following.  Can you elaborate a little more?

Also, when I get some time, I can add in a patch to disable the use of the proxy.
Sep 15, 2011
Project Member #16 ariya.hi...@gmail.com
I fixed the potential crash in https://github.com/ariya/phantomjs/commit/37404ba1.
Sep 15, 2011
#17 jgorn...@gmail.com
Awesome!
Sep 15, 2011
Project Member #18 ariya.hi...@gmail.com
Closing this one as it is implemented already.

For the API/settings improvement, head to  issue 236 .
Status: Fixed
Sep 18, 2011
Project Member #19 ariya.hi...@gmail.com
Reopened. Likely needs to be rescheduled soon.

It causes regression, see  issue 238 .
Status: Accepted
Sep 18, 2011
Project Member #20 ariya.hi...@gmail.com
(No comment was entered for this change.)
Blocking: 238
Sep 19, 2011
Project Member #21 ariya.hi...@gmail.com
I can't find a quick way to resolve this. Probably I will revert it for the time being.
Sep 20, 2011
Project Member #22 ariya.hi...@gmail.com
Unfortunately I have to revert this, see https://github.com/ariya/phantomjs/commit/eb255817

We need to have a better working solution for 1.4.
Labels: -Milestone-Release1.3 Milestone-Release1.4
Oct 3, 2011
#23 jgorn...@gmail.com
Ariya, do you have any other information on what needs to be added to the proxy so we can get this back in?  I'd like to be able to help and get this resolved.
Oct 4, 2011
Project Member #24 ariya.hi...@gmail.com
I have not investigated further. Basically the fix should not regress netlog and netsniff examples (easy to test). See also  issue 238 .
Oct 4, 2011
#25 jgorn...@gmail.com
Sounds good.  I will try to look into this and see if I can find a solution.  I will also look into adding the flag to disable the proxy for performance reasons.
Dec 18, 2011
Project Member #26 ariya.hi...@gmail.com
Not enough time to resolve for 1.4. Postpone to 1.5.
Labels: -Milestone-Release1.4 Milestone-Release1.5
Mar 8, 2012
Project Member #27 ariya.hi...@gmail.com
No activity, reschedule.
Labels: -Milestone-Release1.5 Milestone-FutureRelease
Mar 12, 2012
Project Member #28 ariya.hi...@gmail.com
 Issue 422  has been merged into this issue.
Apr 14, 2012
Project Member #29 ariya.hi...@gmail.com
 Issue 501  has been merged into this issue.
May 2, 2012
#30 webmas...@webmaster.ms
I think, the embedded phantomjs server can act as a proxy which intersects and saves the content of the dowloading files and the sending POST.
The only problem is its unability to parse queries like "GET http://www.google.com/ HTTP/1.1".
The server responses with "Error 400: Bad Request, Cannot parse HTTP request: [GET]".
It seems that fixing the server would be an easier and faster solution for the people wanting to access the bodies of requests and responses.
May 2, 2012
#31 jgorn...@gmail.com
I do have a commit that adds the ability to read the responses @ https://github.com/jgornick/phantomjs/commits/reply-proxy-issue-158. However there are 2 other issues I know of that need to be fixed in order to get this accepted:

1.  Issue #238  - REGRESSION: netsniff.js does not produce HAR dump
2.  Issue #236  - Settings to control network capture

I'm pretty slammed right now so if someone else is interested in helping out, I'd appreciate it :)
May 8, 2012
#32 Pavel.evst
I made simple solution for that, https://github.com/ariya/phantomjs/pull/246
In my fork page.onResourceRequested accept hash with postData if its POST-request

Why you need proxy for that? I think its quite simple
May 12, 2012
#33 webmas...@webmaster.ms
jgornik, there is one more issue with NetworkProxy approach:
synchronious XMLHttpRequest doesnt work (but asynchronious still)

testcase: https://gist.github.com/2664974
May 14, 2012
Project Member #34 james.m....@gmail.com
I would want the resource response text to be not just readable but writable as well (so I can modify a particular resource's response body for instrumentation purposes). See https://code.google.com/p/phantomjs/issues/detail?id=539 for more info.
Feb 5, 2013
#35 bsham...@gmail.com
For what it's worth, I have a workaround that I've been using. I followed phantomjs's code into the QTNetworking core and found where the cache is being saved and how the names are hashed. I replicated that using phantomjs and then pulled the content from the QT's cache file. I only tested this with images since that's all I need it for, there will definitely be cases where text documents don't work with it, but for what it's worth, here is my code: https://gist.github.com/bshamric/4717583
Feb 6, 2013
Project Member #36 james.m....@gmail.com
Interesting idea, thanks for sharing. Unfortunately this approach also breaks down when the responding server indicates (via HTTP response headers) that the requested resources cannot be cached.
Mar 15, 2013
Project Member #37 james.m....@gmail.com
Closing. This issue has been moved to GitHub: https://github.com/ariya/phantomjs/issues/10158
Status: Migrated
Sign in to add a comment

Powered by Google Project Hosting