Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow performance (on Linux) #59

Closed
GoogleCodeExporter opened this issue Apr 1, 2015 · 14 comments
Closed

Slow performance (on Linux) #59

GoogleCodeExporter opened this issue Apr 1, 2015 · 14 comments

Comments

@GoogleCodeExporter
Copy link

Are you using the CLI, web service or calibre plugin?

calibre plugin

What is the story URL?

http://www.fanfiction.net/s/7102418/1/Hermione-s-Song

What is the output format you used?

epub

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update

Add New

EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

Yes

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?

Performance is incredibly slow.  Cancelled file download after 10 minutes.

What version of the product are you using? On what operating system?

1.2, Windows 7

Please provide any additional information below.

Original issue reported on code.google.com by williamt...@gmail.com on 10 Sep 2013 at 9:11

@GoogleCodeExporter
Copy link
Author

The most recent version has additional delays built in to meet ffnet's 
requirements.  It should take that story around 2 minutes to download, and 
that's what I see when I try it.

If it's taking you 10 minutes, I speculate that ffnet is throttling.  Unless 
you have a significant slow_down_sleep_time set in personal.ini, nothing FFDL 
does should be causing it to take that long.

Original comment by retiefj...@gmail.com on 10 Sep 2013 at 2:27

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Operating as designed and no further response from reporter.

Original comment by retiefj...@gmail.com on 13 Oct 2013 at 3:56

  • Changed state: Invalid
  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

I have the same issue. I've found that on Windows (64-bit Windows 7) downloads 
are reasonably fast; on Ubuntu (several versions, since I've had this issue for 
a long time) downloads are very slow.
https://www.fanfiction.net/s/3934461/ has been downloading for 11 minutes now.

On Linux, no other application shows this behaviour; I have a high-speed (20+ 
Mbits/s) connection. I've disabled ipv6 support in my OS, as that did slow 
everything down (I'm on ipv4).

Original comment by kovacs...@gmail.com on 2 Mar 2014 at 2:15

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

https://www.fanfiction.net/s/3934461/  took 13 minutes, another, 
https://www.fanfiction.net/s/2544950/, took 12.

Original comment by kovacs...@gmail.com on 2 Mar 2014 at 2:19

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

With the sleep times involved these days to keep from triggering the throttling 
at ffnet, those times, while a little long, aren't outrageous.

Downloading both in CentOS6 for me took ~11m and ~7m on Win8.

With some added debug output, it looks like connection times average the same 
within 30ms.  

So, it looks like there is some additional slow down in Linux vs Windows 
somewhere.

I'll see if I can isolate it any better sometime.

Original comment by retiefj...@gmail.com on 7 Mar 2014 at 2:02

  • Changed title: Slow performance (on Linux)
  • Changed state: Accepted
  • Added labels: Priority-Low, Type-Enhancement
  • Removed labels: Priority-Medium, Type-Defect

@GoogleCodeExporter
Copy link
Author

Turns out I had different slow_down_sleep_time settings on my linux and my 
windows.

Now that I have them set the same, I'm getting extremely similar times on linux 
and windows.

I've posted the test version I'm using with additional profile timing debug 
output at:
http://www.mobileread.com/forums/showthread.php?p=2781869#post2781869

If you want to check your slow_down_sleep_time settings and try the posted test 
version, posting the job output here, it might tell us more.

Original comment by retiefj...@gmail.com on 8 Mar 2014 at 2:46

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

I'll give it a try, thanks.

Original comment by kovacs...@gmail.com on 10 Mar 2014 at 6:17

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Well, I'm not sure I did it correctly, but I saw no change in speed, 
https://www.fanfiction.net/s/3934461/ took 13m 49s.

1. downloaded zip from linked forum post;
2. found FanFictionDownloader in ~/.config/calbire/plugins
3. moved it elsewhere (a temp directory)
4. copied downloaded zip to said directory
5. launched calibre from the command-line
6. removed e-book from calibre's menu
7. copied URL to clipboard
8. used FF downloader plugin to download and convert

The console output does not contain any kind of debug info, I think:
kofa@eagle:~$ calibre
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(155):Plugin 
FanFictionDownLoader macmenuhack 
file_path:/home/kofa/.config/calibre/plugins/fanfictiondownloader_macmenuhack.tx
t
Created new window in existing browser session.
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(725):FanFicti
onDownLoader v1.8.13
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(760):url:http
s://www.fanfiction.net/s/3934461/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters
.adapter_fanfictionnet(90):URL: https://www.fanfiction.net/s/3934461/1/

===========
 connection time 15.75
===========
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(933):from 
URL(https://www.fanfiction.net/s/3934461/1/)
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(1086):title:V
ictus per Reproba Monumentum
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(1087):outfile
:/tmp/calibre_1.26.0_tmp_w_CUTi/fanfictiondownloader_tCZODR/Victus per Reproba 
Monumentum-firefly5151-2e9wda.mobi
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(1176):add/upd
ate Victus per Reproba Monumentum https://www.fanfiction.net/s/3934461/1/


Original comment by kovacs...@gmail.com on 10 Mar 2014 at 7:55

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

After running a download, click where it says 'Jobs' in the lower right corner 
of calibre.  That brings a list of past & current jobs.  Double click the 
finished Download Fanfiction job to see the logged background output for the 
download.

Original comment by retiefj...@gmail.com on 10 Mar 2014 at 9:07

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

[deleted comment]

@GoogleCodeExporter
Copy link
Author

Done (log attached).
FYI, using wget (I got the URLs from the calibre log):
FINISHED --2014-03-10 22:42:43--
Total wall clock time: 38s
Downloaded: 53 files, 3,5M in 12s (293 KB/s)
There was a ~5sec 'Resolving www.fanfiction.net (www.fanfiction.net)... 
173.205.184.6' pause at the beginning of the wget run, the rest was fast.

Original comment by kovacs...@gmail.com on 10 Mar 2014 at 9:45

  • Added labels: ****
  • Removed labels: ****

Attachments:

@GoogleCodeExporter
Copy link
Author

Well, the lookup pause was probably longer, not 5s, as download time was 12s, 
wall clock time was 38s. I re-ran it, cancelled when it started to download, 
and the lookup took 15s.
DNS resolution slowness seems to be ipv6-related, and not calibre-specific. 
wget -4 (forced ipv4) and nslookup / dig take no time to resolve names.

Let's put this on hold until I sort wget out.

Original comment by kovacs...@gmail.com on 10 Mar 2014 at 9:52

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

FFDL is never going to be as fast d/l'ing from ffnet as wget because there's ~9 
seconds of sleeps added.

But the 'connection time' lines are truly that--just the connection times.  On 
my systems, both windows and linux, those averaged around 330ms.

Since yours are in the neighborhood of 5s longer and you see the DNS takes 
that, I agree that it's likely a DNS issue for you.

Why it would be resolving on every hit instead of caching is probably a related 
question.

If you want to prove whether it's specific to calibre or common to python, I've 
attached an extremely basic python url fetch program.  Just give it a file 
containing only a list of urls.

Original comment by retiefj...@gmail.com on 10 Mar 2014 at 10:57

  • Added labels: ****
  • Removed labels: ****

Attachments:

@GoogleCodeExporter
Copy link
Author

Still no response from original poster.

Second poster's problem appears to be a DNS unique to them.

Closing issue.

Original comment by retiefj...@gmail.com on 16 Mar 2014 at 8:37

  • Changed state: WontFix
  • Added labels: ****
  • Removed labels: ****

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant