New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
performance: writing to file using IOSink(add) is very slow #17951
Comments
Set owner to @skabet. |
Tatumizer, can you attach the Java program for completeness? That would make it easier to understand the differences and why we appear to be that much slower. I have a CL that fixes some of this, but not a factor of 8x. Thanks. |
This comment was originally written by @tatumizer I will send in a couple of hours, it's on my work comp. |
This comment was originally written by @tatumizer Hi Anders, The ratio of results is more or less the same : |
This comment was originally written by @tatumizer Maybe it's a windows-only phenomenon? |
This comment was originally written by @tatumizer for IO requests, all the "action" occurs in the driver inside OS. Library just has to call correct OS function, and it should be more or less the same for all languages. Because "add" in IOSink is anynchronous, there's additional optimization: library doesn't wait - it stores the buffer, and when IO interrupt allows writing, it writes. (At least, in principle it should be implemented like this). It can be faster, not slower, than java's OutputStream. "asynchronous" functionality exists in java, too - in java.nio package. |
tatumizer, I was able to run the program, with the following results (with my fix that was just landed): File does not exist: Java(Sync): 321ms File exists: Java(Sync): 1049ms So, it's clear that async writing is slower. This is due to two things. Doing the copy for the writing isolate, and the extra delay in sending messages between isolates. My fix helper with the former. But when comparing with sync code, Dart is actually a little bit faster than Java. Also, there is a HUGE difference in if the file exists or not. Be sure you are testing the same. |
This comment was originally written by @tatumizer The result I got from your program in dart are |
This comment was originally written by @tatumizer Anders, sorry, prev. post was directed to mezoni. |
This comment was originally written by @tatumizer Anders, |
Interesting, this is on Linux. I'll try out on Windows, when I get a chance. |
This comment was originally written by @tatumizer What affects speed of writing is: fragmentation of disk. And of course the |
This comment was originally written by @tatumizer Anders, Mezoni: my apologies. You are a good guy. Just a bit rude. You have to learn, if not HW, then good manners. |
This comment was originally written by @tatumizer Mezoni: if block size is not multiple of 65536 (e.g. 8192), then java works 1.5-2 times faster on my comp on bigger blocks. It depends on too many factors (fragmentation, OS, block size) - no way to compare "objectively". |
This comment was originally written by @tatumizer Anders: the mystery about speed of writes on Windows can be resolved by this: It caches writes by default! |
Ah, yes, that's quite common. I'm sure it happens on Linux as well. The extra cost probably comes from actual HD activity, where we start out by |
This comment was originally written by @tatumizer Anders: turns out, the issue is more complicated. I just tested writing to "nul" device on Windows. It shouldn't depend on any properties of hardware, of whether the file is new or old - there's no file. Data is just discarded. |
tatumizer, can you clarify what you are comparing. Are you comparing IOSink with synchronous Java? It's important that we fully understand what async writing means. Doing async writing will do the exact same as synchronous, except it'll copy the data to another isolate (thread) and let that isolate perform the action. Once done, it'll notify the isolate that issued the write. It's very obvious that writing has a higher overhead when async - that can not change (though we are trying to minimize the overhead). However, what it allows is to not block the isolate issuing the write. This is very important for programs where we have many simultaneous operations, e.g. a HTTP server. If the results you have (73 vs 1760) is gather from the two programs show in this isolate, we are comparing oranges and apples. |
This issue was originally filed by @tatumizer
Writing to file is almost 10 times slower than in java
var sw = new Stopwatch()..start();
IOSink out=new File("c:/temp/foo.txt").openWrite();
out.done.then((v)=>print("total time: ${sw.elapsedMilliseconds}"));
for (int i=0; i<4000; i++) {
out.add(new Uint8List(65536));
}
out.close();
Takes 1813 ms in dart; same in java completes in 212 ms
The above code writes 256MB. I tested how it might be affected by size - reduced iterations from 4000 to 1000
Took 484ms in dart, 55 ms in java. Same ratio.
The text was updated successfully, but these errors were encountered: