Any suggestion to a utility method for comparing two streams?

This might be a good start:
http://stackoverflow.com/questions/4245863/fast-way-to-compare-inputstreams

Thanks. I will try NIO based implementation.

The NIO implementation is not a very good one, as it requires moving data back and forth from jvm to native memory multiple times. It would be better if you used a standard ByteBuffer, rather than a DirectByteBuffer (as that avoids the crossing jvm/native memory).
You could also use IOUtils from commons-io.
http://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/IOUtils.html#contentEquals(java.io.InputStream,%20java.io.InputStream)

I think it's the other way around - using direct ByteBuffers allows the OS to perform the I/O directly into the same memory space that is then accessed by the Java code, whereas with non-direct buffers the I/O is first done in some other native buffer and then copied into a Java array. http://stackoverflow.com/questions/5670862/bytebuffer-allocate-vs-bytebuffer-allocatedirect is the first link I found on this, and confirms it. You can also make the buffer a bit larger than 1K - probably not much difference in performance, but still most systems nowadays read from disk in at least 4K blocks anyway, so might as well pass it through like that.

In any case, the performance difference between the two buffer types in this case is likely negligible, and both should be several orders of magnitude better than the current state of affairs.

That is only true if you are reading directly from a
fileinputstram/filechannel. If you are reading from nearly any other source
(such as a ZipInputStream used to read resources in jars) all of the bytes
are pulled into the jvm then pushed back into native then back into jvm 1
byte at a time to do equality comparison.
On Jul 1, 2013 11:35 AM, "amichair" notifications@github.com wrote:

I think it's the other way around - using direct ByteBuffers allows the OS
to perform the I/O directly into the same memory space that is then
accessed by the Java code, whereas with non-direct buffers the I/O is first
done in some other native buffer and then copied into a Java array.
http://stackoverflow.com/questions/5670862/bytebuffer-allocate-vs-bytebuffer-allocatedirectis the first link I found on this, and confirms it. You can also make the
buffer a bit larger than 1K - probably not much difference in performance,
but still most systems nowadays read from disk in at least 4K blocks
anyway, so might as well pass it through like that.

In any case, the performance difference between the two buffer types in
this case is likely negligible, and both should be several orders of
magnitude better than the current state of affairs.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-20293570
.

Basically there are 2 really good uses for direct byte buffers and several
edge uses.

1. Transferring data between files, when none of the data is actually
   looked at in jvm.
2. Interacting with jni.

The edge uses are usually around addressing very large chunks of memory.

If all of the data is examined within the jvm (and there is no later jni
involved) it is better to read directly into a byte[].
On Jul 1, 2013 6:07 PM, "Brett Okken" brett.okken.os@gmail.com wrote:

That is only true if you are reading directly from a
fileinputstram/filechannel. If you are reading from nearly any other source
(such as a ZipInputStream used to read resources in jars) all of the bytes
are pulled into the jvm then pushed back into native then back into jvm 1
byte at a time to do equality comparison.
On Jul 1, 2013 11:35 AM, "amichair" notifications@github.com wrote:

I think it's the other way around - using direct ByteBuffers allows the
OS to perform the I/O directly into the same memory space that is then
accessed by the Java code, whereas with non-direct buffers the I/O is first
done in some other native buffer and then copied into a Java array.
http://stackoverflow.com/questions/5670862/bytebuffer-allocate-vs-bytebuffer-allocatedirectis the first link I found on this, and confirms it. You can also make the
buffer a bit larger than 1K - probably not much difference in performance,
but still most systems nowadays read from disk in at least 4K blocks
anyway, so might as well pass it through like that.

In any case, the performance difference between the two buffer types in
this case is likely negligible, and both should be several orders of
magnitude better than the current state of affairs.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-20293570
.

I just created a snapshot version that simply compares two InputStreams.
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5.1-SNAPSHOT/

@amichair
Could you test this version in your platform?

@bokken You're right, as in our case one of the streams is not directly read from a file...

@xerial I tried to recreate the issue using 1.0.5.1-SNAPSHOT but it did not occur, so looks like it's solved. Thanks!