I'm a little worried by this.  I haven't seen it on any other platform.  Can you tell
me the compiler version?   Issue 51  is related to a compiler bug as well.  I am not
familiar with the arm/xscale platform.  What is xscale?

XScale is an ARM processor (http://en.wikipedia.org/wiki/Xscale).

The compiler I used:
Configured with: /work/crosstool-0.27/build/arm-linux/gcc-3.4.1-glibc-2.3.2/gcc-
3.4.1/configure --target=arm-linux --host=i686-host_pc-linux-gnu --prefix=/usr/local/
arm/3.4.1 --with-headers=/usr/local/arm/3.4.1/arm-linux/include --with-local-prefix=/
usr/local/arm/3.4.1/arm-linux --disable-nls --enable-threads=posix --enable-
symvers=gnu --enable-__cxa_atexit --enable-languages=c,c++ --enable-shared --enable-
c99 --enable-long-long
Thread model: posix
gcc version 3.4.1

What the official debian binaries were compiled with I don't know. But at least the 
error is consistent with everything I've tried so far. I even tried with the 
compiler option I use for all my other ARM programs:
-Wall -fomit-frame-pointer -finline-functions -funroll-loops -falign-functions=2 -
falign-loops=2 -falign-jumps=2 -mcpu=xscale -mtune=xscale

no difference...

Compared to the expected output everything seems to be shifted by two bytes. Is 
there anything I can do to provide you with further clues for this?

Can you try compiling without any optimization, and re-run the test?

Also, -fomit-frame-pointer has caused problems in  issue 51 .

I do a lot of development on gcc 3.4, so this seems like a real issue.

xdelta3 compiled with no optimizations and xdelta3-debug produce the same error as 
well.

If you could, I'm interested in the output of "xdelta3 config".  I'm not sure what
I'm looking for, but I'm interested.

I believe this and  issue 51  are caused by the same underlying issue, uninitialized
data. I will keep you posted.

Ok, here's the "xdelta3 config" output for version 3.0s:

Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8

Would you mind testing with SVN 207? I believe the problem is solved, there was a bug
introduced in 3.0s.

The test also fails with versions k and q as stated above.
Same thing unfortunately:

root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0s, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t...................................................................
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 C14/
6@12 failed: wrong result: XD3_INTERNAL

Can you try a few more tests?  The xdelta3-32 binary compiles with a 32-bit file
offset, and its test might shed some light on this.  I've released 3.0t now, and this
is my highest priority bug report.

Ok, I can do this on monday at the earliest.

Run with svn rev 210

root@familiar:/tmp# ./xdelta3-32 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=0
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=4
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t...................................................................
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 C14/
6@12 failed: wrong result: XD3_INTERNAL


root@familiar:/tmp# ./xdelta3 test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t...................................................................
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 C14/
6@12 failed: wrong result: XD3_INTERNAL

Thanks. I will go study the code and see if I can guess at a cause.

I have a theory.  ARM processors do not support unaligned access the way x86 does. 
There is an unaligned access which may have contributed to this problem.  Could you
please run "xdelta3-debug test" and see if it produces the same errors?  I am hoping
it will crash with an assertion prior to the test failure. 

SVN 211 has a few cleanups that should not make a difference, but it would be best if
you could test with the latest version.

I will go about cleaning up the unaligned accesses using an memcpy for platforms that
do not support efficient unaligned access.

It produces the same output as the other tests. But then again it doesn't seem to 
violate any bounds. The string match result is offset by two bytes but seems ok 
otherwise.

root@familiar:/tmp# ./xdelta3-debug test
Xdelta version 3.0t, Copyright (C) 2007, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=1
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=1
XD3_ENCODER=1
XD3_POSIX=0
XD3_STDIO=1
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t...................................................................
 success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing address_cache (XD3_ALT_CODE_TABLE)... success
xdelta3: testing string_matching...test 1: expected R0/10 C12/8@10: got R0/10 C14/
6@12 failed: wrong result: XD3_INTERNAL

It is finding a correct match, but only when it starts at 12, not at 10.  I believe
this is because 12 is word-aligned and 10 is not.  Let me see if I can add some
debugging (to diagnose further) and/or correct the problem (the solution involves
memcpy).

SVN 212 may solve the problem.

To be EXTRA sure, you could add -DUNALIGNED_OK=0 to CFLAGS,
but I've set things so that only __i386__ and similar defs
will set that variable by default.

Good job, that fixed it :)

Another failing test:

xdelta3: testing no_output...command was ./xdelta3 -q -f -e /tmp/xdtest.target.3 /
tmp/xdtest.noperm.3 failed: command should have not succeeded: XD3_INTERNAL

Do you think it's related to that one or should I open a new ticket?

We can keep this ticket open.

The test that fails now is related to file permissions.  Are you running as root,
possibly?  Are there file permissions?

(No comment was entered for this change.)

Actually I want to keep this issue open for another reason.

Now that we know the string_matching problem is fixed, I'd like to test a more
efficient solution.  Stay tuned.

Argh--I just cleaned up the UNALIGNED_OK=0 case while sitting at the laundromat and
spill bleach on my computer (bad) on the way home!

Aw, shit. Hope it still works...

root is actually the only user on the device... I created a test user now and with 
that it works alright. Next fail:

xdelta3: testing recode_command...xdelta3: printhdr command: /tmp/xdelta3 printhdr -
f /tmp/xdtest.copy.17 /tmp/xdtest.recon2.17 failed: printhdr cmd failed: XD3_INTERNAL

There's a sizzling sound from the bleach on the battery contacts.  Not good.

Oh well, it was old.  Now on my OS X 10.3 machine, SVN 220 has some changes for the
UNALIGNED_OK=0 case, test should still pass string_matching.

You could perform a test by hand, since it seems like "printhdr" isn't working, and
you might get more info.

e.g.,

./xdelta3-debug -s xdelta3.h xdelta3.c OUT
./xdelta3-debug printhdr OUT  # should print to stdout
./xdelta3-debug printhdr OUT PRINT # should print to PRINT

This might shed some light...

(No comment was entered for this change.)

The printhdr commands segfault... I've attached the output of strace.

strace-xd3.txt 5.5 KB   Download

I can't get much from the strace output.  If you were to build "xdelta3-debug" and
run it under gdb, and show me the stacktrace, that would be best:

# gdb ./xdelta3-debug
(gdb) run printhdr INPUT
....
CRASH!

That doesn't really help... I've never gotten gdb to work correctly on the device. 
All the output I get from the stacktrace is:

#0 0x00000003 in ?? ()

(No comment was entered for this change.)

I'd love to know if this is still a problem in the 3.0u release which I just made.

New test results for the 3.0u release:

# ./xdelta3 test
[...]
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits set: 
XD3_INVALID_INPUT
xdelta3: printhdr command: xdelta3 printhdr -f /tmp/xdtest.copy.17 /tmp/
xdtest.recon2.17
 failed: printhdr cmd failed: XD3_INTERNAL

# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT

Thanks, this should help track it down.

Can you upload the file "OUT" from the last printhdr command?  This should help
diagnose whether it's an encoder-side or decoder-side issue.  I've been looking for
issues in the code but nothing sticks out.

Would you try with SVN 277?  I think a signed-enum with sign-extension issue may have
cropped up.

See related report in  issue 47 .

Ok, here's the OUT file. SVN 277 doesn't change the errors I get.

OUT 16.0 KB   Download

The file looks good, I was able to print it and reconstruct a copy of xdelta3.h using
OUT.  I will try to instrument the code with more debugging info.  Thanks for helping.

How about now?

Same thing:

# su -c'./xdelta3 test' nobody
Xdelta version 3.0w, Copyright (C) 2007, 2008, Joshua MacDonald
Xdelta comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "COPYING" for details.
EXTERNAL_COMPRESSION=1
GENERIC_ENCODE_TABLES=0
GENERIC_ENCODE_TABLES_COMPUTE=0
REGRESSION_TEST=1
SECONDARY_DJW=1
SECONDARY_FGK=1
UNALIGNED_OK=0
VCDIFF_TOOLS=1
XD3_ALLOCSIZE=16384
XD3_DEBUG=0
XD3_ENCODER=1
XD3_POSIX=1
XD3_STDIO=0
XD3_WIN32=0
XD3_USE_LARGEFILE64=1
XD3_DEFAULT_LEVEL=3
XD3_DEFAULT_IOPT_SIZE=32768
XD3_DEFAULT_SPREVSZ=262144
XD3_DEFAULT_SRCWINSZ=67108864
XD3_DEFAULT_WINSIZE=8388608
XD3_HARDMAXWINSIZE=16777216
sizeof(void*)=4
sizeof(int)=4
sizeof(uint32_t)=4
sizeof(uint64_t)=8
sizeof(usize_t)=4
sizeof(xoff_t)=8
xdelta3: testing random_numbers... success
xdelta3: testing decode_integer_end_of_input... success
xdelta3: testing decode_integer_overflow... success
xdelta3: testing encode_decode_uint32_t..................................... success
xdelta3: testing 
encode_decode_uint64_t................................................................... 
success
xdelta3: testing usize_t_overflow... success
xdelta3: testing forward_match... success
xdelta3: testing address_cache... success
xdelta3: testing string_matching... success
xdelta3: testing choose_instruction... success
xdelta3: testing identical_behavior... success
xdelta3: testing in_memory... success
xdelta3: testing iopt_flush_instructions... success
xdelta3: testing source_cksum_offset... success
xdelta3: testing decompress_single_bit_error.... success
xdelta3: testing decompress_single_bit_error (XD3_ADLER32).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_FGK).... success
xdelta3: testing decompress_single_bit_error (XD3_SEC_DJW).... success
xdelta3: testing force_behavior... success
xdelta3: testing stdout_behavior... success
xdelta3: testing no_output... success
xdelta3: testing command_line_arguments................ success
xdelta3: testing source_decompression... success
xdelta3: testing externally_compressed_io......sh: bzip2: command not found
bzip2=0...sh: compress: command not found
compress=0 success
xdelta3: testing recode_command...xdelta3: unrecognized window indicator bits set: 
XD3_INVALID_INPUT
xdelta3: printhdr command: ./xdelta3 printhdr -f /tmp/xdtest.copy.18 /tmp/
xdtest.recon2.18
 failed: printhdr cmd failed: XD3_INTERNAL


# ./xdelta3 -s xdelta3.c xdelta3.h OUT
# ./xdelta3 printhdr OUT
xdelta3: unrecognized window indicator bits set: XD3_INVALID_INPUT

OUT 15.9 KB   Download