Attached file is not decoding correctly.

With r856 of trunk and openjpeg-1.5 branch :

[16:58:48 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin] ./j2k_to_image -i ~/Desktop/p0_07.j2k -o test.bmp

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number of tile-parts (0) [INFO] tile 1 of 256 [ERROR] Expected EPH marker [ERROR] tcd_decode: incomplete bistream [INFO] - tiers-1 took 0.011788 s [INFO] - dwt took 0.000748 s [INFO] - tile decoded in 0.013870 s ERROR -> j2k_to_image: failed to decode image! [16:58:52 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin]

With tag version.1.4 :

[16:58:16 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin] ./j2k_to_image -i ~/Desktop/p0_07.j2k -o test.bmp

[INFO] tile 1 of 256 [ERROR] Expected EPH marker [ERROR] tcd_decode: incomplete bistream [INFO] - tiers-1 took 0.011339 s [INFO] - dwt took 0.001735 s [INFO] - tile decoded in 0.014901 s ERROR -> j2k_to_image: failed to decode image! [16:58:20 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin]

Any idea on how to fix this is welcome.

This issue can be followed on the openjpeg dashboard : http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&date=2011-08-11

The name of this test is ETS-C1P0-p0_07.j2k-decode and it could be found for trunk and openjpeg 1.5 branch.

Mickaël

I have used OpenJPEG-1.5.0-revision-863.

Problem 1:

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number of tile-parts (0)

This warning tells that '0' is greater than '0'. But the test condition is:

if (partno >= numparts)

Problem 2:

[ERROR] Expected EPH marker

I have written a trace line into 't2.c':

t2.c:494:hd(0xff,0x92) <== EPH nr 73 t2.c:494:hd(0x06,0x05) <== expected nr 74 [ERROR] Expected EPH marker

I have a simple J2K reader that shows:

Observation 1:

NAME(p0_07.j2k) LENG(7885684)

ENTER read_jp2c [0]marker(0xff4f) soc len(0) [2]marker(0xff51) siz len(47) capabilities(1) x(0 : 2048) y(0 : 2048) xt(0 : 128) yt(0 : 128) nr_components(3) component[0]signed(1) prec(12) hsep(1) vsep(1) component[1]signed(1) prec(12) hsep(1) vsep(1) component[2]signed(1) prec(12) hsep(1) vsep(1) [51]marker(0xff52) cod len(12) [65]marker(0xff5c) qcd len(13) [80]marker(0xff64) com len(16) R[1](General use (ISO 8859-1 (latin-1) values)) T(Kakadu-3.0.7) [98]marker(0xff90) sot tile_nr(0) Psot(9951) TPsot(0) TNsot(0) len(10)

(...)

[7836303]marker(0xff90) sot tile_nr(255) Psot(34820) TPsot(0) TNsot(1) len(10)

and after this tile nr 255:

[7871123]marker(0xff90) sot tile_nr(0) Psot(14559) TPsot(1) TNsot(2) len(10)

Observation 2:

[10049]marker(0xff90) sot tile_nr(1) Psot(23681) TPsot(0) TNsot(1) len(10) [10061]marker(0xff58) plt len(134) [10197]marker(0xff93) sod len(0) [10199]marker(0xff91) sop data(4) [10209]marker(0xff92) eph skip_len(137) <== EPH nr 73 [10348]marker(0xff91) sop data(3) [10357]marker(0xff92) eph skip_len(172) <== EPH nr 74 (...)

winfried

Many thanks Winfried.

Problem 1 is actually a separate problem I should fix aside from this issue. Thanks for pointing this.

Problem 2 seems to come from the fact that the first tile is divided into 2 tile-parts, one located at the very beginning of the code-stream and the second one at the very end (with all the 255 remaining tiles in between).

Attached is a useful file showing the structure of all the code-stream.

Still investigating ...

Antonin

[98]marker(0xff90) sot tile_nr(0) Psot(9951) TPsot(0) TNsot(0) (...) [10049]marker(0xff90) sot tile_nr(1) Psot(23681) TPsot(0) TNsot(1)

98 + 9951 => 10049

Is there no possibility to test against the end of the tile?

winfried

tileno(0) is split into two parts.

The first part has a totlen of 9951. The second part has a totlen of 14559.

j2k.c:1529:j2k_read_eoc tileno(0) tile_len(24319) (...)

t2.c:724:ENTER t2_decode_packets tileno(0) tilelen(24319)

(...)

t2.c:340:ENTER t2_decode_packet len(14942) t2.c:365:SOP(0xff,0x91) t2.c:445:bandno(0) cblkno(0) t2.c:445:bandno(1) cblkno(0) t2.c:445:bandno(2) cblkno(0) t2.c:506:bio_numbytes(7) t2.c:520:EPH(0xff,0x92) t2.c:743:t2_decode_packet returns(448) len(24319) curlen(9825) t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14494) t2.c:365:SOP(0xff,0x91) t2.c:445:bandno(0) cblkno(0) t2.c:506:bio_numbytes(1) t2.c:520:EPH(0xff,0x92) t2.c:743:t2_decode_packet returns(9) len(24319) curlen(9834) t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485) t2.c:365:SOP(0xff,0x91) t2.c:445:bandno(0) cblkno(0) t2.c:506:bio_numbytes(2) t2.c:515:ERR(0x06,0x05) [ERROR] Expected EPH marker

t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834)

tileno(0) seems to use the second len value for the first part. So the pointer runs into tileno(1).

Apart from the different length values: is splitting is allowed?

If yes: could there be a collector?

add to tileno(0) add to tileno(1) (...) add to tileno(255) add to tileno(0)

winfried

Comment deleted

No, I am wrong. The contents of both tile parts are in one buffer. The pointer does not run into tileno(1). The len of the first part is 9825 byte.

t2.c:520:[13]EPH(0xff,0x92) t2.c:743:t2_decode_packet returns(448) tilelen(24319) curlen(9825) t2.c:733:tileno(0) pino(0) poc.prg(0)

------------ start of second part of tileno(0) ------------ t2.c:340:ENTER t2_decode_packet len(14494)

t2.c:365:[0]SOP(0xff,0x91) t2.c:445:bandno(0) cblkno(0) t2.c:506:bio_numbytes(1)

t2.c:520:[7]EPH(0xff,0x92) t2.c:743:t2_decode_packet returns(9) tilelen(24319) curlen(9834) t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485)

t2.c:365:[0]SOP(0xff,0x91) t2.c:445:bandno(0) cblkno(0) t2.c:506:bio_numbytes(2)

t2.c:515:ERR(0x06,0x05) +2(0xd5,0x80) +4(0xff,0x92) +6(0xe2,0x63) [ERROR] Expected EPH marker t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834) [ERROR] tcd_decode: incomplete bistream

ERROR -> j2k_to_image: failed to decode image!

winfried

In 'j2k_read_sod()' I have sent some values of part2 of tileno(0) to stderr:

tileno(0) OLD data(0x812a060) len(9825) NEW len(14494)

The new data:

    [0](0xff,0x91)  sop(0xff,0x91)
    [1](0x00,0x04)  len(4) + data(1)
    [2](0x00,0x48)  seq_nr(72)(0x00,0x48)

    [3](0x80,0xff)  eph(0xff,0x92), data(0)

    [4](0x92,0xff)  sop(0xff,0x91)
    [5](0x91,0x00)  len(4) + data(6)
    [6](0x04,0x00)  seq_nr(73)(0x00,0x49)
    [7](0x49,0xc1)
    [8](0xdd,0x06)  ERR((0x06,0x05)) : bio_numbytes(2)
    [9](0x05,0xd5)

    [10](0x80,0xff) eph((0xff,0x92), data(108)

(...)

winfried

seq 70 and 71, the last two sequences in part 1, have the 'included' flag set; their sum of all 'segs[segno].newlen' is correct (the same holds for all other before).

seq 72, the first sequence in part 2, does not have the 'included' flag set.

seq 73 has the 'included' flag set. And fails. 'segs[segno].newlen' is 7, 'bio_numbytes()' is 2. The restart begins at the third byte within the sop: '0x06'; instead of at the first byte after the sop: '0xff'.

winfried

p0_07.j2k has two different resno1(i.e. REpoc) values:

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(0) new_nr(1) [0]resno0(0) compno0(0) layno1(9) resno1(3) compno1(3) prg(0)

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(1) new_nr(2) [1]resno0(0) compno0(0) layno1(9) resno1(8) compno1(3) prg(0)

The first part ends with seqno(71). The second part starts with seqno(72).

t2.c:720:TRACE ENTER t2_decode_packets tileno(0) max_pino(1)

t2.c:339:TRACE ENTER t2_decode_packet resno(0) compno(0) layno(0) precno(0) t2.c:375:SOP(0xff,0x91) len(4) seqno(0)

(...)

t2.c:339:TRACE ENTER t2_decode_packet resno(2) compno(2) layno(7) precno(0) t2.c:375:SOP(0xff,0x91) len(4) seqno(71)

t2.c:339:TRACE ENTER t2_decode_packet resno(0) compno(0) layno(8) precno(0) t2.c:375:SOP(0xff,0x91) len(4) seqno(72)

t2.c:339:TRACE ENTER t2_decode_packet resno(0) compno(1) layno(8) precno(0) t2.c:375:SOP(0xff,0x91) len(4) seqno(73) [ERROR] Expected EPH marker [ERROR] tcd_decode: incomplete bistream ERROR -> j2k_to_image: failed to decode image!

winfried

I have the same problem with trunk and V2 style (rev961): http://my.cdash.org/testDetails.php?test=7655496&build=242949

Mickaël

Hi,

Dashboard (rev998 + winfried heap corruption fix) provide some observations about this bug: all the platform and v1.5 failed to decode this file: http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&date=2011-10-12 because EPH marker is not detected as attended.

with trunk, windows platform decode the file (without eph marker error) and the output image is equal to the conformance image: http://my.cdash.org/testDetails.php?test=7684271&build=246254 and http://my.cdash.org/testDetails.php?test=7684272&build=246254

On the other hand, the macOS and ubuntu decode the file with the EPH error and the comparison between output image and reference image failed: http://my.cdash.org/testDetails.php?test=7664016&build=246236

Any idea, why these differences ?

Mickaël

Looks like a struct is not being initialized:

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number of tile-parts (0)

[INFO] tile 1 of 256 ==7949== Invalid read of size 2 ==7949== at 0x4E3DA57: pi_next_lrcp (pi.c:109) ==7949== by 0x4E413A2: pi_next (pi.c:713) ==7949== by 0x4E49070: t2_decode_packets (t2.c:706) ==7949== by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367) ==7949== by 0x4E36662: j2k_read_eoc (j2k.c:1566) ==7949== by 0x4E36CF5: j2k_decode (j2k.c:1877) ==7949== by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163) ==7949== by 0x4E3D2D7: opj_decode (openjpeg.c:156) ==7949== by 0x404458: main (j2k_to_image.c:631) ==7949== Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd ==7949== at 0x4C2380C: calloc (vg_replace_malloc.c:467) ==7949== by 0x4E3FB65: pi_create_decode (pi.c:505) ==7949== by 0x4E48B68: t2_decode_packets (t2.c:697) ==7949== by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367) ==7949== by 0x4E36662: j2k_read_eoc (j2k.c:1566) ==7949== by 0x4E36CF5: j2k_decode (j2k.c:1877) ==7949== by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163) ==7949== by 0x4E3D2D7: opj_decode (openjpeg.c:156) ==7949== by 0x404458: main (j2k_to_image.c:631) ==7949== ==7949== Invalid write of size 2 ==7949== at 0x4E3DA71: pi_next_lrcp (pi.c:110) ==7949== by 0x4E413A2: pi_next (pi.c:713) ==7949== by 0x4E49070: t2_decode_packets (t2.c:706) ==7949== by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367) ==7949== by 0x4E36662: j2k_read_eoc (j2k.c:1566) ==7949== by 0x4E36CF5: j2k_decode (j2k.c:1877) ==7949== by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163) ==7949== by 0x4E3D2D7: opj_decode (openjpeg.c:156) ==7949== by 0x404458: main (j2k_to_image.c:631) ==7949== Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd ==7949== at 0x4C2380C: calloc (vg_replace_malloc.c:467) ==7949== by 0x4E3FB65: pi_create_decode (pi.c:505) ==7949== by 0x4E48B68: t2_decode_packets (t2.c:697) ==7949== by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367) ==7949== by 0x4E36662: j2k_read_eoc (j2k.c:1566) ==7949== by 0x4E36CF5: j2k_decode (j2k.c:1877) ==7949== by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163) ==7949== by 0x4E3D2D7: opj_decode (openjpeg.c:156) ==7949== by 0x404458: main (j2k_to_image.c:631) ==7949==

Have you tried to read this same image using V2? I get expected EPH marker errors there too and the first tile is messed up.

Here is what I see in the svn log:

r1007 | savmickael | 2011-10-12 16:44:46 +0200 (mer. 12 oct. 2011) | 1 ligne

[trunk] WIP: resolve heap corruption with p0_07.j2k (credit to Winfried)

openjpeg trunk was able to cope (= parse) with this file during changes introduce in r961

the following code in j2k.c is bogus:

partno = cio_read(cio, 1);
numparts = cio_read(cio, 1);

if (partno >= numparts) { opj_event_msg(j2k->cinfo, EVT_WARNING, "SOT marker inconsistency in tile %d: tile-part index greater (%d) than number of tile-parts (%d)\n", tileno, partno, numparts); numparts = partno+1; }

The case where numparts is 0 (unknown) is not handled at all.

Seems like this is working today, I can get an output very close to the expected PGX (conformance). Need to add a PSNR test.

PSNR test is already included in the test suite. This is precisely because PSNR is out of tolerated range that compare2ref test is failing

yes this file is not correctly decoded according to the conformance file. I have spend some time to understand why but it is not tricky.

I suspect the issue comes from the weird tile ordering of the file:

[...] 99 : New marker: SOT (Start of tile-part)

Tile       : 0
Length     : 9850
Index      : 0
Tile-Parts : unknown

[...] 7833153 : New marker: SOT (Start of tile-part)

Tile       : 0
Length     : 14519
Index      : 1
Tile-Parts : 2

Confirmed. If I re-organize the file:

$ kdu_transcode -i data/input/conformance/p0_07.j2k -o clean.j2k Porder:T0={0,0,9,8,3,LRCP}

openjpeg is able to decompress the first tile properly (= the whole image).

Note the original image was compressed using something like:

kdu_compress [...] Porder:T0={0,0,9,3,3,LRCP} Porder:T0={0,0,9,8,3,LRCP}

Let's divide the issue into sub issues:

$ kdu_expand -i input/conformance/p0_07.j2k -o 1.tif,2.tif,3.tif -record ref.txt $ kdu_compress -no_info -i 1.tif,2.tif,3.tif -o kdu_ref.tif.j2k Creversible=yes -record test.txt Stiles="{128,128}" Corder=RLCP Clayers=8 Cuse_sop=yes Cuse_eph=yes Clevels=3 Sprofile=PROFILE0 Porder:T0="{0,0,9,3,3,LRCP}" Porder:T0="{0,0,9,8,3,LRCP}" Cycc=no ORGgen_plt=yes

-> this is approximately the same image as p0_07.j2k

Let's start by removing the tile handling:

$ kdu_compress -no_info -i 1.tif,2.tif,3.tif -o kdu_ref.tif.j2k Creversible=yes -record test.txt Corder=RLCP Clayers=8 Cuse_sop=yes Cuse_eph=yes Clevels=3 Sprofile=PROFILE2 Porder:T0="{0,0,9,3,3,LRCP}" Porder:T0="{0,0,9,8,3,LRCP}" Cycc=no ORGgen_plt=yes

Using a smaller image might be easier:

$ kdu_compress -no_info -i lena512.pgm -o lena512.i80.j2k Creversible=yes -record test.txt Corder=RLCP Clayers=8 Cuse_sop=yes Cuse_eph=yes Clevels=3 Sprofile=PROFILE0 Porder:T0="{0,0,9,3,3,LRCP}" Porder:T0="{0,0,9,8,3,LRCP}" Cycc=no ORGgen_plt=yes

Steps: $ ./bin/opj_decompress -i lena512.i80.j2k -o lena512.i80.j2k.pgm

[INFO] Start to read j2k main header (0). [INFO] Main header has been correctly decoded. [INFO] No decoded area parameters, set the decoded area to the whole image [INFO] Header of tile 0 / 0 has been read. Error : expected EPH marker Error : expected EPH marker Error : expected EPH marker Error : expected EPH marker Error : expected EPH marker Error : expected EPH marker [INFO] Tile 1/1 has been decoded. [INFO] Image data has been updated with tile 1.

Generated Outfile lena512.i80.j2k.pgm

Just for reference EPH handling seems broken according to previous notes. Removing the EPH leads to the same error.

Comment deleted

This issue was updated by revision r2690.

This issue was closed by revision r2692.

This issue was updated by revision r2693.

Both images are OK. Thank you :-) winfried

This issue was updated by revision r2710.