avcodec/jpeg2000dec: Fix undefined shift in the jpeg2000_decode_packets_po_iteration...
authorMichael Niedermayer <michael@niedermayer.cc>
Fri, 4 May 2018 17:18:25 +0000 (19:18 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Sun, 6 May 2018 19:02:04 +0000 (21:02 +0200)
Fixes: shift exponent 47 is too large for 32-bit type 'int'
Fixes: 7955/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-6016721977606144

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/jpeg2000dec.c

index 5aa0713..12dcc5c 100644 (file)
@@ -1181,10 +1181,10 @@ static int jpeg2000_decode_packets_po_iteration(Jpeg2000DecoderContext *s, Jpeg2
                         int xc = x / s->cdx[compno];
                         int yc = y / s->cdy[compno];
 
-                        if (yc % (1 << (rlevel->log2_prec_height + reducedresno)) && y != tile->coord[1][0]) //FIXME this is a subset of the check
+                        if (yc % (1LL << (rlevel->log2_prec_height + reducedresno)) && y != tile->coord[1][0]) //FIXME this is a subset of the check
                             continue;
 
-                        if (xc % (1 << (rlevel->log2_prec_width + reducedresno)) && x != tile->coord[0][0]) //FIXME this is a subset of the check
+                        if (xc % (1LL << (rlevel->log2_prec_width + reducedresno)) && x != tile->coord[0][0]) //FIXME this is a subset of the check
                             continue;
 
                         // check if a precinct exists