svq1enc: fix signed multiplication overflow
authorMans Rullgard <mans@mansr.com>
Sat, 26 Nov 2011 16:50:51 +0000 (16:50 +0000)
committerMans Rullgard <mans@mansr.com>
Sat, 26 Nov 2011 18:50:27 +0000 (18:50 +0000)
This multiplication can overflow the signed range but not the
unsigned.  After right-shifting it will thus fit in the signed
range again.

Signed-off-by: Mans Rullgard <mans@mansr.com>
libavcodec/svq1enc.c

index 79aaa6c..01d8b31 100644 (file)
@@ -160,7 +160,7 @@ static int encode_block(SVQ1Context *s, uint8_t *src, uint8_t *ref, uint8_t *dec
     }
 
     best_count=0;
-    best_score -= ((block_sum[0]*block_sum[0])>>(level+3));
+    best_score -= (int)(((unsigned)block_sum[0]*block_sum[0])>>(level+3));
     best_mean= (block_sum[0] + (size>>1)) >> (level+3);
 
     if(level<4){