softfloat: handle INT_MIN correctly in av_int2sf
authorAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Sun, 8 Nov 2015 16:19:10 +0000 (17:19 +0100)
committerAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Sun, 8 Nov 2015 20:06:40 +0000 (21:06 +0100)
Otherwise v=INT_MIN doesn't get normalized and thus triggers av_assert2
in other functions.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
libavutil/softfloat.h

index 4cc3ceb..5b285e3 100644 (file)
@@ -153,7 +153,12 @@ static inline av_const SoftFloat av_sub_sf(SoftFloat a, SoftFloat b){
  * @returns a SoftFloat with value v * 2^frac_bits
  */
 static inline av_const SoftFloat av_int2sf(int v, int frac_bits){
-    return av_normalize_sf((SoftFloat){v, ONE_BITS + 1 - frac_bits});
+    int exp_offset = 0;
+    if(v == INT_MIN){
+        exp_offset = 1;
+        v>>=1;
+    }
+    return av_normalize_sf(av_normalize1_sf((SoftFloat){v, ONE_BITS + 1 - frac_bits + exp_offset}));
 }
 
 /**