rtmpdh: Make sure ret is initialized in the nettle version of bn_hex2bn