flvdec: Check the return value of a malloc