faxcompr: Check malloc results and unify return path