Return VLC_ENOMEM when malloc fails.