libavformat/rtsp: return error if rtsp_hd_out is null instead of crash