avformat/hcom: check probe buffer size