avformat/smacker: use av_malloc_array() and check for allocation error