avformat/thp: fix variable types to avoid overflows