avfilter/avf_avectorscope: Assert that format is valid