avfilter/vf_vectorscope: make color mode more useful