avfilter: fix plane validity checks