avfilter/vf_rotate: fix several of by 1 errors