avfilter/vf_rotate: increase fixed point precision