avfilter/paletteuse: fix error dithering accuracy