avfiler/vf_mix: fix crash with >8 bit depth