swscale: move format handling to its own function