checkasm/jpeg2000dsp: refactor rct_int test