flacenc: ensure the order is within the min/max range in LPC order search