lavc/jpeglsenc: fix allocation in case of negative linesize, and add malloc check