rtsp: Lazily set up the pollfd array once