Don't swap back un-deblocked lines for intra prediction when