hey there, this is what i would do.
You need 1 variable to store the lerping ammunt, let call this one "num".
So when the player is not mirrored (going right), you add 0.1 to num as long as it is < 1.
when the player is mirrored (going left), you substract 0.1 from num, as long as it is > 0.
and then you scroll.X to: lerp( player.X - 40 , player.X + 40 , num)
I hope it's what you wanted. There is no reason this should flicker, and using lerp every thick should not be a problem either.