the angle is not random, you are trying to assign a linear value to an angle. to see what I mean, put a text object next to your player
Now in a tick event, set the text value to mouse.absoluteX, then move your mouse left and right (on the X axis)...
that value is the value you are trying to feed into your layout angle, as you can see, moving your mouse to X position 3600px for instance will spin your layout around 10times. (And land up at "a random angle") This is the only reason why you are seeing this "jump", because the update only happens when you right click.
(Try take the first action in Event 25 (Set layout_angle to Mouse.AbsoluteX) and move it into a system tick event and move your mouse left and right to see what I mean.
Now saying that, I think the mouse lock plugin you mentioned should be able to give you a pixel difference between a zero point (wherever the mouse is when you pressed the right button) and the current position of the mouse once moved. My guess is that you should be able to use this difference to emulate the keyboard rotations (Add rotSpeed*dt to layout_angle)
For what it's worth, I don't think this is the best implementation of rotating the world around the character, as you'd probably not be able to use a UI with this "effect" (it'll be spinning around with the Layout... good luck with reading your score... (or you'd land up creating more overhead for your processor by trying to set UI elements the same way you set the player on Every tick to rotate "up")
Also the player becomes rather glitchy when holding down the right mouse button, running forward, and trying to rotate with the keyboard. ( A good indication that this isn't the right way of implementing your idea...
There is a saying... "There are many ways to skin a cat" I think it applies here... Try a different approach to doing the same thing. (Like emulating the keyboard movements (And I'm not saying that it is the right way, in fact it's probably not, but it might be ), or maybe rotating other objects, or layers, or something... just try some creative thinking around your problem.)
Let me know if this points you out of the wrong direction...
PS. Just try and emulate what you are already doing with the keyboard A + D keys onto your mouse as a start, or as an alternate solution, check out performance etc... who knows, there is probably a better way than that too