If you click on your time object in the layout you will notice on the left hand properties section that it will have a different "layer" to your graphics. This layer is likely set to not scroll (unless you are setting the time to stay in the middle via events).
You have some options really, set it to (or set up) a layer that doesn't scroll. Give it the anchor behaviour via the behaviours or set it via events using maths with viewport calculations.
If that sounds confusing then check over Kyatrics FAQ as I am sure you will find your answer there with a helpful example cap.x