I have run multiple times into the following problem: My M5 - balanced perfectly, all the screws and bolts tight - starts jittering only when tilted strongly downwards. I have set the stiffnesses conservatively - tilt and roll 10 units from jittering and pan 20 units. Also made sure that there is no binding in the cables. I think it is pan axis that jitters. If I decrease pan stiffness more than 20 units from jitter, pan becomes a little restless. Other than that there are no jittering or other problems. Canon 5D3 and Canon EF 16-35mm f/4. Anyone had this problem/found a solution?
Have you checked balance with the camera tilted down to the same spot where you are seeing the problem?
Tuukka: Can you use the Configurator app and go into Chart mode and see what the tilt, pan, and roll motors are doing -- you can easily tell which axis is then causing the problem. Andy.