It would be great if this option could be disabled in the settings, as it really gets in the way when you try to click on the balancer in a hard-to-reach place.
That's true. If you are trying to click on a balancer that is occluded by port arrows, exit build mode to hide all arrows and then you will be able to click directly on the balancer.