Right Click Dragging the Camera does not cancel selection in hand

When you have a building tool selected and you're trying to place an object, right-clicking to drag the camera deselects that tool.
It would be great if holding right click would not cancel selection. If you only right click once then it would cancel selection.

Pressing Alt key while you drag the mouse with the right mouse button does this, but I always forget to do that. When I am moving around the window by right button mouse dragging, there should be a different key available to cancel selection. Like the ESC button. Or a button that toggles the mode.