Click the annotation text.
Right-click any manipulator and check All Around from the contextual menu.
Un-select the annotation.