All offices attached to the seat of government shall be exercised in the District of Columbia, and not elsewhere, except as otherwise expressly provided by law.