Spot-on observation. Sometimes I find myself mentally groaning at the circumlocutions of language required (esp. at work), when all I really want to get across is something as abrupt and easy-to-enunciate as, for example, "oh, shut up."
Rolig, your comments remind me of how my mother used to ask us to clean up a messy room. "Do you want to straighten up a bit in here?" she'd say. One day, on a whim, I decided to be a smart alec about it and answer her literally. "No thanks, I don't."
The telephone use of "I'm going to let you go now" is that polite American trick of acting like you're doing someone a favor when in fact you are telling them to do something (end the conversation). It is related to the "I need you to" locution (as when a nurse says, "I need you to lift your arm"). In both cases, suggesting that you are doing someone a favor or asking someone to do you a favor is a way of avoiding a direct imperative ("Get off the phone"; "Lift your arm"), which Americans interpret as rude. Similarly, we tend to phrase imperatives as questions: "Why don't you open the window?" "Could you turn down the volume a bit?" There is a deep-seated dislike, it seems, for the grammatical imperative.