Perhaps I am, although I don't think so...

What do you think a better reaction would be?

(My current approach is if the person themselves is problematic, I grow distant. If they're a friend, I ask whether I could have some time to myself. This is for all kinds of relationships, although nowadays I rarely feel the need for distance in the first place.)