I've had this problem in many contexts. Some of it stems from childhood circumstances, but it's also that I often read people better than they seemingly know themselves, i.e., noticing anger that they don't think they have. This can cause a lot of stress.