Anxiety
Social anxiety can make it hard to converse with someone if they make you feel uncomfortable or pressured.
Conflict avoidance
Even without pre-existing anxiety, many people dislike conflict and the drama or aftermath it brings.