I don't let anyone's insecurities, emotions, or opinions bother me. I know that if I am happy, that's all that matters to me. And it's OK to be selfish like that sometimes, when it comes to your well-being. Do what makes you happy, and don't care what others think.
