Madhav has been looked down upon and ignored many times since his childhood..as we saw in the school episode, for not having a conventional pretty face!
He has developed a sense of inferiority complex from then and that resulted in a "Madhav" who is not convinced enough that he'd be liked/loved by anyone despit being a successful gentleman with a good heart!