Here's what I am thinking. If you've liked this guy for 4 years, then obviously it's not some kind of a crush. Do you know if he likes you back? Does he like you as a friend? Do you guys hang out often?
Secondly, if you and his ex aren't really friends then why are you so worried about what she would think? She is no longer with him. So she can't complain if you make a move on him. Also, it's not like you cheated her and went behind her back and stole her boyfriend.
You have to be honest with yourself as to what you feel for him and do you like/love him enough to trust yourself and your feelings?
I hope it helps.
Anam