Yes it's harder for the one whose disappointed. It takes a lot for someone like HS to get to that point, which is why it's so difficult for me to see him in this state.
That last look at the end of the episode... he was still fighting with himself to respond to her smile with one of his own, but failing miserably. Woh sach mein suli pe chadh jayega, aasmaan se chaand taare tod ke layega, apni jaan de dega, aur pyaar ki qurbani bhi, sirf apni Veer Balika ke liye. He will continue to battle with himself.
How does a parent feel when their child rejects them? It's gotta hurt more than being rejected by any other type of relation.
God forbid, no one should ever feel like that. NEVER.