Does Ramdhari not stick around each year during his wife and daughter's shrad because he knows that they are in fact still alive?
I think that could be why he said to his Bhabi when she asked him to stick around that he does not deny her anything but don't ask him to stay as he will have to say no to her (I think this was in the 2nd episode but I paraphrased it as I can't remember exact details)