Baldev has yet to interact with Anami directly. Somewhere he is scared to. That's why he wants her out of Lal Mahal. Apart from his power struggle with Satrupa - he is unconsciously protecting himself from Anami and the emotions she could bring out in him. After losing Vatsalya - that's understandable.
His parting shot to Satrupa that he will never love Anami - makes me think the converse will happen and that too rather fast.