It comes from the story of Julius Cesear when he is stabbed at the back by his foes and among them he finds his good friend Brutus who also stabbed him. Cesar says it then which means that you too joined others in destroying me
Similarly in this case when Sid says - kaho woh Aarman nahi tha - he essentially expressed his shock and pain at the betrayal from a friend! This sentence only means that he had been stabbed from the back.
So much so for calling Sid dumb when these are words signify utmost betrayal