When Yudhisthir lost the game of dice, only he was exiled to the forest, not his brothers. They couldn't have gone to Indraprastha, but they could have gone anywhere else - Dwarka, Chedi, Kashi, Magadha, Panchala... Yet they chose to go into exile w/ him.
Do you seriously think that had Yudi decided to go to the forest w/ Dhritarashtra, that Bhima and Draupadi would have accepted the throne?
Brothers and Panchali were included in the 2nd dice game contract. But yeah, younger Pandavas would've again accompanied Yudhishtira, leaving perhaps Parikshith as king in name and Dhritharashtra/Vidur as regents. I have my doubts about Panchali accompanying them at that point, though. In that little snippet, she doesn't even ask Yudhishtira to stay, simply derides him as a lunatic who should've been locked up.
Edited to add: contract included Yudhishtira and Panchali, not the brothers. I'm guessing the Kaurava side really wanted her out after what she pulled off in the dice hall.
"Sakuni then said,--'The old king hath given ye back all your wealth. That is well. But, O bull of the Bharata race, listen to me, there is a stake of great value. Either defeated by ye at dice, dressed in deer skins we shall enter the great forest and live there for twelve years passing the whole of the thirteenth year in some inhabited region, unrecognised, and if recognised return to an exile of another twelve years; or vanquished by us, dressed in deer skins ye shall, with Krishna, live for twelve years in the woods passing the whole of the thirteenth year unrecognised, in some inhabited region. If recognised, an exile of another twelve years is to be the consequence. On the expiry of the thirteenth year, each is to have his kingdom surrendered by the other. O Yudhishthira, with this resolution, play with us, O Bharata, casting the dice.'
p. 146