Why even "refuse" the visit?
In my opinion, tolerance is something that should not depend on laws, but rather on one's ethical conscience. Since - in my view - monks bound by a strict vow of celibacy should have the right to avoid contact with women (provided they do not deny women their rights, which BAPS does not do within the women's section of its organization), it is acceptable to tolerate this and accommodate the guests.
What I do criticize, however, is the lack of transparency in the communication with the women - t h a t was clearly a display of power and should not get accepted.
I also consider a blanket rejection of such requests - like those made by the monks - to be unfair and 'not okay'... I am convinced that informative communication among the involved parties should be the base of decisions to make.
15