You can’t say that completely illogical. It could be borderline for some. if you can’t wear a piece of cloth in isolation, without matching colors/design; you will find it very logical. But if you’re comfortable without matching top & bottom, you will find it illogical.
Just like a shoe for wearing purposes, you can always wear two shoes of two different design/color but you never do that. Bcoz It’s already a deep rooted culture in the society (Rich/poor) for shoes but not in case of matching clothes bcoz many don’t afford. That’s why you find this comparison illogical. Otherwise it’s pretty much the same for someone who afford. I’m sure you won’t wear two pieces from two different suits of clothes that don’t match in color/design.