It could use slots to allow mixing of articles of clothing. With, for example, "shirt", "pants", and "dress" slots, a a shirt and pants can be worn together, but a dress will conflict with both.
These slots can be made configurable so you can set them up in as many categories as you like, with per-slot conflict rules.
The animations are already created, the program only needs to display them, correct? With support for dragging an article of clothing to and from the model?
I know Blender pretty well, no experience with Maya or OptiTex, and I'm learning graphic design and digital art, so I can help with those aspects of your project as well, if you want.
Thank you for your consideration.
Martin