I once removed the O-ring "gasket" from a R1100S fuel tank to replace the filter, and could not get the O-ring back due to it being swollen. I washed it with soap and water and left it to dry, and a week later it had shrunk to the extent that it could be re-installed and it functioned without problems. I've done this twice with the same results. The swelling is a natural phenomenon and aids in the sealing action, and in some instances may be reversible. I would however not recommend this for a brake or clutch system, and doubt if it would yield similar results.
The rubber seals in a brake/clutch system designed for DOT 3/4 fluids are most likely EPDM which is not resistant to mineral oil. The swelling is a result of the rubber absorbing the oil, and leads to a change in the rubber's mechanical properties such as becoming softer. It would be difficult if not impossible to "extract" the mineral oil from the rubber seals, and even then the rubber may have permanently changed its properties.
However, not to confuse the issue, the advice given by John Zeiler is correct. You will have to replace every seal from the master cylinder right down to the slave cylinder, in fact you should replace every rubber part that was in contact with the mineral oil. I would be hesitant to retain the contaminated hoses, since depending on the construction thereof it may use EPDM liners which would also have absorbed the mineral oil. Once refilled with DOT 3/4 the mineral oil may be released to contaminate the new DOT 3/4 fluid. Some publications recommend that rubber brake/clutch hoses be replaced after 6-10 years, so it may be a good time to get that done too. In for a penny, in for a pound!