Both cases are considered harmful, the surer and heavier more general case being for the former.
The inertness alone means boost in inflamattion, damaged cells, etc. Just consider that asbestos is biologically/chemically inert too - the issues come from inflammation, scarring, dna damage, etc.
But (and varying per microplastic case on lots of factors: composition, dyes used, etc), the chemical interaction can also play a role.
