I was thinking about taking the standard silver wheel trims off and just having black steel wheels for a change, but it appears that the wheel trims are shaped in the middle to protect the centre wheel hubs.

Would driving in the rain etc without the wheel trims on speed up damage to the hubs/bearings/other wheel component?

I am happy to continue driving with the trims on, if that is what is best for the car...

No, the wheel bearings are covered by a cap full of grease.