100%! Car came with 18in allows, and the previous owner bough 16in black steel wheels for mounting winter tyres. During summer, the wheels looks better with the 18in wheel but the ride is harsher, noisier and even the acceleration is worse. The improvement with the smaller wheels is quite stark. As I am not aggressively cornering or driving, I prefer how the 16in wheels feel, both while driving and riding, given the comfort is that much better.
I can see how bigger wheels are fashionable and help sell the cars, but engineering-wise they seem like a horrible choice for the everyday driver. Most definitely for the average superb driver.
Also, I have tried quieter tyres over the years, and it may help a bit. But the wheel size has much more impact on comfort than the tyre brand/quality.
For me, it is quite practical to have two sets of rims for winter-summer, thus I won't throw away the 18in. However, if I were to buy another car, or for whatever reasons I had to buy new rims, I would definitely go for the smallest wheel available.