In my case, when I dropped from 19 to 18, combined with my tire changes I did reduce the overall circumference. And yes, mass matters more the further it is from the center of the wheel.
The "top speed" factor isn't worth mentioning since the gearing on our cars goes way past the point of drag-limit anyways. You'll never reach the top speed of the gears+wheels regardless.
"Increased turn-in response" isn't a function of wheel size
AFAIK. It's a function of running shorter (as in thinner) tires, which you can do on any reasonable wheel size.
The only good "performance" argument for increasing your wheel size is to fit larger brake rotors. Otherwise the ideal is a smaller wheel with a low profile tire.