The Z meets the small and fun factor well.
As you probably now, from a financial perspective you would be better just putting the miles on the M3 versus running a second car. From a fun factor though, it is nice to have a choice of what to drive.
I have two completely different types of vehicle and enjoy the contrast. My second car is a Jeep, which here in Phx can have the top off 80% of the year, that works great for running to Home Depot, just taking in the sun or driving more casually. Plus we have some great off road trails not far from town.
If I had an M3, and did not need more room in a second vehicle, my vote would be for the Z4. At least then you are getting a vert for a different feel and use. Not trying to compare the 370Z's performance to an M, but they would feel too much alike for me to want both vehicles at the same time.