To debunk your theory, the "slowest" car that ever tried to pull on me was a FRS. And just last night some guy in 535 tried...but failed, we got to a red light he kept saying how much of a monster the Z was.
Maybe they see your face and think they can take you just by the way you look?
jk. you're just unlucky i suppose?