That's happened to me too - I actually came back out to the car from my apt and found both windows down!
It hasn't happened again so I put it down to the keyring pressing on the keyfob while it was in my pocket - needless to say I don't have any other keys on the fob now.