I definitely would have had to say something. It probably would have been somewhere between Mr. Onzedge's and Mr. Kenchan's responses, where the former was perhaps a tad too harsh and the latter too mild. Also, I wouldn't have taken the time to go inside to get my fob. Immediate attention would have been required. I get it that these are co-workers...but they're being azz-hats.
