Remember, if you do wear armor chances are someone else will know about it, if just yourself wasn't enough. Might as well have printed on it "I'm a wuss that can't take a hit nor get my stupid azz out of the way of paint that I know where is coming from in the first place!"
Plus the extra stuff on will slow ya down, which might tick some people off if you're reffing a tourney.