A fair point. Well... I guess it depends on what atmosphere surrounds you, whether you can opt out without feeling ostracized. Maybe I just don't care what other people think of me...? (Probably why I'll never get promoted!)
I don't know that it's entirely harmless. If you choose not to participate, you run the risk of being ostracized. If you participate against your conscience, you feel terrible. It's exactly the wrong thing for holiday spirit.