You guys can give all the reasons you want for staying silent, but if you stay silent just know that you're culpable. And if you're ok with that, that's cool.
I sat by silent during a racist diatribe a year or so ago at a store and it pissed me off so much afterwards that I didn't say anything, that it will never happen again. It wasn't a physical altercation or anything violent. It was just a stupid, ignorant, racist piece of garbage loudly proclaiming her idiotic bigoted beliefs to the world around her. I had a duty to tell her to shut the fuck up and that no one wants to hear it, but I didn't do it. You don't have to be a physically intimidating person, or strapped, to tell someone to shut up.
Part of the reason why people act like assholes in public is because no one's ever told them not to.