The problem of the “Punch a Nazi” line of thought is not particularly that Nazis are subject to violence : most people (centrists included) couldn’t care less about what happens to them specifically.
No, the real issue here is that people don’t trust the perception of others. You don’t attack a fascist, you attack someone who you think is a fascist. And polarization of the political discourse mean that you can be easily accused of crypto-fascism for pretty much anything (see Hexbear for example). And some people will take it at face value, and hence feel justified to attack you.
Crabs. All will be crabs.