Maybe it is the accent. When they scream stuff, it just sounds more malicious, maybe.
But, I also know this... I've followed the Browns to a couple of stadiums (Pittsburgh, Buffalo)... When I'm in a visiting stadium with my squad, I instinctively temper my enthusiasm, just 'because.'
It is an unspoken decorum thing. Most other visiting fans also abide by this principle. That isn't to say that I don't get excited. When something big happens, I'll go nuts on the road. But, you generally don't act as boisterous as you might at home.
New England fans, in general, don't abide by that unspoken courtesy. Chants, mocking of the opponent, etc... It's all on the table.