When I was 16 I was stood at a bus stop when a group of men started shouting at me. "Give us a smile sweetheart," they called. Embarrassed, I looked away and pretended they weren't talking to me. "Fuck you then, you ugly bitch" they yelled. A year later I was walking down a busy street in the middle of the afternoon and suddenly felt a hand between my legs. A man who was walking towards me had reached out, grabbed me and then continued walking without even looking up.

Skip forward to my late 20s. I'm walking with my baby daughter in her pram when I received a text message. I glanced at my phone to see who it was from and as I did so, a middle aged man leaned out of his car window and shouted: "Look after your fucking kid you stupid cow, stop looking at your phone."

This type of abuse of women is not unusual. Most women have experienced something similar – certainly most of my female friends have. It's not just on the street. Some have experienced verbal and physical abuse in the home. Others have dealt with sexual harassment in the workplace.

Some of these incidents were part of prolonged abuse. Some were one-offs, like the time a young office junior mentioned to me that she had a urinary infection, only for a 65-year-old man to immediately ask her if she had "had too much cock". She blushed and hurried out of the room. "I was just teasing," he said.

While each of these incidents is very different, the one thing they have in common is that the perpetrators were all men.

In recent weeks, there has been a focus on the abuse received by MPs – and rightly so. Abuse should always be called out and condemned, no matter who it is aimed at. But both the main political parties and have turned this into a weapon to attack the other side with. Tories blame Corbyn for not clamping down on his abusive supporters. Labour point to the dog-whistle campaign against Sadiq Khan in the London mayoral race and the emphasis placed on Diane Abbott during the general election campaign. Both sides make valid points but in doing so they miss the bigger problem.

To suggest that abuse is a problem for one particular political tribe is disingenuous. It comes from all sides. And it's not just a party political issue. A few weeks ago Stella Creasy was sent images of aborted fetuses because she dared to speak up for the rights of Northern Irish women to access abortions in the UK. Women who speak out against male violence receive threats and are called names. The writer and campaigner Caroline Craido Perez received rape threats for standing up for women. It goes on and on.

The problem with focusing on the particular group these people claim to be part of, whether they're Corbyn supporters, Tories, Scottish Nationalists or Ukippers, is that it detracts from the perpetrators themselves. Instead of focusing on the person who sends the abuse, an argument erupts over which side gets it worst and who is to blame. It becomes about left and right rather than abuser and victim.

Online abuse is the same as real-world abuse in at least one respect: More often than not, it comes from men. They hide behind other issues to 'justify' what they are doing, but the fact is that they are abusers and many are also misogynistic, racist and homophobic. Their political views have little to do with it. They will always find somebody to attack, regardless of the issue. Just like the men who harass women in the street or abuse them in the home, they always have a 'reason'.

As teenage girls grow up, one of the lessons they unavoidably learn is how to deal with harassment and abuse. When I was at school, a man used to stand outside the gates pretending to be blind. As girls came out at lunchtime he would brush up against them and pretend it was an accident. A good friend was flashed at when she was about 14 by another man. Women are forced to get used to this type of abusive behavior from an early age.

The only thing the internet has changed is to provide anonymity and distance to those who would previously have shouted at us in the street. This shouldn't be about party politics. The left doesn't have any more of a problem with misogyny and abuse than the right does. If we really want to tackle the issue we must look at the roots of it: male aggression.

Natalie Bloomer is a journalist for Politics.co.uk. You can follow her on Twitter here.

The opinions in politics.co.uk's Comment and Analysis section are those of the author and are no reflection of the views of the website or its owners