Many moons ago – in the mid-1980s to be precise – I was fortunate enough to enjoy some of the best experiences local football can provide.

Wisbech Town were my team of choice – Fenland Park was a 15-minute walk from my home which, curiously was in Norfolk whilst my garage was in Cambridgeshire, but that's another story.

It was a vibrant place – fearsome for many visiting teams, with a red-hot atmosphere, particularly for big games when the fan base would increase and the, erm, unsavoury element sometimes appeared.

In consecutive season Wisbech reached the semi-finals of the FA Vase, losing to Halesowen Town ion 1985 and Southall the following year. They were ferocious affairs on and off the pitch – at one stage I thought the stand which housed the press box which housed myself was going to be kicked down by visiting fans.

Wisbech got themselves a reputation. I found this reference on a non league fans site that summed things up: 'Wisbech v Stortford in the East Anglian Cup final about 20 years ago was the most intimidated I've ever felt at a non-league game. They were looking for trouble from the minute we got there to the minute we left. The local plod were out in force, and that's the only reason that we didn't get our heads kicked it. Stortford's keeper still clams he missed the sixth and decisive spot kick in the shoot-out to save our lives.'

Wisbech now play at the Elgood' Fenland Stadium: very different from the old place... most of the time.

Last weekend, Wisbech were in FA Vase action at home to Bromsgrove Sporting. As crowd of 650 made for a cracking atmosphere, and it was an exciting game, which Wisbech were unfortunate to lose 3-1. But there was always a frisson of uneasiness. Without wishing to sound rude, some of those in attendance looked like they could handle themselves although, to be fair, they never bothered to try and prove my theory. Their biggest 'crime' was to let off a few flares.

But there were incidents which don't have a home at football, no matter where. Just before half-time, the visiting keeper, Reece Francis, was spoken to at length by referee Edward Pidduck. It seems he complained that he had been racially abused by someone behind his goal, which is where many Wisbech fans were standing. As he walked off the pitch at half-time he was visibly upset. Such abuse does not have any part to play in our society: that's a given, yes?

I spoke to a number of Wisbech fans after the game and they all said he was mistaken; that fans had chanted a song that used the word fat... and he believed they were saying the word black.

It wouldn't be the first time this confusion has arisen: Norwich fans were accused of the same thing at Everton back in 2004.

Everton officials claimed they were targeting Joseph Yobo and used the word black when in fact they were directing a chorus towards Wayne Rooney that used the word fat.

Anyway, Francis heard what he heard. But still I have a concern or two: why, in the second half, did he do monkey impressions – you know, hand twisted under the arm in a scratching motion – to the home fans? That doesn't compute.

I know he was upset but I really hope he was mistaken, in an Everton way.

I felt sorry for him, but I felt a lot less sorry for his team-mates, whose celebrations at the final whistle were over-the-top. Fair enough, they had a right to celebrate. But gloating as they headed back to the dressing room? Not on. It is what caused one or two Wisbech fans to react badly – and incorrectly – and led to some fisticuffs.

All every unnecessary downsides to a cracking game – even though Wisbech did have two players sent off. Paul Bastock was one of them, with two yellows in as many minuets – but did the ref protect the veteran keeper as flares headed in his direction and as the ball was clearly kicked out of his hands? Nope, not at all).

It may sound like sour grapes on my part, but it's not. I just think Wisbech got a bit of a raw deal. I hope I am right about the racist comments and I believe I am. But we don't need a throwback to the dark old days, at Wisbech or anywhere else.

Having a red-hot atmosphere is fine, as long as people are treated with acceptable levels of civility.