Last week the prime minister was forced to write to his party associations after reports that a member of his inner circle called grassroots members 'swivel eyed loons'. Political editor Annabelle Dickson met activists on the ground to ask if that accusation is fair and to gauge the mood within the party two years from the general election.

'Forty swivel-eyed loons are coming for curry tonight,' South-West Conservative Association chairman Lady Fisher said with a mischievous smile when I arrived at her house on Friday.

Most in the party have taken the alleged comment by an as yet un-named senior Conservative with good humour.

Broadland's Tory chairman, Greg Peck, even replied to the letter hurriedly sent out by David Cameron signing himself off as a swivel-eyed loon – but also with his considered thoughts about what the party should be doing.

Rural Norfolk may be a world away from the corridors of national power in Westminster, but big political issues such as what should be top of the legislative programme, how the government can improve the economy and how to tackle the rise of UKIP are being carefully considered.

Certainly there is an acknowledgement that there are problems for the party, but not necessarily the high profile issues of gay marriage and the EU which have made headlines recently.

Mr Peck, who settled in Norfolk after a career in the car industry which took him around the world, leads the party association in Broadland MP Keith Simpson's constituency. He acknowledged the county council elections were a warning that the rise in UKIP's popularity in Norfolk could not be ignored.

'It means we are not connecting with the voters as we should be,' he said.

'I think it is important that the leadership is not complacent and doesn't assume that the voters lost to UKIP in the county council elections will necessarily return to the Conservatives at the general election.

'Whilst governing as part of a coalition is fraught with difficulties, the party leadership needs to take control of the agenda,' he added.

However, Mr Peck showed none of the pessimism reported to be present in some quarters of the party. In fact he was upbeat and felt there were many positive things to talk about.

'We need to emphasise the good things we are doing, particularly regarding benefit and educational reforms. Plus, of course, the efforts we are making to fix the broken economy we inherited,' he added.

For most members there is an acceptance that things have to be different within a coalition, but there is certainly a view that the Tories should be more willing to stand up to their Liberal Democrat partners. Lady Fisher said the coalition made it 'very trying to enact proper Conservative policies'.

And former Mid-Norfolk chairman, Cliff Jordan, who has been involved with the party for 40 years and served as a district and county councillor, was also clear that he wanted to see Conservative values where the 'government should get out of the bloody way'.

But he was charitable about the difficulties of governing.

'To be honest, I think Cameron has a hard job. It is easy to criticise someone when he is up the front. It is easy to sit back here and throw it at him,' he said.

But he was clear that he had been unhappy with the gay marriage bill.

While he said he would not leave the party over it, he said he was opposed to gay marriage on religious grounds.

'It is not in the scriptures and I do not believe any government has the right to countermand them' he added.

Ian Sherwood, who is party agent for Mid-Norfolk, South-West Norfolk and North-West Norfolk, also acknowledged there were mixed views on gay marriage within his three associations.

He said: 'We have members who are comfortable and other members who think there are other things that may need attention. It definitely isn't as polarised as sections of the media would have you believe. I've always thought the Conservatives have a very broad membership of individuals.

'Whatever political party you belong to, there are always going to be issues that do not always fit your viewpoint. I have certainly had members of the party speak to me about it from both viewpoints.'

But while there were concerns about presentation and a feeling that gay marriage should not be top of the agenda for the government, there were things the membership was happy with. Mr Jordan said: 'I think the EU is bureaucracy gone crackers and it needs to be sorted, but just jumping ship will not sort anything out.

'And I like academies and free schools and Michael Gove is trying to push that forward.'

And Lady Fisher agreed that she liked the message of 'raising standards' in education put forward by secretary of state Mr Gove and his Norfolk childcare minister and her MP Elizabeth Truss.

Nevertheless, following the county council elections where UKIP gained many more seats than predicted, the party is still preoccupied with political engagement and why UKIP is doing so well.

Many in the party said they knew former Tory voters were supporting the anti-Europe and anti-immigration party in the last election.

Lady Fisher said most of her staff had voted UKIP, while Mr Jordan said one of the members of his ward who he had helped also told him he was no longer voting Conservative.

Mr Jordan said: 'He said I had done a good job, but we are going to vote UKIP. We are sending a message to Cameron. I said to him the other day that he had got a hung council out of it and we cannot now do what we were doing. We had a programme to go forward and we cannot do it now.

'This tells me that people are sending messages to the top elite of their parties that their heads are in the clouds and could they come down and see them.'

Political parties are losing members and in some quarters memberships are declining. While Paul Wells, chairman of the newly formed Norwich Conservatives (created by combining the Norwich North and Norwich South constituencies) said the party is growing its membership through better organisation.

But South-West Norfolk Conservative membership fees fell from £20,000 over three years, to £12,000 last year and £8,000 this year, Lady Fisher said. It is not just the loss of votes to UKIP which is a worry.

Ian Sherwood said general political disengagement was a big problem for all the parties, with very few people bothering to come out and vote.

He was keen to emphasise how much it mattered – citing the count in Thetford in the most recent county council elections where there had been one vote in whether UKIP became the second biggest party on the council.

Certainly on the ground there are strong views both publicly and privately that all is not rosy and there is a lot of hard work needed if the Conservatives are to win a majority at the next general election. There is concern about the size of the party, but that has not been prompted by the gay marriage debate. They all acknowledge there are many other reasons for declining memberships in some quarters.

But this is not a recent change and members have hardly batted a swivelled eyelid about the loon accusation.

As Mr Jordan said: 'I've been called a lot worse.'