There have been one or two calls to brand Ukip a racist party, and some media debate about whether it is. But what’s become clear during the last fortnight is that there is a strong, cross-party consensus both that Ukip isn’t a ‘racist party’ – and that it must get better at keeping out individuals that hold racist views.

On debates such as the BBC’s Question Time, you can clearly see that this consensus has extended across the front and backbenches of the different parties. Just one or two MPs take a different view. Labour left-winger Diane Abbott told the BBC that she did not regard Ukip as racist but saw it as having US ‘tea party’ tendencies.

There are good reasons why the general public, the media and his political rivals treat Nigel Farage differently to Nick Griffin. Ukip does not have extremist roots. It was founded to get Britain out of Europe, a legitimate cause. But an anti-EU party which campaigns to end EU free movement will attract both legitimate voices for that cause, and those who have more virulent and toxic motives.

Claiming that Ukip is just the BNP in blazers doesn’t work – because it isn’t credible with the public, the media or the mainstream parties.

A better challenge to prejudice within Ukip would be to welcome the fact that Nigel Farage does not want to lead a racist and toxic party like the BNP. Since they have attracted some of the same people, Ukip’s willingness and ability to police the boundary seriously becomes a test of Nigel Farage’s leadership and credibility. It is precisely because UKIP is not considered a racist party that it makes sense to call on Nigel Farage to kick racists out of Ukip. Nobody has ever bothered to challenge Nick Griffin to kick racists out of the BNP. What would be the point? There would be nobody left.

As the final, fatal demise of the BNP this month will exemplify, Nigel Farage knows that no party which does not accept that black and Asian citizens are equally British will get ahead in the Britain of 2014.

Nigel Farage has promised he’d clean up Ukip’s act. He’s expelled people who have been caught, but needs to do much more to keep his promise to properly vet his candidates so that he gets embarrassed less often. He should also take more responsibility for Ukip’s local campaigns that fall on the wrong side of any reasonable democratic debate.

Its important that we debate how to manage immigration effectively. But Nigel Farage should promise that there will be no more Ukip leaflets comparing British people to Native Americans who ‘didn’t worry about immigration and now live on a reservation’.

Nigel Farage needs to tell Ukip candidates and local parties that voicing extreme slogans like ‘no more mosques’ is unacceptable. It’s a view that falls on the wrong side of the British tradition of religious freedom. It also voices a prejudice which makes British Muslim integration harder to achieve.

After all, if Nigel Farage is serious about his side having a chance in a future EU referendum, he will need to do much more to broaden his support. Ukip’s vote is overwhelming over-65, strongly male and 99.5 per cent white. That looks like a surefire way to lose an EU referendum where you need to appeal to over half of the British people, even if it could win a low turnout European Election – where 15 million General Election voters stay at home.

As committed Eurosceptics Douglas Carswell MP and Dan Hannan MEP have argued, persuading over half of the British people that we’d be ‘Better Off Out’ would be quite impossible were a campaign to offer nothing but anger and pessimism about what has changed over the last half century.

So rather than calling Ukip racist, we should be clear about the boundaries and norms which its claim to be a mainstream party depends upon. We should be confident about protecting these norms – and about Ukip’s need to adopt them too.

One blogger recently claimed that anti-prejudice norms had retreated fifty years in the last fortnight. This simply isn’t the case.

There is no chance of a timewarp back to 1964. The reduction in racial prejudice in Britain is not the type of shift that could so easily be reversed. Media commentators are particularly prone to overstate the importance of specific news events. In the long run, prejudice norms are more like values than attitudes. It seems intuitively natural to worry, for example, that levels of prejudice might rise in harder economic times, but the attitudes data shows no evidence that has happened since 2008.

That’s why Ukip has had to ditch and expel candidates who have made racist statements – because voicing such views is repulsive to all but the most extreme fringe of voters.

At the same time, many potential Ukip voters agree with Nigel Farage’s claim that charges of racism have been used too loosely, too quickly, and too often to close down debate about immigration.

It is important, therefore, to make clear that we must talk about immigration but that we should do so without prejudice or scaremongering. This will help secure the broadest possible coalition for keeping racism out of the public debate.

Appeals to xenophobia in Ukip’s language should be countered and challenged – while always taking care not to reinforce claims that legitimate debates about immigration are being closed down.

Shouting Ukip down as a racist party risks reinforcing one of their most effective claims with target voters: that legitimate issues, which need talking about, are being swept under the carpet.

Ukip’s vulnerability is different: ‘You can’t still be complaining that people can’t talk about immigration – we’re all talking about it now. You are good at the grievances, but I don’t suppose you’ve got any answers too?’

If Nigel Farage wins the European Elections, it would be odd to try to deny that he has a legitimate democratic voice. The better question is whether he is ready for prime-time, or whether a protest vote in 2014 will be his peak.

Ukip’s political opponents will, and should, continue to expose extremists within its ranks. But they should realise that this will never be a knock-out blow.

What would make most difference is mainstream political voices showing how anxieties about identity and immigration can be addressed, constructively and fairly, in a way that does not scapegoat people or stir up prejudices.

Sunder Katwala is director of British Future. Follow him on Twitter @Sundersays