Analysis: Lisbon treaty U-turn haunts Cameron’s Europe long game

David Cameron's frustrating Europe obfuscation can be traced back to his other big crisis on the issue: abandoning the Tories' "cast-iron guarantee" over the Lisbon treaty referendum.

There's a lot of headscratching going on in Westminster right now about David Cameron's upcoming Europe speech. Quite right, too. He's all for Britain being in Europe, but thinks Europe should give Britain more powers back, but thinks Britain should choose whether it's in Europe, but isn't going to let that choice take place any time soon. Simple, really.

The real key to understanding Cameron's approach is to look back at the only other time when, as Tory party leader, he has had to confront the Europe monster head on. Then, as now, the aim was to preserve as much flexibility as possible.

A lot has changed in British politics since 2009, but in one respect life was not so very different: the leader of the divided Tory party had no option but to make a painful policy adjustment on European issues. It harnessed frustration with the Lisbon treaty, which eurosceptics complained was suspiciously close to the abandoned European constitution project, to help win hundreds of councillor seats in local elections. But with the 2010 general election approaching, Cameron had a problem. He had promised a "cast-iron guarantee" of a referendum on the Lisbon treaty promised to voters. Now he and his policy advisers thought about it, the reality of such a referendum taking place didn't seem so attractive. He had to back down.

November 4th, 2009. The St Stephen's Club, close to St James' Park. And with that statue of Winston Churchill gazing down upon him, Cameron explained the new Conservative approach to Europe. It was a masterpiece in obfuscation, containing just about enough to keep the Europhobes happy while containing enough meat for eurosceptics. The 'referendum lock' tied Cameron down, but only in the event of real change on the continent. The idea of a grand sovereignty bill seemed so vague as to be virtually meaningless. But it did the trick: the following spring, Conservative candidates asked about Europe on doorsteps across the country were able to offer a broadly sceptical line, with plenty of built-in flexibility to suit themselves and their constituencies.

Four years later, and the Tory leader – now the prime minister in a coalition government whose two parties have comprehensively different approaches to Europe – faces his next big European headache.

Once again, the PM faces calls for a referendum. Once again he must offer his eurosceptics something, while avoiding actually giving them the in-or-out choice they seek. As in 2009, we can expect him to cover his bases. The PM's problem is Europe is so divisive for the Tories there will never be a single position that pleases the majority. So he has to pursue the more limited aim of being as close to the majority's views.

The task is harder in 2013 than it was in 2009, because Cameron does not have the luxury of being in opposition to help him along. The promise of a sovereignty bill is no longer realistic. Yet the principle of pushing the can along the road is as valuable as ever. Deferring the issue until the next parliament, making it as drawn out as possible, helps the PM by putting time on his side.

Europe is not like other policies. It is not toxic like the NHS, where the public's view of the Conservatives is suspicious and full of spite. No, Europe is the kind of poison that rots from within. The larger the controversy, the more damage it causes. A controversy worth putting off however possible.

This time around Cameron has the legacy of the Lisbon treaty hanging around his neck, though. The rebelliousness of his backbenchers, fuelled by that Lisbon treaty U-turn, has already resulted in two humiliating Commons defeats for the prime minister on the issue. Cameron's room for manoeuvre is shrinking as a result. Still, he can keep on fighting for now.

We now know the prime minister is playing for 2020. That is his target date to survive in politics. If he succeeds, he will have been the Tory leader in three parliaments: one as leader of the opposition, one in coalition as prime minister, and one – he hopes – as PM in his own right. Maintaining his MPs' support on Europe over that period involves a gradual narrowing of options. The trick is to manage the pace of concessions so they do not threaten his leadership. Using the eurozone crisis to delay the confrontation with Europe fits Cameron's personal leadership timetable very well.

When the Tory leader stands up to deliver his Europe speech next week, he is not just responding to the situation in the eurozone. His is a long, long game of power politics as much as principle. Eurosceptic pressures are slowly cranking up, made worse to this day by the Lisbon treaty U-turn. Cameron cursed himself with that policy, and must spend the rest of his leadership avoiding the consequences.