If you go down to the British Library today, you’re sure of a big surprise. Because as of last weekend, it’s archiving not just every book published in the UK (its traditional role), not just every e-book published in the UK – it’s archiving every website based in the UK. In terms of what we’ve conventionally understood by the word ‘library’, it’s as big a change as there has ever been.

‘Capturing the nation’s digital memory’ – that’s the phrase the British Library themselves are using about the venture. Your first response might be: ‘the internet archives itself, doesn’t it? It’s called Google.’ But as Lucie Burgess, the library’s Head of Content Strategy, tells me: ‘There are lots of websites that cease to exist – politicians’ ones, for instance. Straight after a general election candidates’ sites get taken down, or, if they’ve been elected, get turned into their site as an MP.’ Plenty of politicians would no doubt be happy for those sites to disappear; no awkward promises to be reminded of. As the years go by, and more and more content vanishes into web’s black hole, the library’s archive will become ever more valuable.

Initially it’s going to be just an annual snapshot, a once-yearly record of the websites ending ‘.uk’ (or ‘.com’, ‘.org’, etc sites that have UK-generated content). They number 4.8 million at the moment, with over a billion web pages between them, though who knows what that total will reach in the future. And you’ll have to visit the British Library itself to access the content. But Burgess says that the technology may change. The capability is there even now to capture the web daily; in the future it might be possible hourly. The servers doing the work look sleek and modern, rather like a row of designer fridge-freezers, though I can’t help thinking of HAL in 2001: A Space Odyssey.

‘The nation’s cultural and intellectual output is increasingly appearing in digital form,’ says Burgess. ‘We’ve got to reflect that in the way we operate. The historians of the future are going to find this an invaluable resource.’ It so happens that we’re talking an hour after the announcement of Baroness Thatcher’s death. For someone in 2063 to research the public’s response to that event without having access to Twitter would be unimaginable. (The library, according to Burgess, hasn’t always taken this forward-thinking approach. ‘One of the reasons newspapers were stored up at Colindale in the 1920s was that back then they weren’t thought important.’) It isn’t only at the British Library that attitudes are changing: Christopher Platt of the New York Public Library wrote in the recent issue of Spectator Life about the way his institution is responding to the digital age.

Part of me feels a little uneasy about the new system. In the old days the finest of everything – scientific knowledge, philosophy, poetry, you name it – went into books, which the British Library stored. Now there’s no filter, no sense of quality control. Anyone can publish any statement about any subject and the library will preserve it. A teenager in her bedroom in Swindon blogging about the latest X-Factor contestant gets as many bytes as the Poet Laureate. It’s the equivalent of a thousand billion cave paintings, with historians of the future given a search engine (however sophisticated its algorithms) and told: ‘Pick the cultural significance out of that lot.’

But then who was the filter in the past? A few dozen people in central London, essentially. Publishers (be they of books or newspapers) decided what picture you and I would get of their time. OK, they found Shakespeare for us, and Dryden and Darwin and Shaw. But who knows what they missed? If publishers are such good judges of what people will like, how come 12 out of 13 of them rejected Harry Potter? It’s not even as though the British Library has limited itself to books up till now: as well as those unimportant newspapers at Colindale, their collections already include journals, maps, photographs and sound recordings. That they’re widening their scope to include e-books and the internet is nothing more than a recognition that those filters of the past have gone. We’re all publishers now. Anyone can put their thoughts on a blog, their book on Kindle. Very few people are Shakespeares or JK Rowlings – but those that are will definitely be found. And now, thanks to the British Library, they’ll never be lost.

Tags: British Library, Internet, Literature, publishing