NSA's XKeyscore gives one-click real-time access to almost any internet activity

New revelations about NSA surveillance systems show that it was enough to fill in a short ‘justification’ form before gaining access to any of billions of emails, online chats, or site visit histories through a vast aggregation program called XKeyscore.

The structure of XKeyscore, leaked by the UK’s Guardian newspaper, is sourced from a classified internal presentation from 2008 and a more recent Unofficial User Guide, presumably obtained by Edward Snowden when he was a contractor for the National Security Agency in the past year.

It shows that XKeyscore – then located on 750 servers around 150 sites worldwide – is a vast collection and storage program that served as the entry point for most information that was collected by the NSA. The Guardian claims that in one 30-day period in 2012 the program acquired 41 billion records.

The information is not just metadata – depersonalized analytical usage statistics that allow spies to spot patterns – but includes almost all types of personal information. Using any piece of personal data on a subject – an email address, or the IP address of a computer – an agent could look up all online user activities, such as Google map searches, website visits, documents sent through the internet or online conversations. The service operates both, in real time, and using a database of recently stored information.

All that appears to have been necessary to log into the system is to fill in a compulsory line on a form that gave a reason for why a certain person needed to be investigated. The form was not automatically scanned by the system or a supervisor, and did not require a US legal warrant, as long as the person whose name was typed in was a foreigner (even if his interactions were with a US citizen).

The slides appear to vindicate security specialist Edward Snowden’s claims made during the original video he recorded in Hong Kong last month.

"I, sitting at my desk, could wiretap anyone, from you or your accountant, to a federal judge or even the president, if I had a personal email," he alleged then.

Mike Rogers, the Republican chairman of the House intelligence committee later said that Snowden was “lying”. 

The NSA has issued a statement to the Guardian, which does not now appear to deny the capabilities of XKeyscore, but merely to defend its use.

"NSA's activities are focused and specifically deployed against – and only against – legitimate foreign intelligence targets in response to requirements that our leaders need for information necessary to protect our nation and its interests,” said the NSA.

User-friendly global spying

What strikes an outsider as particularly astonishing is the integration, user-friendliness and functional variety provided by the XKeyscore suite, as seen in the available slides (and its capabilities may have expanded significantly in the intervening years). 

The presentation makes XKeyscore appear little more complicated than a search engine – but one that can find almost anything. After the heading “What can you do with XKeyscore?” the slides outline ways in which the system can fish useful information out of the morass of internet noise.

The presentation gives examples of possible questions that can be answered with the aid of the search abilities provided:

• My target speaks German, but is in Pakistan, how can I find him?
  
• I have a jihadist document that has been passed around through numerous people, who wrote this and where were they?
  
• My target uses Google maps to scope target locations – can I use this information to determine his email address?
  

The document then goes on to show that even basic facts do not need to be known to root out potential terrorists, with the system offering a nuanced analysis of “anomalous” behavior that makes a suspect stand out.

Once information has been fished out, it can be categorized using a selection of plug-ins that index user activity, including all the emails, phone numbers and sites that suspect has come into contact with.

While XKeyscore collects so much data that it has to be wiped within days, anything useful be stored on a smaller, though still sizeable database called Pinwale that keeps records for up to five years.

The presentation boasts that 300 terrorists were caught with XKeyscore in the years leading up to 2008.

While Fisa regulations mean that US citizens can only be clandestinely observed with a warrant, the search does not always identify the nationality of those it spies on, and in any case, having a foreign contact is sufficient to expand the search to a large proportion of US citizens.

And while they are not the stated target of XKeyscore, communications between ordinary Americans, whether those pass through domestic and international servers, are inevitably caught in the net.

Technically, a person with access to the system could just as easily spy on an American within a few clicks, as they could on a foreigner, though this would be illegal.

In last month’s Guardian interview Snowden insisted that there was nothing stopping this from happening.

"It's very rare to be questioned on our searches, and even when we are, it's usually along the lines of 'let's bulk up the justification'," he claimed. While NSA officials have admitted "a number of compliance problems", which they say were not made “in bad faith”, they say the latest revelations are not proof that the system is illegal vulnerable to abuse.

"Allegations of widespread, unchecked analyst access to NSA collection data are simply not true. Access to XKeyscore, as well as all of NSA's analytic tools, is limited to only those personnel who require access for their assigned tasks … In addition, there are multiple technical, manual and supervisory checks and balances within the system to prevent deliberate misuse from occurring," said the NSA statement.

"These types of programs allow us to collect the information that enables us to perform our missions successfully – to defend the nation and to protect US and allied troops abroad."