The Open Source Intelligence (OSINT) industry has grown rapidly over the past decade. Private companies and state agencies are now collecting and analysing “publicly available” data on a vast scale.

This article by Ben Hayes, published in the Statewatch Journal last year, looks at the evolution, theory and practice of OSINT; its use by police and security agencies; the rapidly developing OSINT industry; the blurring of the boundaries between OSINT and covert surveillance; and the embrace of OSINT by the EU.

The full article is available here (pdf). It concludes:

Writing recently in the Guardian, Professor John Naughton observed:

[T]he internet is the nearest thing to a perfect surveillance machine the world has ever seen. Everything you do on the net is logged – every email you send, every website you visit, every file you download, every search you conduct is recorded and filed somewhere, either on the servers of your internet service provider or of the cloud services that you access. As a tool for a totalitarian government interested in the behaviour, social activities and thought-process of its subjects, the internet is just about perfect.

The present threat to civil liberties, however, comes neither from the internet nor totalitarian governments, but from a neo-McCarthyite witchhunt for “terrorists” and “radicals”, and a private security industry bent on developing the “perfect surveillance” tools to find them. For all the concern about Facebook’s privacy policy, that company is no more responsible for its users’ wishes to ‘broadcast themselves’ than travel agents are for tourism. Of course Facebook should offer maximum privacy protection for its users, but those of us concerned with freedom and democracy need to see the bigger picture in terms of who is doing the watching, how, and why. We must then develop the tools and communities needed to bring them under democratic control.

In July the Washington Post began its ‘Top Secret America’ series, examining the rapid growth of the USA’s heavily privatised intelligence establishment. Investigative journalism at its best, the series and its findings  should prompt those of us in Europe who care about such matters to start making the same kind of inquiries about our own security-corporate nexus.

The most alarming findings (summarised here) include:

* 1,931 intelligence contracting firms doing work classified as “top secret” for 1,271 government organisations at over 10,000 sites around the USA; 533 of these  firms were founded after the ‘9/11’ attacks.

* Contractors make up nearly 30 percent of the workforce of America’s intelligence agencies. At the Department of Homeland Security the ratio of contractors to permanent staff is 50-50. The Washington Post estimates that of 854,000 people with top-secret clearances, 265,000 are contractors.

* 18 government organisations contract 37 private companies to conduct psychological operations;

* 16 government organisations use 50 companies for “special military operations” (e.g., SWAT teams and unconventional warfare);

* 14 government organisations contract 50 companies for top-secret conventional military operations;

* 32 government organisations employ 36 different companies for counter-drug operations.

* The National Security Agency intercepts 1.7 billion e-mails, phone calls and other types of communications every day and divides some percentage of these between 70 different databases.

* At least 263 intelligence organisations have been created or reorganized in response to 9/11.

Why does any of this matter? As the authors of Top Secret America point out: “What started as a temporary fix in response to the terrorist attacks has turned into a dependency that calls into question whether the federal workforce includes too many people obligated to shareholders rather than the public interest — and whether the government is still in control of its most sensitive activities”.

See Washington Post TSA series. Part one: “A hidden world, growing beyond control“, part two: “The secrets next door“, part three: “National Security Inc.” and Top Secret America blog.