Forecasting Quality - What Makes a Forecast Good?

[admin]

Interesting paper from a couple of HP researchers on the accuracy of social media based predictions.

[admin]

I couple of interesting articles about problems of complexity in the US intelligence community:

System gets so much data, dots can't be connected

July 20, 2010 Umar Farouk Abdulmutallab ... clues went unnoticed. Photo: Reuters


BECAUSE so much of the material is classified, examples of what goes on every day in Top Secret America can be hard to ferret out. But every so often, examples emerge.

Last year, after eight years of growth, the enterprise was at full throttle when word emerged that something was seriously amiss in Yemen. In response, the US President, Barack Obama, signed an order sending dozens of secret commandos to that country to target and kill the leaders of an al-Qaeda affiliate.

In Yemen, the commandos set up a joint operations centre packed with hard drives, forensic kits and communications gear. They exchanged thousands of intercepts, agent reports, photographic evidence and real-time video surveillance with dozens of US top secret organisations.


That was the system as it was intended. But when the information reached the National Counterterrorism Centre in Washington for analysis, it arrived buried within the 5000 pieces of general terrorist-related data that are reviewed each day.

Analysts had to switch from database to database, from hard drive to hard drive, from screen to screen, just to find what might be interesting to study further.

As military operations in Yemen intensified and the chatter about a possible terrorist strike increased, the intelligence agencies ramped up their effort. The flood of information into the centre became a torrent.

Somewhere in that deluge was vital data: partial names of someone in Yemen; a reference to a Nigerian radical who had gone to Yemen; a report of a father in Nigeria worried about a son who had become interested in radical teachings and had disappeared inside Yemen.

There were many clues to what would happen when a Nigerian named Umar Farouk Abdulmutallab left Yemen and eventually boarded a plane in Amsterdam bound for Detroit. But nobody put them together because, as officials would testify later, the system had grown so big that the lines of responsibility had become hopelessly blurred.
''Everyone had the dots to connect,'' the Director of National Intelligence, Dennis Blair, explained to members of Congress. ''But I hadn't made it clear exactly who had primary responsibility.''

And so Abdulmutallab was able to step aboard Northwest Airlines Flight 253. As it descended towards Detroit, he allegedly tried to ignite explosives hidden in his underwear.
It was not the very expensive, very large post-September 11 enterprise that prevented disaster; it was a passenger who saw what he was doing and tackled him.

The Washington Post





Blinded by information overload

DANA PRIEST AND WILLIAM ARKIN

July 20, 2010 Unwieldy beast ... security efforts. Photo: The Washington Post


WASHINGTON: When it comes to national security, all too often no expense is spared and few questions are asked. The result is an enterprise so massive that nobody in government has a full understanding of it.

In the US Department of Defence, where more than two-thirds of America's intelligence programs reside, only a handful of senior officials - called Super Users - have the ability to even know about all the department's activities.

One Super User recounted that for his initial briefing, he was escorted into a tiny, dark room, seated at a small table and told he couldn't take notes. Program after program began flashing on a screen, he said, until he yelled ''Stop!'' in frustration.

''I wasn't remembering any of it,'' he said.


Underscoring the seriousness of these issues are the conclusions of a retired Army lieutenant-general, John Vines, who was asked last year to review the method for tracking the Defence Department's most sensitive programs. Vines, who once commanded 145,000 troops in Iraq and is familiar with complex problems, was stunned by what he discovered.

''I'm not aware of any agency with the authority, responsibility or a process in place to co-ordinate all these inter-agency and commercial activities,'' he said. ''The complexity of this system defies description.''
The result, he added, is that it's impossible to tell whether the US is safer for all this spending and all these activities.

The US Secretary of Defence, Robert Gates, said he did not believe the system had become too big to manage but that getting precise data is sometimes difficult. Singling out the growth of intelligence units in the Defence Department, he said he intends to review those programs for waste. ''Nine years after 9/11, it makes a lot of sense to sort of take a look at this and say, 'OK, we've built tremendous capability, but do we have more than we need?' '' he said.

The CIA director, Leon Panetta, said he's begun mapping out a five-year plan for his agency because the levels of spending since 2001 are not sustainable.

The former director of national intelligence, retired Admiral Dennis Blair, said: ''After 9/11, when we decided to attack violent extremism, we did as we so often do in this country. The attitude was, if it's worth doing, it's probably worth overdoing.''

Every day across the US, 854,000 civil servants, military personnel and private contractors with top-secret security clearances are scanned into offices protected by electromagnetic locks, retinal cameras and fortified walls that eavesdropping equipment cannot penetrate.

The US intelligence budget is vast, publicly announced last year as $US75 billion ($86 million), which is 2½ times the size it was on September 10, 2001. But the figure doesn't include many military activities or domestic counterterrorism programs.

At least 20 per cent of the government organisations that exist to fend off terrorist threats were established or refashioned in the wake of the September 11, 2001, terrorist attacks. Many that existed before grew to historic proportions as the Bush administration and Congress gave agencies more money than they could responsibly spend.
It was phenomenal growth that began almost as soon as the September 11 attacks ended.

Nine days after the attacks, Congress committed $US40 billion beyond what was in the federal budget to fortify domestic defences and to launch a global offensive against al-Qaeda. It followed that up with an additional $US36.5 billion in 2002 and $US44 billion in 2003. That was only the beginning.

With the quick infusion of money, military and intelligence agencies multiplied. Twenty-four organisations were created by the end of 2001, including the Department of Homeland Security and the Foreign Terrorist Asset Tracking Task Force. In 2002, 37 more were created to track weapons of mass destruction, collect threat tips and co-ordinate the new focus on counterterrorism. That was followed the next year by 36 new organisations; and 26 after that; and 31 more; and 32 more; and 20 or more each in 2007, 2008 and 2009.

In all, at least 263 organisations have been created or reorganised as a response to the 2001 attacks.
With so many more employees, units and organisations, the lines of responsibility began to blur. To remedy this, at the recommendation of the bipartisan 9/11 Commission, the Bush administration and Congress decided to create an agency in 2004 with overarching responsibilities called the Office of the Director of National Intelligence to bring the colossal effort under control.

While that was the idea, Washington has its own ways.
The first problem was that the law passed by Congress did not give the director clear legal or budgetary authority over intelligence matters, which meant he wouldn't have power over the individual agencies he was supposed to control.
The second problem: even before the first director, John Negroponte, was on the job, the turf battles began. The Defence Department shifted billions of dollars out of one budget and into another so the ODNI could not touch it, according to two officials who watched the process. The CIA reclassified some of its most sensitive information at a higher level so National Counterterrorism Centre staff, part of the ODNI, would not be allowed to see it, former intelligence officers involved said.

Today, many officials who work in the intelligence agencies say they remain unclear about what the ODNI is in charge of.
Any improvements have been overtaken by volume, as the increased flow of data overwhelms the system's ability to analyse and use it. Every day, collection systems at the National Security Agency intercept and store 1.7 billion emails, phone calls and other types of communications. The NSA sorts a fraction of those intercepts into 70 separate databases. The same problem bedevils every other intelligence agency, none of which have enough analysts and translators for all this work.

Among the most important people inside these agencies are the low-paid employees carrying their lunches to save money. They are the analysts, the 20- and 30-year-olds making $US41,000 to $US65,000 a year, whose job is at the core of everything Top Secret America tries to do.
When hired, a typical analyst knows very little about the priority countries - Iraq, Iran, Afghanistan and Pakistan - and is not fluent in their languages. Still, the number of intelligence reports they produce is overwhelming, current and former intelligence officials who try to cull them every day say. The ODNI doesn't know exactly how many reports are issued each year, but in the process of trying to find out, the chief of analysis discovered 60 classified analytic websites still in operation that were supposed to have been closed for lack of usefulness. ''Like a zombie, it keeps on living'' is how one official describes the websites.

Beyond information overload, secrecy within the intelligence world hampers effectiveness in other ways, say defence and intelligence officers. For the Defence Department, the root of this problem goes back to an ultra-secret group of programs for which access is extremely limited and monitored by specially trained security officers.

These are called Special Access Programs - or SAPs - and the Pentagon's list of code names for them runs 300 pages. The intelligence community has hundreds more of its own and those hundreds have thousands of sub-programs with their own limits on the number of people authorised to know anything about them. All this means that very few people have a complete sense of what's going on.

''There's only one entity in the entire universe that has visibility on all SAPs - that's God,'' said James Clapper, Undersecretary of Defence for Intelligence and the Obama administration's nominee to be the next director of national intelligence.


The Washington Post
www.topsecretamerica.com