Data Mining

[Steve Bennett]

Thought we could use a thread on this topic. Am I wrong?

[Steve Bennett]

Data mining Know-alls Part 1

Sep 25th 2008
From The Economist print edition

Electronic snooping by the state may safeguard liberty—and also threaten it

Kobal Collection



IF A Muslim chemistry graduate takes an ill-paid job at a farm-supplies store what does it signify? Is he just earning extra cash, or getting close to a supply of potassium nitrate (used in fertiliser, and explosives)? What if apparent strangers with Arabic names have wired him money? What if he has taken air flights with one of those men, with separate reservations and different seats, paid in cash? What if his credit-card records show purchases of gadgets such as timing devices?

If the authorities can and do collect such bits of data, piecing them together offers the tantalising prospect of foiling terrorist conspiracies. It also raises the spectre of criminalising or constraining innocent people’s eccentric but legal behaviour.

In November 2002 news reports revealed the existence of a big, secret Pentagon programme called Total Information Awareness. This aimed to identify suspicious patterns of behaviour by “data mining” (also known as “pattern recognition”): computer-driven searches of large quantities of electronic information. After a public outcry it was dubbed, perhaps more palatably, Terrorism Information Awareness. But protests continued, and in September 2003 Congress blocked its funding.

That, many people may have assumed, was that. But six of TIA’s seven components survived as secret stand-alone projects with classified funding. A report in February by America’s Department of Homeland Security named three programmes it operates to sniff out suspicious patterns in the transport of goods. Similar projects have mushroomed in, among other countries, Britain, China, France, Germany and Israel.

Civil-liberties defenders are trying hard to stop data-mining becoming a routine tool for the FBI to spy on ordinary Americans. They say that the administration is racing in its final months to formalise in law programmes that have run solely under authorisation from the White House that bypasses Congress. One pending change would authorise more intelligence sharing between federal and local officials. In a federal court filing made public on September 20th, America’s attorney-general, Michael Mukasey, sought legal immunity for telecoms firms which have provided details on international phone calls. What happens in practice, and what the law permits, is a hot and unresolved issue.
Last month, after a briefing by the Department of Justice about a secret data-mining plan for the FBI, a group of American lawmakers wrote to Mr Mukasey complaining that the plan would allow the FBI to spy on Americans “without any basis for suspicion”. The proposed project could be made public in coming weeks.

No similar pan-European data-mining programme is operating, at least to public knowledge. Yet under an agreement signed in July last year airlines flying from the European Union to America have had to provide the authorities there with reservations data, as well as information obtained by airport-security screeners. This can include passengers’ race, religion, occupation, relatives, hotel reservations and credit card details. Internet service providers and telecoms firms in the EU must now keep for up to two years, though not automatically hand over, data on websites visited and phone calls made and received (but not the content of conversations).


Fast company

FAST, a Norwegian company bought by Microsoft this year for $1.3 billion, collects data from more than 300 sources (including the web) for national data-mining programmes in a dozen countries in Asia, Europe and North America. In April British members of Parliament learned that almost a year earlier the home secretary, Jacqui Smith, had secretly authorised the transfer of licence-plate data recorded by roadside cameras to foreign intelligence agencies. In June the Swedish Parliament voted into law a data-mining programme strongly backed by the defence ministry. From January 1st it will provide sweeping powers to monitor international electronic messages and telephone traffic.
The staggering, and fast-growing, information-crunching capabilities of data-mining technology broaden the definition of what is considered suspicious. In June America’s Departments of Justice and Homeland Security and a grouping of American police chiefs released the “Suspicious Activity Report—Support and Implementation Project”. Inspired in part by the approach of the Los Angeles Police Department, it urges police to question people who, among other things, use binoculars, count footsteps, take notes, draw diagrams, change appearance, speak with security staff, and photograph objects “with no apparent aesthetic value”.

Companies, and especially credit-reporting firms, generally enjoy more latitude than government bodies do in making personal information available to third parties. They find intelligence agencies are eager clients. Chris Westphal, head of Visual Analytics, a firm in Poolesville, Maryland that operates data-mining software for security and intelligence agencies, says the data provided by such firms is “very significant”. Narayanan Kulathuramaiyer, an expert in data mining at UNIMAS, a Malaysian university, says companies are selling database access to intelligence and law-enforcement agencies “at a level you would not even imagine”.

Legal challenges to governments’ use of personal information held by companies have reached high courts in many countries, including America’s Supreme Court. Rulings, however, have for the most part frustrated privacy advocates. Suzanne Spaulding, a former legal adviser to the Senate and House intelligence committees, says improvements in data-mining technology have enabled intelligence agencies to milk favourable court rulings in ways that exceed judicial intent. For example, such cases typically concern permission to use data from a single source, such as a phone company’s billing records. When different databases are mined simultaneously, the value of information increases exponentially.

Spies are increasingly snooping on private internet use. Katharina von Knop, a data-mining expert at the University of German Federal Armed Forces in Munich, says many systems remotely analyse the content of web pages people visit. A man who has travelled to, say, Peshawar, a stronghold of Islamist extremism in Pakistan, is considered more dangerous if he also reads the blog of an extremist Muslim cleric. If the cleric lives in Peshawar, the man’s suspicion score rises further. Data-mining software develops profiles by taking into account all web pages visited by a computer user; if a suspect visits a stamp-collecting website, the suspicion score is lowered.

Such profiling increasingly relies on “sentiment analysis”. Hsinchun Chen, head of the Artificial Intelligence Lab at the University of Arizona says this technique, which he performs for American and international intelligence agencies, is an emerging and booming field. The goal is to identify changes in the behaviour and language of internet users that could indicate that angry young men are becoming potential suicide-bombers. For example, a person who exhibits curiosity by visiting many Islamist websites and asking numerous questions in online forums might be flagged by sentiment-analysis software if he shows signs of resentment and eventually turns to “radicalising” others by, say, justifying violence and providing links to militant videos. Mr Chen says intelligence agencies in the United States, Canada, China, Germany, Israel, Singapore and Taiwan are customers for this technique.

[Steve Bennett]

Data mining Know-alls Part 2

Sep 25th 2008
From The Economist print edition

Electronic snooping by the state may safeguard liberty—and also threaten it

Does it work?
Donald Tighe, vice-president for public affairs at In-Q-Tel, a non-profit investment outfit that helps the CIA stay abreast of advances in computing, says that data mining is now so powerful it has become “essential to our national security”. But campaigners for privacy have many worries. One fear, prevalent in Britain after incidents in which officials lost huge quantities of confidential personal information, is that the state may be even more careless with data than private firms are. Another is that innocents are flagged for further investigation or added to “watch-lists” that may impede air travel, banking and gaining jobs in places where radioactive materials are used, such as hospitals. The American Civil Liberties Union (ACLU), a lobby, says the list maintained by the Terrorist Screening Centre at the FBI now has more than 900,000 names, with 20,000 more every month. Being removed is tricky.

Data-mining may be bad for national security as well as for civil liberties. The software is often modelled on the fraud-detection applications used by financial institutions. But terrorism is much rarer. So spotting conditions that may precede attacks is harder. Mike German, a former FBI agent who now advises the ACLU, says intelligence agencies too readily believe in the “snake oil” of total information awareness, which drains effort from more useful activities such as using informers and infiltrators.

AFP


Explosive data

Abdul Bakier, a former official in Jordan’s General Intelligence Department, says that tips to foil data-mining systems are discussed at length on some extremist online forums. Tricks such as calling phone-sex hotlines can help make a profile less suspicious. “The new generation of al-Qaeda is practising all that,” he says.

Last year two pattern-detection programmes, ADVISE and TALON, run respectively by America’s Department of Homeland Security and the Pentagon, were shut down following privacy concerns and irregularities. Privacy advocates, however, say that other programmes continue—and many are operated, with minimal oversight, by the National Security Agency. The NSA insists that it does keep Congress informed. It also vigorously defends data mining, saying that if today’s systems were in place before the terrorist attacks of September 11th 2001, some of the hijackers would have been identified.

In July, after fierce debate, Congress imposed new limitations on government wiretapping when it renewed the expiring Foreign Intelligence Surveillance Act (FISA) sought by President George Bush after September 11th. The main law governing data mining, this has provided the administration with broad and unprecedented electronic-spying powers. But civil-liberties lobbies such as Amnesty International and Human Rights Watch say the renewed, restricted law leaves largely untouched far-reaching secret “black” programmes, run by the NSA, which crunch data on great numbers of people, including millions of Americans. Much of that is personal financial information collected by the Treasury.

Mr Bush says that FISA helps protect citizens’ liberties “while maintaining the vital flow of intelligence”. Several hours after the president signed the bill into law, the ACLU filed a federal lawsuit, on the grounds that the executive branch’s expanded wiretapping powers violated the constitution.
In 2001 American-led forces routed the Taliban in Afghanistan, destroying al-Qaeda training camps there. Berndt Thamm, who advises Germany’s armed forces on terrorism, says that in retreat the Islamists left valuable clues about their online communications and electronic plotting. It is in following up these leads that data mining and pattern analysis can, and should, be used. Such techniques, says Mr Thamm, are “the only answer” to jihadist extremists. That is the argument which the strenuous objections of civil libertarians need to overcome.

[JohnG]

Here's some useful insight on the value of data mining by the US government. Not very inspiring.

I found the article here.

Data mining for fool's gold by fools

Category: Civil liberties

Posted on: October 11, 2008 7:49 AM, by revere

The US government response to the bogus war on terrorism is a mixture of the stupid, the super stupid and the evil. Taking off our shoes at the airport isn't evil but it is probably super stupid. So is the 3 oz. liquid ban. On reflection, maybe there are only two categories: super stupid and evil. In the evil category are the pseudoscientific "datamining" and "behavior detection" scams. Data mining is my favorite, since I use one of its techniques in my research work (Association Rule Mining) and I'm familiar with a number of other techniques. Familiar enough to not be surprised that when the National Research Council (a component of the National Academies of Science) looked at it of dubious scientific value:


The most extensive government report to date on whether terrorists can be identified through data mining has yielded an important conclusion: It doesn't really work.
A National Research Council report, years in the making and scheduled to be released Tuesday, concludes that automated identification of terrorists through data mining or any other mechanism "is neither feasible as an objective nor desirable as a goal of technology development efforts."

Inevitable false positives will result in "ordinary, law-abiding citizens and businesses" being incorrectly flagged as suspects.

[snip]

They admit that far more Americans live their lives online, using everything from VoIP phones to Facebook to RFID tags in automobiles, than a decade ago, and the databases created by those activities are tempting targets for federal agencies. And they draw a distinction between subject-based data mining (starting with one individual and looking for connections) compared with pattern-based data mining (looking for anomalous activities that could show illegal activities).

But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so." (CNET)


The NRC Committee, which included Charles Vest, former President of MIT and William Perry, a professor at Stanford University, former Secretary of Defense, were not any kinder to "behavior detection":

Behavior detection, used by the Transportation Security Administration and some police departments to isolate possible criminals from crowds, likewise falls short of meeting scientific standards, the group said.
"There is not a consensus within the relevant scientific community" that behavior detection is "ready for use ... given the present state of the science," the group said.

The group cautioned that "inappropriate ... responses to the terrorist threat ... can do more damage to the fabric of society than terrorists would be likely to do."

[snip]

The council said terrorism is a threat to American society, but "inappropriate or disproportionate responses to the terrorist threat also pose serious dangers to society."

"History demonstrates that measures taken in the name of improving national security, especially in response to new threats or crises, have often proven to be both ineffective and offensive to the nation's values and traditions of liberty and justice," the report says.(CNN)


Data mining to detect terrorists is the phrenology of computer science. It sounds and looks scientific. But it isn't. The same with behavior detection. Plausible sounding but with no scientific basis for detecting deception by terrorists. If you've ever used these tools, you'd know how limited they are.

Unless you were in the Bush administration. Then you wouldn't care.

ShareThisFind more posts in: TechnologyPolitics

Comments
In the early 1990's, the Federal Government spent twenty million dollars on the "Stargate Project," largely in an effort to "scientifically" study the feasibility of "remote viewing;" the project was dumped in 1995. These assholes never learn.

Posted by: Dylan | October 11, 2008 12:07 PM

Not to worry, we'll just fire up the law and order noise machine...

"Did you know that so called "false positives" and their extreme liberal allies are standing in the way of the sophisticated intelligence tools needed to save your children from the terrorists?"

"In our post-9/11 world, we cannot afford to tolerate the deviant behavior of the "false" positives who conceal the terrorists in our midst."

A couple rounds of that, establish the notion that being a false positive = obstructing justice, and no more false positives, just another reason to add people to one of the nebulous lists of enemies.

Posted by: phisrow | October 11, 2008 2:51 PM

The fun part is that driftnet methods like "data mining", and, of course, promiscuous pervasive monitoring, simply guarantee paralysis when the notional threat turns real.

You don't have enough analysts to keep up with the data, because there is simply too much there to analyze. The result is that your actual enforcement gets bogged down in pursuit of political and personal vendettas.

Meanwhile, the attackers carve you up at leisure.

Adherence to old, outdated concepts like "probable cause" would strongly mitigate this risk.

But nobody in authority seems to care. What the hell, let's just hurl ourselves off history's cliff faster.

Glad I'm in my late 50s. With any luck, I may be safely dead before I get to see how this sort of mass insanity will work out, firsthand. What I can extrapolate is more than bad enough.

Posted by: Charles | October 11, 2008 6:40 PM

Dylan -

Earlier this decade the Air Force spent $20M to determine whether teleportation was possible. The money went to either Mitre or Booz Allen. If you actually read it (I did, ~3 years ago), it sounds like it was written by Bugs Bunny with a side of Roger Ramjet. If I were either company I would've been ashamed to publish it. But hey, $20M is $20M. If you're curious, I believe you can google it. Just use Teleportation study & US Gov.

Phisrow - and if you think it can't happen to you, it can. I was tagged in the mid 90's when I made the mistake of flying one airline one way and a second one on the return leg of a business trip. I was pulled out for questioning - and I work for the goddamn government as a contractor. Other strikes against me? I was flying alone, I had "long hair, beard and an earring" and my complexion was too dark (I am of eastern European descent, but I have been taken for arabic). I am grateful - if it can be said - that this occurred before 9/11.

Posted by: Pineyman | October 12, 2008 8:42 PM

There's a wonderful textbook lesson in the False Positive Paradox and Bayes' Theorem lurking in here.

Posted by: Blake Stacey | October 13, 2008 1:03 AM


Dylan, your lack of objectivity is showing. The remote viewing studies produced a rescued fighter pilot who had been held hostage, intel on a new type of Soviet submarine, and the arrests of suspects in a kidnapping case. But you can believe whatever you want. Even if it's wrong. And no, I'm not going to let myself get dragged into a thrash on this subject.

As for data mining and the risks to national security: the major risk, aside from the risk of tyranny, is that of swamping the relevant agencies so they can't do their real job.

Consider NSA's translation backlog that resulted in missing the key intercept about 9/11. Consider it in light of the fact that Bush apparently tasked NSA with broad-spectrum domestic collection in the months preceding the attack, even as he (Bush) was downplaying terrorism entirely. Yes, I'm asserting that Bush putting NSA on a wild goose chase swamped the agency's translator resources and led to the delay in processing the Al Qaeda intercept.

Next, consider this instance. FBI asked NSA for raw intercepts (something that is not ordinarily done: NSA releases digests and first-level abstracts, not the raw stuff). NSA opened the floodgates. FBI said "stop! stop!, we can't deal with all of this stuff!" Useful lesson. Military resources are not appropriate tools for domestic law enforcement.

Fortunately we're about to have a smart guy in office. All the good folks who worked for the various agencies that Bush screwed up with his King Midas of Poo touch, will be rejoicing bigtime.

Posted by: g347 | October 14, 2008 8:33 AM

[fgeorge]

Computerworld has an article equally scathing:

Data Mining for Terrorists Is Futile
Report, commissioned in part by the DHS, also warns of potential privacy problems.

Jaikumar Vijayan, Computerworld
Monday, October 13, 2008

The kind of pattern-seeking data mining and behavioral surveillance technologies that are being used by several federal agencies to identify potential terrorists are far too unreliable to be of any real value, according to a report issued by the National Research Council.

The continued and unchecked use of such tools also poses potential privacy problems for individuals, the NRC said in its 376-page report, which was prepared at the request of the U.S. Department of Homeland Security (DHS) and the National Science Foundation.

In light of the findings, the 21-member committee that conducted the study is recommending that agencies using or planning to adopt such tools for counterterrorism purposes should first be required to thoroughly evaluate their effectiveness, lawfulness and impact on privacy . The committee also called on Congress to consider revising national privacy laws in order to ensure better protection for U.S. residents.

The NRC and the National Academy of Sciences, the National Institute of Medicine and the National Academy of Engineering make up what is known as the National Academies, which advise the government on science and technology issues.

The findings detailed in the NRC's report hammer home concerns that have been voiced by many privacy advocates, said Marc Rotenberg, executive director of the Electronic Privacy Information Center (EPIC) in Washington.

"What the [NRC] has concluded is that there needs to be much more effective oversight of these programs," Rotenberg said. "It's a very timely and significant report." He noted that despite the privacy concerns, the government has gone ahead with many data mining programs in the name of countering terrorism. But the NRC's report raises questions about whether such programs really work, Rotenberg said.

As of January 2007, there were nearly 200 data mining programs planned or already operating throughout the federal government. Among them were the Automated Targeting System at the DHS for assigning "terror scores" to U.S. citizens and the Transportation Security Administration's Secure Flight program for analyzing data about airline passengers. The FBI has several data mining initiatives underway, including some that target terrorists.

One of the most controversial programs was the Total Information Awareness (TIA) initiative that was quietly launched in 2002 by the Defense Advanced Research Projects Agency but then abandoned in 2003 after Congress stopped funding for it following a public outcry.

William Perry, co-chair of the NRC committee that wrote the new report, said in a prepared statement that technology should be used as needed to combat terrorism. "However, the threat does not justify government activities that violate the law, or fundamental changes in the level of privacy protection to which Americans are entitled," he added.

The NRC committee didn't look specifically at any counterterrorism-related data mining initiatives, nor did it conduct any direct evaluations of behavioral surveillance tools being used by agencies. Instead, the report is based on a generalized study of the effectiveness of such technologies in identifying potential terrorists.

What the report highlights are the severe limitations of automated data mining techniques for counterterrorism purposes and their potential privacy impacts, said committee member Fred Cate, who is the director of the Center for Applied Cybersecurity Research at Indiana University.

Needle in a Haystack?

Automated data mining tools typically work by searching through mountains of data in large databases for unusual patterns of activity, which are then used to predict future behavior. The tools have proved to be useful for commercial applications such as detecting payment card fraud and predicting purchasing trends, Cate said.

"We can look at 50,000 people buying television sets and know that many of them are going to be buying a DVD at the same time," Cate said. But using the same techniques to try to identify a potential terrorist is futile because there simply isn't enough historical data upon which to base any predictions, he claimed, adding that there is little information available about patterns that could reliably point to terrorist activity.

On the consumer side, "you have millions of examples of the target data you want to emulate, so you know certain patterns look like fraud," Cate said. "With terrorists, we fortunately don't have too many examples."

And unlike shoppers, terrorists are likely to make deliberate attempts to hide their activities, making it even harder to pick them out using an automated pattern-matching program, according to Cate. As a result, data mining tools generate an unacceptably high rate of false positives when used in counterterrorism applications, he said.

Such tools can prove useful in situations in which they are given specific pieces of information -- such as a suspect's name -- and asked to look for other data, such as purchases made or places visited by the suspect. That could help show if there is any basis for further action, Cate said.

There are similar problems with many behavioral surveillance tools, Cate contended. Such tools are supposed to help counterterrorism efforts by measuring physiological states, including facial expressions, body temperatures and body language, in order to predict terrorist activity. But there is no evidence that the tools work at all, Cate said. He recommended that at the most, they should be used for preliminary screening purposes only.

On top of the technical concerns are the potential privacy implications of data mining and behavioral surveillance, the NRC said in its report. The pieces of information that are used to mine data often are compiled from numerous sources and databases, some of which could be outdated or contain poor-quality data, according to Cate.

As a result, he said, data mining is error-prone -- and the high rate of false positives could lead to unnecessary intrusions into personal privacy. Going forward, Cate said, there need to be safeguards to ensure that any data being collected for counterterrorism uses is fresh, compiled from reliable sources and within the scope of the inquiry being conducted.

[Jane B]

Perhaps a more benevolent use of mining?!:

By Angelica Mari
10 November 2008

The World Wildlife Fund (WWF) is extending the use of business intelligence (BI) tools and implementing more web innovation to better use data about charitable donors.
A period of ineffective information management in the early 2000s prompted a rollout of business analytics software provided by SAS as the charity needed more sophisticated tools to improve revenue.

Use of BI in direct marketing has already generated a 25 per cent return on investment to WWF and more than US$1bn of donations for its latest campaign. Now the charity wants to integrate the systems into the GPS-based applications used by its research division to measure progress in achieving environmental conservation.

“We need to see what else is beyond direct marketing for BI,” said Terry Macko, chief marketing officer at WWF. “We are investigating ways to give that functionality to our scientists to support their work via improved planning and construction of advanced environmental patterns.”

To attract more supporters, WWF’s upcoming projects may also include further use of Web 2.0 tools and neural network systems.

“We are not sure of the extent to which the downturn will affect fundraising but it is important that we have a steady stream of donations. And to get them, we need to know who to ask, as well as when and how much we should ask,” said Macko.

“BI is not something you just turn on and off; it needs to be embedded.”

[Steve Bennett]

Check out my blog entry on social mining here. I take a look at some interesting emerging services and experiments that might just leave you asking for more.

Steve

[admin]

Wednesday, April 29, 2009 - Electronics and ICT Association

Innovative simulation software that enables Australian organisations to manage water and carbon resources more efficiently has won a prestigious double prize in the South Australian iAwards competition. The SimulAIt program, developed by Adelaide-based Intelligent Software Development, collected awards in both the financial application category and the sustainability and green IT award.

SimulAIt is a powerful simulation application that is used to create virtual cities to model and test products, policies or infrastructure, especially in high impact areas such as water management and climate change. Used as a predictive analysis support tool for financial and environmental sustainability, the software extracts behavioural business intelligence by using behavioural simulation to test ideas in a virtual replica of the real world where mistakes are inexpensive and involve minimal risk.

Of the eight companies that won iAwards on Monday, Intelligent Software Development was the only one to win awards in two categories. Winners will represent SA at national finals in Melbourne on May 27.

Intelligent Software’s chief executive officer Dr. Don Perugini said SimulAIt was a powerful decision support tool used by business and Government for financial and environmental sustainability. “SimulAIt uses advanced artificial intelligence to bring data to life,” he explained.

“By crystalising a vast quantity of complex data and human behaviours, SimulAIt enables you to create virtual replicas of complete cities. This allows strategic planners to test the effectiveness of new products, policies or infrastructure before they are implemented, thereby gaining insight into the future impact of their strategic decisions.”

Founded in 2006, Intelligent Software is an Adelaide-based company that uses state-of-the-art defence technologies to deliver cost-effective customised software solutions and consulting using its in-house developed SimulAIt platform. Intelligent Software’s founders, Dr. Don Perugini and Dr. Michelle Perugini, come from defence and health backgrounds. Drawing parallels between complex problems in defence and health, and the commercial arena, Intelligent Software’s founders identified commercial opportunities and an urgent market need in financial and environmental sustainability, which they could address with SimulAIt.

“Intelligent Software’s products are testimony to the wealth of business creativity harnessed in the South Australian electronics and ICT industry,” commented EICTA CEO, Steve Ad****.

‘Many people aren’t aware that the electronics and ICT industry generates $7.5 billion in revenue for South Australia or than it includes more than 1,600 companies,’ added Mr Ad****.
About Electronics and ICT Association
The Electronics and ICT Association (EICTA) is the peak body in South Australia representing the $7.5 billion electronics and ICT industry. EICTA is a non-profit, member based organisation that delivers tailored networking events, professional development courses, business advice and lobbying opportunities.More

About Intelligent Software Development

Intelligent Software Development Pty Ltd has drawn on expertise from the defence industry to use Artificial Intelligence technologies to create realistic virtual models of cities for strategic planning and forecasting purposes. As with the popular SimCity computer game, Intelligent Software’s AI approach models elements of a city, such as households and their demographic characteristics. This simulation can be used to assist with strategic planning and demand forecasting by running “what-if” scenarios and analysing the impact that proposed strategic decisions, infrastructure and policy will have on the population’s behaviour, including demand, spending and consumption.

[Doug Heywood]

Powerful tool to scour document metadata updated
FOCA examines the metadata in documents, which reveals a wealth of information about networks and users

Jeremy Kirk (IDG News Service) 14/09/2009 07:43:00

A Spanish company has released an upgraded version of a powerful software application that can be used to perform intelligence gathering on a company's Web site and network.

The application, called FOCA (Fingerprinting Organizations with Collected Archives), will download all documents that have been posted on a Web site and extract the metadata, or the information generated about the document itself. It often reveals who created the document, e-mail address, internal IP (Internet Protocol) addresses and much more.

It's possible to see the number of computers in an office, which ones are connected to printers and get a good idea for how a network is structured. FOCA is currently being used by government security agencies, penetration testers and even hackers, said Chema Alonso, a security researcher with Informatica64, a Spanish consultancy.

FOCA can also identify OS versions and application versions, making it possible to see if a particular computer or user has up-to-date patches. That information is of particular use to hackers, who could then do a spear phishing attack, where a specific user is targeted over e-mail with an attachment that contains malicious software.

The latest version of FOCA, release candidate 1, has new abilities to discover subdomains or other alternative domains within a company network. If FOCA discovers an internal domain name, FOCA will look for other servers that may be on the internal network, Alonso said.

"The idea is to dig and dig and dig in the metadata to obtain more information," he said.

FOCA has also been equipped to query Robtex, a Web site that offers services such as DNS (Domain Name System) lookups and IP information. FOCA will send an IP address to Robtex and then get a list of other servers on the network, Alonso said.

"That gives us more power to analyze the internal network," Alonso said.

Within the next couple months, Alonso said he expects to release another version of FOCA that will improve the application's ability to identify OSes based on the documents it analyzes. FOCA can do that somewhat now, but not for every kind of document on every kind of operating system.

FOCA is a free application but isn't open source. Alonso developed FOCA as part of his doctorate's degree work, and now Informatica64 manages the project.

The company sells a product called Metashield Protector, which clears the metadata from a document before it leaves the company network. Metashield Protector does this by modifying the document on the fly. The original metadata is preserved, though, on the company network, Alonso said.

[admin]

Downloaded some new white papers I thought worth sharing:

• TDWI Checklist Report: Data Requirements for Advanced Analytics
• TDWI Checklist Report: Self-Service BI
• TDWI Checklist Report: Data Synchronization