International News On Open Source

[Steve Bennett]

Let's put the most interesting open source news from overseas here.

[Steve Bennett]

TechWorld

Qizmt is a distributed computation framework based on the MapReduce programming model for processing large data sets

Juan Carlos Perez (IDG News Service) 16/09/2009 05:36:00

MySpace on Tuesday will release as open source a technology called Qizmt that it developed in-house to mine and crunch massive amounts of data and generate friend recommendations in its social-networking site.

Qizmt is a distributed computation framework based on the MapReduce programming model for processing large data sets in processor clusters.

The company hopes that the developer community will be able to benefit from Qizmt, as well as enhance and extend it, said Hala Al-Adwan, MySpace's vice president of data.

In use at MySpace for about a year, Qizmt is used to power the site's "People You May Know" feature, which makes friend recommendations to users.

MySpace hopes to extend its use to other types of recommendations, like suggestions for movies or books, and for products to buy, said Al-Adwan, who will demo Qizmt on Tuesday at Computerworld's Business Intelligence Perspectives conference in Chicago with MySpace COO Mike Jones.

"We want to look at our entire data set and start exploring the realm of social analytics. We're really trying to understand the nature of our users' behavior and the relationship with each other and with content as well," Al-Adwan said.

Before creating Qizmt in-house, MySpace looked at available options for doing large scale data processing in near real time, but didn't find anything it considered appropriate to its needs and to its .Net-based development platform, she said. Qizmt was developed using C# .Net for Windows.

In June, MySpace released as open source another internally developed tool called MSFast for tracking the performance of Web sites.

[admin]

Data.gov delivers, but fed to go further by hardwiring agencies for open government

Patrick Thibodeau (Computerworld (US)) 10 September, 2009 12:47:00

The White House's data.gov effort is spurring the creation of some entertaining and informative applications from private developers.

Take for instance, DataMasher, which allows you to take two sets of data and merge them.

Popular mashups include: total political contributions to Democrats divided by total political contributions to Republicans by state, and another that maps the "most reproductive states," to highlight which state has the most babies per capita (answer: Utah).

Another application, Govpulse, reorganizes the Federal Register, a daily publication that is a thick and painful read, into something that is now sorted and searchable by department and geography.

And then there is This We Know, which enables users to create an instant snapshot of key data about a particular area, such as their hometown, to determine the number of factories, crimes, demographics, and even people diagnosed with cancer.

Those sites, all useful and fun to work with, were the top three winners this week in the Apps for America contest held by the Sunlight Foundation, an organization that urges transparency in government. About 50 applications were entered in the contest.

An intent of the contest is to draw attention to data.gov, the new effort by President Barack Obama's administration to move government data into machine readable formats that can then be repurposed by developers.

The creation of applications from government data has been compared to the model used by Apple and its iPhone, which has enabled development of thousands of applications specifically for it.

"Government truly is a platform," said Clay Johnson, the director of Sunlight Labs at a Gov 2.0 Summit , held today. The role of government is that of "a wholesaler of data," he said.

This model of interactive, data-rich information offers some interesting possibilities for both users and government.

Among the services operated in the U.K by the non-profit mySociety group is called FixMyStreet , which enables users to report on potholes, or log complaints about an overgrown pedestrian pathway, for example.

What may make FixMyStreet different and interesting is that all the information is transparent and hyperloca.

Viewers can scan the complaints, as well the status of complaint. That level of transparency may also help a government agency get the energy and money it needs to address problems, said Tom Steinberg, who heads mySociety and was at the conference.

The federal efforts to turn its vast troves of data into something that is useful will likely take years, but federal CTO Aneesh Chopra said that government IT officials are only weeks away from issuing a directive to federal agencies on an approach for improving transparency and engaging the public on policy making.

The intent is one of "hardwiring our agencies for open government," he said at the conference.

[admin]

9/16/2009, TDWI By Stephen Swoyer

Just how popular are free or open source software (F/OSS) business intelligence (BI) offerings? Open source vendors like to talk up a looming tidal wave of F/OSS adoption, and there's certainly no shortage of activity in the F/OSS BI and DW markets. What's more, plenty of market forecasts project that F/OSS business intelligence will, over the next half-decade, emerge as a force to be reckoned with.

That change may occur sooner according to a new report published by author and industry veteran Mark Madsen, a principal with consultancy Third Nature.

Madsen's research, which was sponsored by open source software vendors Jaspersoft and Infobright, paints a picture of a teeming open source business intelligence segment that -- although lagging behind the maturity of F/OSS offerings in other spaces -- is quickly evolving, such that it now comprises a credible alternative to proprietary BI. The proof, Madsen says, is in the deployments.

About a quarter of all companies -- from small and midsize shops to large enterprise environments -- are using F/OSS offerings today. (F/OSS use ranges from a high of 32 percent among small shops to 27 percent among large enterprises.) F/OSS is clearly here Madsen points out. What's striking, he continues, is how many shops are evaluating F/OSS BI or DW offerings: nearly 40 percent of small shops and about one-third of midsize and large organizations are mulling F/OSS BI offerings.

The upshot, Madsen suggests, is that F/OSS business intelligence is crossing a threshold of sorts -- moving from niche or tactical to mainstream deployments. That's true of adoption in shops of all sizes: "One persistent myth is that small companies are the primary users of open source," writes Madsen, who stresses that this is no longer the case (assuming that it ever was). "While there are more small organizations evaluating and using open source than mid-sized or large … the data also shows that both small and large organizations are leading adoption over mid-sized organizations."

How shops are using F/OSS tools is also changing. True, small shops are most likely to deploy F/OSS BI offerings on an organization-wide basis. Deployments in mid-sized and large organizations tend to be confined to individual business units, or, smaller still, tactical use-cases, but mid-sized and large shops are expanding their open source vistas, Madsen stresses.

"[S]mall organizations are more likely than medium and large to do company-wide deployments, and large organizations are doing smaller deployments," he concedes, adding that -- from small to large shops, and regardless of the scope of their deployments -- F/OSS adopters tend to share similar characteristics: they're operating with limited budgets, have smaller user bases, and tend to have more "uniform" deployment scenarios.

"Despite this general pattern, there are enterprise-wide deployments of open source in large organizations," Madsen continues. "Forty percent of large organizations plan to or have deployed a BI or DW application corporate-wide with some open source components, demonstrating a level of software maturity."

Similarly, most open source BI applications today support a comparatively small user base: most deployments (54 percent) average between one and 24 users; less than 10 percent support 500 or more. However, the user base that consume F/OSS business intelligence technologies is poised to explode over the next two years. By 2011, Madsen says, almost 20 percent of open source BI deployments will support 500 or more users; by far the largest segment -- at nearly one-third of all deployments -- will support between 51 and 200 users.

Deployment expectations correspond to size, he says: large shops anticipate rolling out open source BI to a large number of (500 or more) users, small- and medium-sized shops have less ambitious -- but by no means reserved -- plans, typically projecting F/OSS deployments that range between 50 and 200 users.

"The number of users in environments with open source is similar to what is reported in actual usage in proprietary data warehouse environments," where shops actually use fewer licenses than they've paid for, Madsen writes. There's a sense, he says, in which the economics of commercial software licensing -- where shops intentionally buy more than they can use and inevitably use less than what they've paid for -- both work to the advantage of F/OSS business intelligence offerings and illustrate one of the primary ways in which open source is disrupting the enterprise software market.

"Open source has an advantage over proprietary solutions regarding scope because there is more deployment flexibility," Madsen continues. "People often buy more software than they need from traditional vendors because the high license cost makes obtaining funds for more seats a challenge, and because of the way the software is discounted in volume purchases. Open source can be less expensive and F/OSS versions carry no penalties for increasing or decreasing usage."


Stephen Swoyer is a technology writer based in Athens, Ga.

[admin]

Open-source vendor will roll on-demand Clearview app into its suite

LinuxWorld, By Chris Kanaracus, IDG News Service, 10/05/09

Open-source BI (business intelligence) vendor Pentaho has purchased technology from failed BI company LucidEra, it was announced Monday. Terms were not disclosed.

News of LucidEra's pending demise broke in June. At the time, a source familiar with the startup's plans told IDG News Service that it intended to wind down operations and put its intellectual property on the market.

Now, Pentaho plans to repackage LucidEra's Clearview, a SaaS (software as a service) reporting and analysis application aimed at non-technical users, as the front end of its Pentaho Analysis products, which are available both on-premises and on-demand.

New and existing customers of Pentaho Analysis Enterprise Edition and Pentaho BI Suite Enterprise Edition will receive the LucidEra technology at no additional charge.

Open-source BI packages such as Pentaho and JasperSoft tend to suffer from a lack of integration, since they are culled from a variety of open-source projects, said Forrester Research analyst Boris Evelson.

This can result in problems such as multiple user interfaces for customers to learn, and a lack of integrated metadata, he said.

But in this case, Pentaho may have a head start. Clearview was built on top of the Mondrian OLAP (online analytical processing) server, of which Pentaho is the lead sponsor, Evelson added.

[admin2]

Exclusive: U.S. Spies Buy Stake in Firm That Monitors Blogs, Tweets
By Noah Shachtman October 19, 2009 | 12:03 pm | Wired

America’s spy agencies want to read your blog posts, keep track of your Twitter updates — even check out your book reviews on Amazon.

In-Q-Tel, the investment arm of the CIA and the wider intelligence community, is putting cash into Visible Technologies, a software firm that specializes in monitoring social media. It’s part of a larger movement within the spy services to get better at using ”open source intelligence” — information that’s publicly available, but often hidden in the flood of TV shows, newspaper articles, blog posts, online videos and radio reports generated every day.

Visible crawls over half a million web 2.0 sites a day, scraping more than a million posts and conversations taking place on blogs, online forums, Flickr, YouTube, Twitter and Amazon. (It doesn’t touch closed social networks, like Facebook, at the moment.) Customers get customized, real-time feeds of what’s being said on these sites, based on a series of keywords.

“That’s kind of the basic step — get in and monitor,” says company senior vice president Blake Cahill.

Then Visible “scores” each post, labeling it as positive or negative, mixed or neutral. It examines how influential a conversation or an author is. (”Trying to determine who really matters,” as Cahill puts it.) Finally, Visible gives users a chance to tag posts, forward them to colleagues and allow them to response through a web interface.

In-Q-Tel says it wants Visible to keep track of foreign social media, and give spooks “early-warning detection on how issues are playing internationally,” spokesperson Donald Tighe tells Danger Room.

Of course, such a tool can also be pointed inward, at domestic bloggers or tweeters. Visible already keeps tabs on web 2.0 sites for Dell, AT&T and Verizon. For Microsoft, the company is monitoring the buzz on its Windows 7 rollout. For Spam-maker Hormel, Visible is tracking animal-right activists’ online campaigns against the company.

“Anything that is out in the open is fair game for collection,” says Steven Aftergood, who tracks intelligence issues at the Federation of American Scientists. But “even if information is openly gathered by intelligence agencies it would still be problematic if it were used for unauthorized domestic investigations or operations. Intelligence agencies or employees might be tempted to use the tools at their disposal to compile information on political figures, critics, journalists or others, and to exploit such information for political advantage. That is not permissible even if all of the information in question is technically ‘open source.’”

[zamir]

I just posted information on an interesting emerging player here.

[admin]

by Angus Kidman, ITWire
Monday, 02 November 2009

Microsoft and SAP are falling behind when it comes to taking advantage of open source technology, despite cut-throat competition in the enterprise software market, a new analysis by Gartner has found.

In a presentation on the four enterprise applications software "megavendors" -- IBM, Microsoft, Oracle and SAP -- Gartner research vice president Yvonne Genovese said that the US and German software giants both had only "minimal" strategies for developing and supporting open source, a situation which is likely to remain the case until 2015 or beyond.

Speaking at Gartner Symposium ITxpo in Cannes, Genovese said IBM was the most sophisticated in its adoption of open source technologies and techniques. "IBM is continuing to advance itself with using and embedding and directly commercialising open source," she said.

While Oracle's proposed purchase of Sun would give it "some momentum", the other two were doing little in the open source space, Genovese said. "If you're looking for open source today you're likely not going to find anything much in an SAP or Microsoft environment."

Genovese also predicted that enterprise sales tactics from the big four will become increasingly desperate as they fight to wring more money from existing business customers and lock them permanently into a single platform.

"If you look at it, each one of these vendors has a plan to encompass the application stack from the business applications all the way down into the database," she said.

"Many of them look like they're competing for the same stack. I wouldn't be surprised if many of you find that your account reps are coming in and trying to encroach on the others on a daily basis. You are a customer and they're going to come back to you time and time again and try to get more revenue."

Those vendors will increasingly rely on the fact that tight economic times mean that businesses have little option to switch away from their existing platforms. "The ability to change things like business applications has literally gone away. Maintenance is a huge issue in the market now, because there's lock-in. Most of you are paying for the software you own every five years."

The desire to make money also meant that industry-specific solutions would be harder to come by despite those fees, Genovese said. "These vendors are serving a lot of clients in multitudes of industries with a lot of different needs. They're not necessarily looking at you and your needs."

"They are building products that fit the largest target market audience size for the least amount of functionality -- I'm right on the edge of saying generic. You have to watch how far they're going to go into industry or customer-specific capabilities."

[Doug Heywood]

It's not open source but is it free??

"Amidst growing enterprise unrest over maintenance fees, SAP has pledged to bundle free analytics with its ERP systems and hopes for a general release of its on-demand Business ByDesign software early next year."

Cross post here.

[Jo Vincent]

This is a very useful case study and guide to setting up a successful decision market within Motorola by Rami Levy
a Technical Lead and Manager in Motorola Open Source Technologies (MOST).

I wasn't sure whether-or-not to post this here or over in the general Analytics Forum, or even in the Forecasting SIG.

Enjoy.