Forecasting Quality - What Makes a Forecast Good?

[admin]

A very interesting article found by Mr Dubossarsky to kick-off a discussion of what makes a forecast good:

Forecasting Guru Announces: “no scientific basis for forecasting climate”

28-01- 2009

It has been an interesting couple of days. Today yet another scientist has come forward with a press release saying that not only did their audit of IPCC forecasting procedures and found that they “violated 72 scientific principles of forecasting”, but that “The models were not intended as forecasting models and they have not been validated for that purpose.” This organization should know, they certify forecasters for many disciplines and in conjunction with John Hopkins University if Washington, DC, offer a Certificate of Forecasting Practice. The story below originally appeared in the blog of Australian Dr. Jennifer Marohasy. It is reprinted below, with with some pictures and links added for WUWT readers. - Anthony


J. Scott Armstrong, founder of the International Journal of Forecasting

Guest post by Jennifer Marohasy

YESTERDAY, a former chief at NASA, Dr John S. Theon, slammed the computer models used to determine future climate claiming they are not scientific in part because the modellers have “resisted making their work transparent so that it can be replicated independently by other scientists”. [1]

Today, a founder of the International Journal of Forecasting, Journal of Forecasting, International Institute of Forecasters, and International Symposium on Forecasting, and the author of Long-range Forecasting (1978, 1985), the Principles of Forecasting Handbook, and over 70 papers on forecasting, Dr J. Scott Armstrong, tabled a statement declaring that the forecasting process used by the Intergovernmental Panel on Climate Change (IPCC) lacks a scientific basis. [2]

What these two authorities, Drs Theon and Armstrong, are independently and explicitly stating is that the computer models underpinning the work of many scientific institutions concerned with global warming, including Australia’s CSIRO, are fundamentally flawed.

In today’s statement, made with economist Kesten Green, Dr Armstrong provides the following eight reasons as to why the current IPCC computer models lack a scientific basis:

1. No scientific forecasts of the changes in the Earth’s climate.

Currently, the only forecasts are those based on the opinions of some scientists. Computer modeling was used to create scenarios (i.e., stories) to represent the scientists’ opinions about what might happen. The models were not intended as forecasting models (Trenberth 2007) and they have not been validated for that purpose. Since the publication of our paper, no one has provided evidence to refute our claim that there are no scientific forecasts to support global warming.

We conducted an audit of the procedures described in the IPCC report and found that they clearly violated 72 scientific principles of forecasting (Green and Armstrong 2008). (No justification was provided for any of these violations.) For important forecasts, we can see no reason why any principle should be violated. We draw analogies to flying an aircraft or building a bridge or performing heart surgery—given the potential cost of errors, it is not permissible to violate principles.

2. Improper peer review process.

To our knowledge, papers claiming to forecast global warming have not been subject to peer review by experts in scientific forecasting.

3. Complexity and uncertainty of climate render expert opinions invalid for forecasting.

Expert opinions are an inappropriate forecasting method in situations that involve high complexity and high uncertainty. This conclusion is based on over eight decades of research. Armstrong (1978) provided a review of the evidence and this was supported by Tetlock’s (2005) study that involved 82,361 forecasts by 284 experts over two decades.

Long-term climate changes are highly complex due to the many factors that affect climate and to their interactions. Uncertainty about long-term climate changes is high due to a lack of good knowledge about such things as:
a) causes of climate change,
b) direction, lag time, and effect size of causal factors related to climate change,
c) effects of changing temperatures, and
d) costs and benefits of alternative actions to deal with climate changes (e.g., CO2 markets).

Given these conditions, expert opinions are not appropriate for long-term climate predictions.

4. Forecasts are needed for the effects of climate change.

Even if it were possible to forecast climate changes, it would still be necessary to forecast the effects of climate changes. In other words, in what ways might the effects be beneficial or harmful? Here again, we have been unable to find any scientific forecasts—as opposed to speculation—despite our appeals for such studies.

We addressed this issue with respect to studies involving the possible classification of polar bears as threatened or endangered (Armstrong, Green, and Soon 2008). In our audits of two key papers to support the polar bear listing, 41 principles were clearly violated by the authors of one paper and 61 by the authors of the other. It is not proper from a scientific or from a practical viewpoint to violate any principles. Again, there was no sign that the forecasters realized that they were making mistakes.

5. Forecasts are needed of the costs and benefits of alternative actions that might be taken to combat climate change.

Assuming that climate change could be accurately forecast, it would be necessary to forecast the costs and benefits of actions taken to reduce harmful effects, and to compare the net benefit with other feasible policies including taking no action. Here again we have been unable to find any scientific forecasts despite our appeals for such studies.

6. To justify using a climate forecasting model, one would need to test it against a relevant naïve model.

We used the Forecasting Method Selection Tree to help determine which method is most appropriate for forecasting long-term climate change. A copy of the Tree is attached as Appendix 1. It is drawn from comparative empirical studies from all areas of forecasting. It suggests that extrapolation is appropriate, and we chose a naïve (no change) model as an appropriate benchmark. A forecasting model should not be used unless it can be shown to provide forecasts that are more accurate than those from this naïve model, as it would otherwise increase error. In Green, Armstrong and Soon (2008), we show that the mean absolute error of 108 naïve forecasts for 50 years in the future was 0.24°C.

7. The climate system is stable.

To assess stability, we examined the errors from naïve forecasts for up to 100 years into the future. Using the U.K. Met Office Hadley Centre’s data, we started with 1850 and used that year’s average temperature as our forecast for the next 100 years. We then calculated the errors for each forecast horizon from 1 to 100. We repeated the process using the average temperature in 1851 as our naïve forecast for the next 100 years, and so on. This “successive updating” continued until year 2006, when we forecasted a single year ahead. This provided 157 one-year-ahead forecasts, 156 two-year-ahead and so on to 58 100-year-ahead forecasts.

We then examined how many forecasts were further than 0.5°C from the observed value. Fewer than 13% of forecasts of up to 65-years-ahead had absolute errors larger than 0.5°C. For longer horizons, fewer than 33% had absolute errors larger than 0.5°C. Given the remarkable stability of global mean temperature, it is unlikely that there would be any practical benefits from a forecasting method that provided more accurate forecasts.

8. Be conservative and avoid the precautionary principle.

One of the primary scientific principles in forecasting is to be conservative in the darkness of uncertainty. This principle also argues for the use of the naive no-change extrapolation. Some have argued for the precautionary principle as a way to be conservative. It is a political, not a scientific principle. As we explain in our essay in Appendix 2, it is actually an anti-scientific principle in that it attempts to make decisions without using rational analyses. Instead, cost/benefit analyses are appropriate given the available evidence which suggests that temperature is just as likely to go up as down. However, these analyses should be supported by scientific forecasts.

The reach of these models is extraordinary, for example, the CSIRO models are currently being used in Australia to determine water allocations for farmers and to justify the need for an Emissions Trading Scheme (ETS) – the most far-reaching of possible economic interventions. Yet, according to Dr Armstrong, these same models violate 72 scientific principles.

********************

1. Marc Morano, James Hansen’s Former NASA Supervisor Declares Himself a Skeptic, January 27,2009. http://epw.senate.gov/public/index.c...d-ecd53cd3d320

2. “Analysis of the U.S. Environmental Protection Agency’s Advanced Notice of Proposed Rulemaking for Greenhouse Gases”, Drs. J. Scott Armstrong and Kesten C. Green a statement prepared for US Senator Inhofe for an analysis of the US EPA’s proposed policies for greenhouse gases. http://theclimatebet.com

You can find the original article here and the sea of comments it has generated.

[Steve Bennett]

Eugene has posed the question 'what makes a forecast good?' Here's my 2 cents worth:

Firstly the forecast should prove to be correct - or at least a better prediction than any other (including gut instinct). So it certainly doesn't have to be perfect.

Secondly, the forecast should be based on a causal relationship not just a correlation. For example, there has been a paper published in 2007 that establishes a clear correlation between the US GDP and the number of sunspots as well as between the Dow Jones Industrial Average and the number of sunspots. Is there really a direct effect working here? The graphs do look good however! See the attached paper.

I'm sure the are more.

[Jo Vincent]

I agree with Steve and would add another:

• Transparent - a prediction based upon observable data and understood method is essential. Black box solutions are bad!

[BI Tragic]

If the forecast can not be acted upon then what use is it? So my vote for what makes a good forecast is:

Actionable.

[Eugene Dubossarsky]

With regard to actionable:

It is not always clear just what is actionable and what is not. Often the actionability of a certain set of forecasts and analyses is only realised after the fact. Actionability is thus a function of the business skills and instincts of the analyst.

Where actionability is not immediately obvious, too much information (and the tools and brains to make at least some sense of it) is better than not enough. There is always a significant uncertainty margin. Winners are on the right side of it.

Ideally, it is the job of a good manager to synthesise all possible information and determine possible action. In reality however the demand for "actionable" information often comes with a demand for advice as to the action itself. The job of the analyst thus becomes strategic advisory. The job of the manager in this situation is unclear, but it might not include such mundane "technical" things as using available information to make strategic decisions. Sadly, this is all too common.

Further, actionability is a function of mandate, scope and timing. Product managers may often want things that can be done NOW, and have little interest in trends that evolve over years, or affect products outside of their domain. These are not "actionable" for the product manager, but certainly of interest to the CEO.

[Steve Bennett]

Good points about what is actionable Eugene.

To take the discussion in a different direction I have been reading a little in the literature and also into weather forecasting.

Here is a definition from Allan Murphy back in the early 1990's. He was a pioneer in the field of forecast verification who wrote an essay on what makes a forecast "good". He distinguished three types of "goodness":

• Consistency - the degree to which the forecast corresponds to the forecaster's best judgement about the situation, based upon his/her knowledge base
• Quality - the degree to which the forecast corresponds to what actually happened
• Value - the degree to which the forecast helps the a decision maker to realize some incremental economic and/or other benefit

Here is an extract from an interesting website by a bunch of meteorologists:

Since we're interested in forecast verification, let's look a bit closer at the forecast quality. Murphy described nine aspects (called "attributes") that contribute to the quality of a forecast. These are:

• Bias - the correspondence between the mean forecast and mean observation.
• Association - the strength of the linear relationship between the forecasts and observations (for example, the correlation coefficient measures this linear relationship)
• Accuracy - the level of agreement between the forecast and the truth (as represented by observations). The difference between the forecast and the observation is the error. The lower the errors, the greater the accuracy.
• Skill - the relative accuracy of the forecast over some reference forecast. The reference forecast is generally an unskilled forecast such as random chance, persistence (defined as the most recent set of observations, "persistence" implies no change in condition), or climatology. Skill refers to the increase in accuracy due purely to the "smarts" of the forecast system. Weather forecasts may be more accurate simply because the weather is easier to forecast -- skill takes this into account.
• Reliability - the average agreement between the forecast values and the observed values. If all forecasts are considered together, then the overall reliability is the same as the bias. If the forecasts are stratified into different ranges or categories, then the reliability is the same as the conditional bias, i.e., it has a different value for each category.
• Resolution - the ability of the forecast to sort or resolve the set of events into subsets with different frequency distributions. This means that the distribution of outcomes when "A" was forecast is different from the distribution of outcomes when "B" is forecast. Even if the forecasts are wrong, the forecast system has resolution if it can successfully separate one type of outcome from another.
• Sharpness - the tendency of the forecast to predict extreme values. To use a counter-example, a forecast of "climatology" has no sharpness. Sharpness is a property of the forecast only, and like resolution, a forecast can have this attribute even if it's wrong (in this case it would have poor reliability).
• Discrimination - ability of the forecast to discriminate among observations, that is, to have a higher prediction frequency for an outcome whenever that outcome occurs.
• Uncertainty - the variability of the observations. The greater the uncertainty, the more difficult the forecast will tend to be.

Traditionally, forecast verification has emphasized accuracy and skill. It's important to note that the other attributes of forecast performance also have a strong influence on the value of the forecast.

Forecast quality vs. value

Forecast quality is not the same as forecast value. A forecast has high quality if it predicts the observed conditions well according to some objective or subjective criteria. It has value if it helps the user to make a better decision.
Imagine a situation in which a high resolution numerical weather prediction model predicts the development of isolated thunderstorms in a particular region, and thunderstorms are indeed observed in the region but not in the particular spots suggested by the model. According to most standard verification measures this forecast would have poor quality, yet it might be very valuable to the forecaster in issuing a public weather forecast.

An example of a forecast with high quality but little value is a forecast of clear skies over the Sahara Desert during the dry season.

When the cost of a missed event is high, the deliberate overforecasting of a rare event may be justified, even though a large number of false alarms may also result. An example of such a circumstance is the occurence of fog at airports. In this case quadratic scoring rules (those involving squared errors) will tend to penalise such forecasts harshly, and a positively oriented score such as "hit rate" may be more useful.

[Doug Heywood]

"Don’t predict the future, just make it better for the ones that come after us."

Hans Schumacher
Bank of America's Quantitative Operations Executive leading modeling, analysis, and testing for Card Services and Small Business. He is also a Center for Future Banking research Research Affiliate.

[Jo Vincent]

The better the data on current and past performance, the better the forecast can be.

I noticed a great initiative from APRA a couple of weeks ago:

APRA launches data revolution

Financial Standard, Thursday, 20 August 2009 12:30pm

APRA has released its fund level returns data and in the process launched a revolution in the extent of data now available for researchers investigating superannuation.

Data now available includes, for each fund, details of what they provide to the regulator on their official returns which the regulator has used to calculate the notional fund-level ‘returns' indicator that they call ROR - return of retirement assets.

In publishing the data to the extent it has, the prudential regulator has taken disclosure in Australia to unprecedented levels and in so doing have dismissed most of the criticisms thrown at them by retail fund interest groups and researchers which have long campaigned that the notional returns information will be misleading because it doesn't describe specific investment option performance.

An initial analysis of the results conducted by Rainmaker reveals the top returning - as opposed to performing - segment to be public sector super funds with an annual ROR for the five years to December 2008 of 9.8 per cent.

Next in line were industry funds with 9.3 per cent, corporate funds with 8.5 per cent, and retail funds with a five year ROR of just 6.6 per cent.

The top funds for their five year ROR were Goldman Sachs JBWere Superannuation Fund with 14.0 per cent, MTAA Superannuation Fund with 12.8 per cent, AustPost with 12.5 per cent, NAB's corporate fund with 11.5 per cent and the CommBank's corporate fund with 11.2 per cent.

The top ranked retail fund, PLUM Superannuation, came in at 61st place with a five year ROR of 9.3 per cent.

Perpetual Investor Choice came in at 63rd place with 9.2 per cent and BT Classic Lifetime came in at 65th place on the same result.

Alex Dunnin

[Jo Vincent]

Here is a cross post on a very useful case study and guide to setting up a successful decision market within Motorola.

[admin]

In the area of fraud mitigation technology, insurers are employing data mining and predictive analytics technologies in increasingly innovative ways to identify obscured data patterns and establish effective benchmarks for claims investigations.

Insurance and Technology, By Nathan Conz
DECEMBER 21, 2009

There is a common misconception that as technology does more, people do less. But while automation relieves people of manual tasks, it also frees them to do their jobs smartly. The benefits of technology are obvious in fraud mitigation, an area of the insurance enterprise where technologies around data mining and analytics have helped special investigations unit (SIU) members take deeper dives into claims data to uncover patterns of potential fraud that previously went undetected. When it comes to fraud mitigation, technology doesn't replace human ingenuity -- it augments it by providing investigators and claims adjusters with more context and more information. As such, the true value of data mining and analytics to fraud mitigation is realized in the innovative ways that carriers leverage it.

And for certain, carriers have had to become more innovative as the frequency and complexity of fraud have increased. Fraudsters have ramped up their efforts recently in large part because of the economic downturn. "The most serious current fraud threat is the increasing number of people who are in economic difficulties -- or think they may be in the near future -- and look for claims checks from insurers as a way of obtaining some quick cash," notes Donald Light, a senior analyst in Celent's insurance practice.

As fraudsters have turned up the intensity of their attacks, their tactics have become ever more sophisticated. When it comes to fraud prevention, it is no longer possible for technology organizations to hope to maintain a status quo -- those that do not proactively stay a step ahead of the bad guys become their prime targets.

"The world is getting smaller, the technology is getting more sophisticated and the bad guys unfortunately are getting more sophisticated too," says Robert Zandoli, VP of IT risk and compliance and chief information security officer at New York-based MetLife ($51 billion in total revenue). "On the other side of that, we have many programs using state-of-the-art technology to continue to keep up with those threats."

Citing her own experience as well as conversations with peers, Sheri Farrar, executive director of Chicago-based Health Care Service Corp.'s (more than 12.3 million members) special investigations department, says there has been an increase in healthcare fraud recently. Traditional schemes -- such as billing for services not rendered, miscoding to be paid for services not covered and up-coding -- remain, she adds, but they've grown and evolved. "The traditional fraud is still there," Farrar notes. "But we're seeing increases in using those types of schemes in more-sophisticated ways."

To fight those schemes, insurers are turning to increasingly sophisticated data analytics tools. According to Celent's Light, data mining and analytics solutions make up the foundation of modern fraud mitigation technology. "Data mining enables analysts within the claims department or an SIU to find patterns that are invisible to individual adjusters because a single adjuster only sees a small portion of the total claim volume," Light says. "Predictive analytics builds on the data mining findings to create red flags and claim potential scores."

Health Care Service Corp. (HCSC) is among the carriers that are employing analytics successfully in the fight against fraud. In June the company announced that it had uncovered a fraudulent scheme involving an Illinois allergist's office that was billing for non-rendered services, unbundling services and balance-billing members. The scam generated $800,000 in fraudulent claims and resulted in $2.5 million in fines and restitution.

According to Mone Petsod, a senior investigator in HCSC's SIU, the initial allegations against the allergist involved very small amounts of money. Even after the FBI contacted HCSC with more complaints from members, the case still had little momentum, he admits, acknowledging that in the past the case could have stalled. But by leveraging a joint solution developed in conjunction with IBM (Armonk, N.Y.) and SAS (Cary, N.C.), Petsod was able to uncover suspicious patterns during a data analysis.

"The allegations involved rather small sums of money, but when we looked into the data we found other issues with this same provider and her billing," Petsod recalls. "That's when the government began to investigate this case further."

The solution, which was custom-developed by HCSC and its vendor partners, is built from IBM's FAMS solution and SAS Enterprise Miner. According to HCSC's Farrar, the two technologies work in tandem. FAMS is a more traditional fraud solution, with rules-based capabilities, while the SAS data-mining tool provides HCSC with the flexibility to perform additional analytics, she explains. "It allows us to drill down into the claims data further than just the code level," Farrar says. "Using those different levels of analytics, you might be able to identify a fraud scheme that, on the surface, wouldn't appear in just your basic data analysis."

The key in the allergist case, Petsod says, was comparative analysis. When only the allergist's total billing was examined, the scheme was not apparent.

Innovative and Proactive Analysis

The break came when Petsod was able to compare how and what the provider in question was billing customers for a particular procedure against other providers' billing. "The main thing that jumped out at me was how the allergist was billing," Petsod says. "It was so different from other allergists."
In the past, Petsod adds, the SIU was limited largely to following leads. The new data analysis capabilities enable much broader review. "Previously I wouldn't have been able to look at the whole picture to see how this particular provider was different," she comments.

The contracts for the solution were signed in the second half of 2004 and a pilot was conducted in 2005. But, Farrar says, the carrier didn't begin to truly realize the benefits of its new data analysis capabilities until 2007. "It takes time to really understand the capabilities that you have, develop the routines that you want to use and begin to see the results as you apply your data-mining capabilities to your pool of claims," she relates.

To ensure that investigators understand the new data capabilities, HCSC has conducted a tremendous amount of training over the past couple of years, reports Farrar, who concedes that in hindsight training should have started sooner, when the solution was in development. "The cases that are being identified [now] are more complex -- there has been a learning curve for the investigators to understand the data as it comes to them," Farrar says. "It requires a lot more interaction between our analysts and our investigators so the analyst can explain to the investigator why the data tends to indicate that something inappropriate is going on."

A Scientific Discovery

A strong analytics layer also is key to Infinity Property & Casualty Corp.'s (IPACC) fraud-detection strategy. But until 2008 the Birmingham, Ala.-based auto insurer relied on a traditional SIU department that worked closely with the claims adjustment staff to identify and investigate fraud. A part of that process included providing adjusters with National Insurance Crime Bureau (NICB) red flag indicators, which were delivered to adjusters via laminated paper documents.

"It was basically up to the adjusters and the supervisor to identify [potentially fraudulent] claims," says William Dibble, SVP, claims, IPACC. "That was how we approached it. It was very traditional, no science applied."

In recent years, though, the carrier determined that fraudulent claims were slipping through the cracks. "[Adjusters] were so wrapped up in getting the claim settled that they were missing basic fraud issues," Dibble admits. "We also were finding that there were delays in getting files to the SIU department. It was just an antiquated approach to fighting fraud. We realized that the bad guys were winning. They were becoming much more sophisticated than we were."

To fight back, IPACC ($897 million in 2008 gross written premium) implemented a solution that incorporates components of the IBM SPSS (Chicago) predictive analytics suite, including its Decision Management Tools and Modeler products. "We wanted to take a solution that we could use across the enterprise, not just for claims but perhaps for underwriting and sales," Dibble explains.

Nonetheless, Dibble stresses, the primary goal of the system was to streamline the claims process. It was important, he adds, to right-track claims that did not have any fraud identifiers -- speeding the settlement process. "I did not want just a fraud model," Dibble says. "I wanted something more robust."

What the carrier got with the SPSS tool -- which was purchased in July 2007 and deployed in February 2008 -- is a solution that identifies suspicious claims earlier in the process, allowing the SIU to start investigations earlier and with access to better information. According to a case study developed by Nucleus Research, IPACC has had more success investigating fraudulent claims largely because investigators are examining evidence before it is stale and interviewing the parties involved while their memories are still fresh.

Further, the tool has allowed IPACC to take a more nuanced approach to fraud identification. For example, "We can now identify fraud based on geographic location," Dibble explains. "It's not just one rule applied everywhere. For instance, we know that clinical or medical fraud is very big in Miami, so we can build certain rules just for Miami."

In one case IPACC found that a specific intersection was seeing a large number of accidents involving its insureds at a certain time of day. "By building rules that looked at all accidents between 10 p.m. and midnight involving multiple-occupancy vehicles and at-fault accidents at a certain intersection, we found a [fraud] ring in that area that was generating claims," Dibble relates.

Currently the carrier's adjusters and SIU personnel leverage the new analytics capabilities to identify trends and create rules to address them. But in the future, Dibble notes, he expects that the tool will identify trends proactively, freeing analysts to focus on rules generation. "There is no reason the SPSS software couldn't do that for us," he suggests. "Here's how it is evolving: When we started, we had two analysts that were looking at rules. We're now putting a third one in, and I surely [hope] to put in a fourth next year. Those rules keep changing daily, and we keep getting more robust."

Unfortunately fraudsters are changing their tactics just as quickly. In the health insurance space, new types of fraud related to diagnostic testing and alternative therapies are creeping up on carriers, according to HCSC's Farrar. Some of those newer fraud schemes are so complex that they are difficult to detect -- even for some data mining and rules-based technologies. "So many of [the fraud schemes] touch on medical necessity -- healthcare fraud that includes a medical necessity component generally requires that you look at the medical records and have some sort of medical review," Farrar explains, adding that these claims require more human analysis than a rules-based system that flags claims can provide.

"The problem seems to be growing, and it seems to be getting more complex as well, making data-mining ... solutions more difficult [to leverage effectively]," Farrar says. "You can't do it with just technology. You need to have the analysts behind that technology to identify the best cases, and then you have to have investigators who understand the data and have the ability to complete investigations."