Friday, May 28, 2010

MITRE Prediction Market Pilot

Couple of days ago I’ve signed up for the MITRE Prediction Markets Pilot. Prediction market(s) is one of the most controversial and recent means of organizational tools. It is in such early development stage, that few organizations have ever made an attempt to formally integrate the method into its organizational design.

In short prediction markets create common and usually electronic platform that allow participants to put their bets on certain prediction, which has asset price and where this price is tide to the probability of this event. Most famous and successful example of prediction market is Iowa Electronic Markets, a non-profit academic exchange, which conducts US Presidential Elections prediction markets and claims to have more accurate results than the election pools.

Here are some basics: let us assume that in the Winner-Takes-All Market one-dollar value is assigned if the prediction that candidate A wins the election and zero dollars if the same candidate looses the election. The price of the contract that fluctuates between the staring point and the end-point of the bet (for example between 0.2 dollars and 0.4 dollars) means that the chances of candidate A winning the election fluctuates between 0.2 and 0.4 probability. Let us assume that the probability of the bet at a certain day is 0.33. This means that one can buy the contact for 0.33 dollar at that day and if the candidate A wins the election at some point in the future, the gain is equal to 0.77 dollar (or loose 0.33 dollar if the candidate fails).

Basic requirements for Prediction market are:



  • Clear rules. Event should have definite outcome and ending date. Preferably known market participants.

  • Clear incentives (monetary or other) to make accurate predictions and bet against other participants.

  • Stable market size.

  • Adequate rewards for risk taking.

  • Mix of participants, which includes experts and non-experts like me.

  • Clearly defined, exhaustive and engaging predictions.


In the case of MITRE pilot, there are predictions such as “When will the oil spill in the Gulf of TX be contained (i.e. no longer allowed to contaminate the water?)”, “Will Greece default on all or part of its sovereign debt by December 31, 2011?”, “Will the iPhone be available on the Verizon Network on Oct 1, 2010?” and many more. Each player has 5,000 units of virtual currency (which for some reason are called dollars, perhaps because organizers would like to honor the currency in which the pilot is funded :) ).

In 2005 Google made an attempt to introduce prediction markets to evaluate its internal performance (such as product launch dates or quarterly figures) and assess external competitive environment (such as actions and products made by its competitors). One key distinction from the Iowa Electronic Markets and MITRE’s pilot was that Google employees/traders were trading in pseudo currency called Goobles.  The authors of the project created indirect reward system, so that monetary benefits would not distort the prediction outcomes. After all, the value generated by the prediction markets was participants’ perception of future development and events. This shift in motivation made traders focus on their reputation, credibility and ability to influence the market.  It is not clear if the prediction market is operational at Google these days. It seems that long-term sustainability of such initiative require strong commitment form both management and market participants. One interesting and unexpected conclusion that can be taken from Google’s initiative is the fact that prediction market can map the flow of information within particular organization. There was a study done by Google and Wharton scholars to prove this point (Using Prediction Markets to Track Information Flows: Evidence from Google, 2009).  One more important detail, in Google’s case the prediction market techniques fit company’s innovative culture with its etiquette of internal communication, as well as diverse, tech-savvy and self-motivated employees.

It is too early to say, but it seems that MITRE’s pilot is also moving into the direction of using non-monetary incentives such as honor badges, surveys, rankings, etc. The main question here is whether the performance of the players will eventually be measured by virtual monetary gains or accuracy of predictions.

So what is the justification for prediction markets? I tried to dig some articles in Google Scholar.  Here is a good summary of justifications that comes from Joyce E. Berg and Thomas A. Rietz; and their 2003 Prediction Markets as Decision Support Systems, Information Systems Frontiers Market paper: “(1) the markets give continuously updated dynamic forecasts; (2) through the price formation process, the markets aggregate information across traders, solving what would otherwise be complex (at best) aggregation problems; (3) markets give unbiased, relatively accurate forecasts well in advance of outcomes; (4) these forecasts can outperform existing; (5) the evidence suggests that market dynamics can overcome biases that individual traders may have, effectively eliminating them from forecasts; (6) the markets can be designed to forecast a variety of issues and provide a variety of types of information” (Joyce E. Berg, Thomas A. Rietz, 2003).



Cass R. Sunstein, Administrator of the White House Office of Information and Regulatory Affairs, former University of Chicago Law School and current Harvard Law School professor, argues that predictions market (information market) tend operate more accurately than exit pools and expert panels due to the dynamic and aggregate nature of the market. In prediction markets (information markets) participants are given the right incentives to disclose information they hold. “Groups often hold a great deal of information, and an important task is to elicit and use the information of their members… Much of the time, informational influences and social pressures lead members not to say what they know. As a consequence, groups tend to propagate and even amplify cognitive errors. They emphasize shared information at the expense of unshared information, resulting in hidden profiles… [Information] markets tend to correct rather than amplify individual error, above all because they allow shrewd investors to take advantage of the mistakes made by others. By providing economic rewards for correct individual answers, they encourage investors to disclose the information they have. As the result, they are more often more accurate than the judgment of deliberating groups”(Sunstein, 2006).

It seems that prediction markets have value, but I will have to try it on my own to be convinced.  So currently I am in the "learning by doing" mode: taking careful strategies and aiming to get into upper quartile rank of market traders. I will keep readers posted on my success.

As for the trading platform of MITRE’s pilot, I should say that it is user-friendly. It has Dashboard and Stats that engage you into the trading process. It also gives you an option to ask clarifying questions and make comments before or after particular trade is made.  So the system is evolving and I am interested to see where it will takes us in the end (the pilot will last for 6 months as far as I know).

You can visit the site https://mitre.inklingmarkets.com/ to be the judge.


Wednesday, May 26, 2010

"The Logic Of Failure: Recognizing And Avoiding Error In Complex Situations"

Great book written in 1989 by West German Professor Dorner. This is a good introduction into System dynamics and System thinking. The book gives several mental models of how people "attack" complex problems and why they often fail. Dorner describes several psychological experiments that help to differentiate "bad decision maker" from "good decision makers," as well as mental traps that lead to failures. Based on this analysis he presents his model of decomposition and planning. At the end of the book Dorner quotes Clausewitz that "...War is not an infinite mass of minor events... War consists rather of single, great, decisive actions, each of which needs to be handled individually." Such "strategic thinking" requires far greater expenditure of mental energy argues Dorner.

Dorner ends his book with the thought that "...If we cannot form a picture of a temporal configuration, we cannot adjust our thinking and actions to take that temporal pattern into account... We human being are creature of the present. But the world today must learn to think in temporal configuration."

The book is easy to read and comprehend. I gave it to my 15 years old son to read it. Hopefully he will use my advise to read it earlier in his life.

"The Field Guide to Understanding Human Error" by Sidney Dekker

I met Professor Dekker couple of weeks ago during his visit to Engineering System Division at MIT.

He is definitely one of the most intelligent and sharp thinking persons I have ever met. I was impressed with his clarity and speed of thought. I could catch with his speed of thinking only after he made an iterative loop in his argument.

There are two major themes that are reflected throughout his book. First is that human error is not the cause but a symptom of a trouble. Accepting this notion is first step towards New view of human error, which can never be the conclusion of an investigation, but rather its starting point. Understanding error means reconstructing the context in which such decision was made. "Human error problem is an organizational problem.  This means that understanding human error hinges in understanding the organizational context in which people work."

People tend to react post factum and backwardly reconstruct sequence of events in a linear manner.  In the aftermath of an accident, one can easily list logical arguments how and why people should have foreseen and prevented the upcoming events. We easily judge people, who failed to take proper actions. We focus on their personal shortcomings such as absence of proper training and experience, health conditions and hours of proper sleep, etc.  We usually tend to focus our attention on people who happened to be closest to the accident in term of time and space.

Second theme of the book is forward looking metal model, which focuses on preventing accidents in the future rather than analyzing the accident with a hindsight bias. Hindsight gives the investigators better and more complete information in comparison to people who made the decision prior to the failure.  It provides with facts that become midpoints in the linear logic flow that the investigator reconstructs. As we walk backwards, each facts is perceive not as an "intersection point" with a list of equally valid choices, but rather as point of the process when/where incorrect decision was recorded. Hindsight bias exaggerates the importance of the recorded facts versus other events that are not directly related to the specific accident.

The author compares three accident models (1) the sequence-of-event, (2) the epidemiological model and (3) the system model. He argues that the latter is a holistic approach that looks at accidents as emerging form interactions between system components and processes, rather than failures within them.

Dekker warns against "quick fixes" and misuse of technical labels, which do not describe the gap between reality and our judgments, whereas "safety improvements come from organizations monitoring and understanding the gap between procedures and practice." He concludes that "a safety culture is a culture that allow the boss to hear bad news."

Tim Brown on Change by Design

I've got the book after Tim Brown's video lecture at Sloan MIT at the end of spring semester. The book is full of so many great and original design ideas. It was hard to concentrate while reading this book, because you want to close it in mid-sentence and try to change the world around you.  Here are some marvelous quotes from the book:

"Design has the power to enrich our lives by engaging our emotions through image, form, texture, color, sound, and smell. ... design thinking points to the next step: we can use our empathy and understanding of people to design experience that create opportunities for active engagement and participation."

"Unlike a prepared script or an operational manual, [blueprint] connects the customer experience and the business opportunity. Every detail holds the potential to sour a relationship... but only few offer possibilities for an experience that is distinctive, emotionally gratifying, and memorable. The blue-print is at one and the same time a high-level strategy document and a fine-grained analysis of the details that matter."

"Design can help to improve our lives in the present. Design thinking can help us chart a path into the future."

"Instead of an inflexible, hierarchical process that is designed once and executed many times, we must imagine how we might create highly flexible, constantly evolving system in which each exchanges between participants is an opportunity for empathy, insight, innovation and implementation."

Thursday, May 20, 2010

Principle 42

"Instead of an inflexible, hierarchical process that is designed once and executed many times, we must imagine how we might create highly flexible, constantly evolving system in which each exchange between participants is an opportunity for empathy, insight, innovation and implementation"  (Tim Brown, 2009).

Principle 41

"Design can help to improve our lives in the present. Design thinking can help us chart a path into the future" (Tim Brown, 2009).