Making more sense with numbers, part 4

This could be called Monty Hall and cognitive dissonance. John Tierney just published And Behind Door No. 1, a Fatal Flaw, a brief review of the Monty Hall problem and a report on its potential application to psychology, including its potential to invalidate some prior claims about subjects such as cognitive dissonance.

I'll leave the psychological arguments to others; the point is that thinking carefully isn't always as easy as it seems. If you're not convinced, read the start of that article down to "Before I tell you the answer, I have a request," and then write down your answer before proceeding. Then try out the online version to see if you got the right answer, to get a visceral feel for the game, and to see the reasoning.

Once you get the hang of those, try out Monty Hall’s Other Problems.

Do you now think you've got the hang of it? Just to confuse things a bit more, read Behind Monty Hall's Doors: Puzzle, Debate and Answer?, Tierney's 1991 report of playing the game with Monty Hall. By the end, you may have an even deeper appreciation of the challenge of making sense with numbers.

And if you wonder what this might have to do with business, remember that the impetus for Tierney's column was Yale economist M. Keith Chen's application of the Monty Hall problem to psychology. Are similar gotcha's waiting for us in business?

PS: Yes, there is a Making more sense with numbers, part 3.

The importance of a focus on disconfirmation

Here's a lesson from John Sterman's Business Dynamics: Systems Thinking and Modeling for a Complex World section 1.3.7: we gain little to no new insight by observing cases where data supports our hypotheses. We gain much from testing cases where data might disconfirm our hypotheses.

For more on that, see Raymond Nickerson's Confirmation Bias: A Ubiquitous Phenomenon in Many Guises, Bob Dick's Rigour and relevance in action research, the Skeptic's Dictionary entry on confirmation bias, Wikipedia's entry on the same subject, or one of my prior essays on skepticism.

System dynamics, black swans, and the management of business

I'm currently reading Nassim Nicholas Taleb's The Black Swan: The Impact of the Highly Improbable. While I intend to tell you more of what I think when I'm finished, I have an early impression, based on stories such as what he calls "Hume's problem" (or the turkey problem). That's a problem in which everything seems to be getting better and better, only to change direction suddenly and drastically for the worse. In his example, the turkey sees life as a daily succession of friendly humans offering food, only to have it cut short in a manner seemingly quite out of character for life as the turkey has perceived it. (As Taleb points out, it all makes eminent sense to the butcher.)

I think that's part of the reason for system dynamics as yet another tool for thinking and working. As Geoff Coyle points out in his System Dynamics Modelling: A Practical Approach, top management is concerned about things such as the consequences of actions, the likely future, and robustness against uncertainty (p. 15). One of the basic parts of the system dynamics approach is to challenge preconceived notions of the extent of the system causing the current situation: are we looking over a broad enough time span, are we including enough of the actors and actions, and are we paying attention to feedback effects (what Taleb calls recursive effects on p. xxii), where something we do today might come back and affect the situation we face tomorrow?

While there are no guarantees, that unfortunate turkey, had she had good training in system dynamics (or a competent system dynamicist at her side), might have been inspired to look at life over a 5-10 year time span, not just the few months she had experienced. That might have surfaced the fate that led to her demise as part of a regular pattern (albeit one that occurred rarely compared to her lifespan). Had she looked not only at the friendly human feeding her and the other turkeys eating with her, she might have noticed the butcher looking eagerly over the fence from time to time and asked about his role in her life. Had she realized the implications of those observations, she might have decided not to become quite so friendly with her "caretaker," she might have decided not to eat nearly as much (if she were scrawny, might her fate have been different?), and she might even have encouraged the other turkeys to join her in an escape attempt.

Now I don't think that the use of system dynamics conveys infallibility; in fact, that's why I'm reading Taleb's work, to figure out more places my insights may be fallible so that I can make them more robust.

Taleb advocates tinkering as a way to make progress; I see system dynamics as a way to tinker faster and think more effectively in support of your (and my) goal of more effective action.

While my comments may be out of the main focus of Taleb's thesis (system dynamicists tend to focus on the deterministic, not the random, even as they seek to help you be able to respond better in the presence of the random), I don't yet see them in contradiction. I offer them to you in the hopes they are of use to you. Now it's my (and your) task to try to disconfirm them; the longer we can't, the greater the likelihood there's something worth attending to!

If you want to tinker faster with the situation you find yourself in but don't want to risk your business each time you tinker, let's talk.

Thanks to Andrew Gelman for his posts that led me to Taleb's work.

Jane Jacobs

Some time ago, I wrote briefly about Jane Jacobs and her The Death and Life of Great American Cities. I found it in a list of recommendations from Andrew Gelman, which makes me want to go back and review the other books on his list that I haven't read.

I liked three things in particular about this book:

• her lessons about cities
  
• her detailed and interesting descriptions of her observations
  
• her very early use of ideas of complex adaptive systems
  

The first was simple: I had never really thought about the functions sidewalks and side streets play, and I had never thought about how a mix of uses during the day plays into keeping a city safe. I had not understood why some recommendations for urban renewal seemed to work so poorly nor what might be done about it. She made all that clear. I'll trust what she says, for I don't live in a city environment to be able to experience it first-hand.

The second was more important to me. Her book was, in a way, one long series of low-level observations, coupled closely with reasonable and reasoned inferences she would draw from those observations. She never got far away from the observations, so it seemed easy to verify her thinking. Using the ladder of abstraction metaphor, she seemed to stay on the lower rungs, and that made her thinking and her arguments more powerful.

The third surprised me. I didn't really expect an early 1960s book about city planning to dive into complexity theory, but she did it at the end of the book, after building up a remarkable story, and she did it in a way that was quite approachable. If you're curious, you can see an excerpt from "The kind of problem a city is," the last chapter of her book, at Katarxis No. 3.

If I took away lessons from this experience, they would include:

• Observe.
  
• Attend to outliers as well as central tendencies; attend to diversity as well as averages.
  
• Make sure inferences are based on observations, and make the chain to the observations as short and as transparent as reasonable.
  
• Explore new ideas and new theories, for some of them make help make better sense of observations. This admittedly may cause tension with the previous lesson.
  
• Be interesting, which comes in large measure from being interested.
  

I like to give links to other sources you can explore, but there are so many options in this case. P.J. Tayor published Jane Jacobs (1916-2006): An Appreciation in Environment and Planning A. Jacobs gave credit to Warren Weaver in her work on complexity; you can read the part of his work she references in The Rockefeller Annual Report, 1958 (start on page 23 of the PDF). I recommend this highly. If you liked Weaver's article and want to read more about making sense of complex situations in the social sciences, F.A. von Hayek's Nobel Memorial Lecture The Pretence of Knowledge might well belong on your reading list.

But, more than anything, read Jacob's The Death and Life of Great American Cities. It's worth it.

If you can say it, it's done

Even in this day and age, computing is a problem. How many of you us take the time to do some of the calculations mentioned here when faced with business or economic data, and how many of you us just read the analyst's summary and take the analyst's advice?

To some degree, that's because it takes time and effort to double-check such work, and that only gets worse if the subject is complex. It's also because the tools we have aren't always set up to help us do such things on the fly, and we're often on the fly (or in meetings, which can be as challenging).

That's one reason I've encouraged some of you who are interested to learn alternative approaches.

At least one APLer, Randy MacDonnell, has written about APL, "If you can say it, it's done." The same is true, of course, about J, its descendant. I had occasion recently to write a program to calculate whether a certain Monte Carlo simulation was done. I found a quotation by Andrew Gelman describing the Gelman - Rubin statistic:

For any given parameter, R-hat is the estimated posterior variance of the parameter, based on the mixture of all the simulated sequences, divided by the average of the variances within each sequence.

That looked easy enough, so I just wrote it down:

R=: var @: , % mean @: var

In English, that's "the variance of the entire set of data" (var @: ,)
"divided by" (%) "the mean of the variance of each data sequence" (mean @: var).

"If you can say it, it's done."

And you thought this was a blog about business, not programming, right? You were right. While J is a language that can be used by programmers, it's also a language that can be used by you and me to express quantitative ideas more powerfully and concisely than a spreadsheet. If you're ever interested in numerical answers from a spreadsheet, you could be interested in J. Perhaps, for some of you, it's worth downloading and trying out. Much as in learning a foreign (human) language, you won't be able to do much at first, but, eventually, you might be surprised what you can do. In a way, it's as much about thinking than about computing, and yet you can process some pretty large data sets with pretty concise "programs," too.

Thanks to Randy and Andrew for the quotations. For those of you interested in the Gelman-Rubin statistic, Andrew has pointed me to two papers giving more information: his Inference from Iterative Simulation Using Multiple Sequences with Donald Rubin and his General Methods for Monitoring Convergence of Iterative Simulations with Steve Brooks.

More on factorial designs and simulation

Factorial designs and simulation: apparently Professor Barlas is teaching it in Istanbul!

For the analytically-minded among you, I'd note that MCSim, coupled with a bit of infrastructure you can develop, can make running small factorial experiments (up to a few hundred runs) a fairly quick and painless task. That includes their development, their execution, and their analysis. If a system dynamics model is deterministic, as many are, there's no need for replication.

You can see a simple example in my upcoming TAFTO contribution. I was pleased at the way in which a factorial design approach enabled me to generate and see useful results from a relatively simple model.

"Scientific thinking" the modern way

Yesterday I wrote about Barry Richmond's notion of "operational thinking." I encourage you to read that essay; Barry had many good and important things to say. If you're not familiar with system dynamics but are curious, his essay offers a good introduction to pieces of the thinking approach that gives system dynamics its power.

Nevertheless, I disagree with his description of "scientific thinking," and I figure I should clarify that before others call me on it. On page 19, he writes, "People thinking scientifically modify only one thing at a time and hold all else constant." That used to be true, before Sir Ronald A. Fisher began describing a process for the design of experiments. Fisher and others gave us means to vary multiple factors at a time in a series of experiments and to learn more accurately, effectively, and productively that way.

One can do such designed experiments using any of the common system dynamics simulators, of course, but one of the reason that I like MCSim so much is that I found myself automatically doing factorial experiments using MCSim from the very beginning without thinking about it, thanks to the way it's designed. With other tools, I have tended to start with simpler approaches and then find myself having to make explicit decisions to design better experiments. Besides, MCSim can give results in a format that seems especially suited to this type of analysis.

Some of you might note that two of Fisher's attributes of designed experiments are randomization and replication. Those don't quite apply to many system dynamics models, those created without modeling any random effects. That's okay; it's still important to understand the effect of changes in various parameters, and, if the system is nonlinear (most are), it's important to understand interactions among those parameters, all of which is done effectively using methods pioneered by Fisher.

What does this all mean? It simply means that Fisher's designed experiments give us better and faster means to extract insight from tests on system dynamics models than the old one-factor-at-a-time approach.

I thank Deb Schenk, then (and perhaps now) statistician at Hewlett-Packard Company, for teaching me and others about the design of experiments using Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building back in 1981-82.

Now go read Barry's essay.

A systems language for business

Some time ago, I had an opportunity to teach system dynamics to a particular work team. It was a small team of perhaps 8-10 people, (only) one of whom had some system dynamics experience and one with some exposure to what system dynamics was all about.

On the first morning of the week-long class, I overheard a discussion between two or three of the participants during a break. They were talking about some issue they faced at work; one person said (let me paraphrase), "ABC," and the other said, "XYZ." The first repeated "ABC," and the second repeated, "XYZ."

You've probably heard these discussions before; you may even have participated in them. Both parties were stating their positions in what they saw as a clear and convincing manner. Neither either saw the need or had a way to try to link the two statements together so they could make progress deciding which was more useful to them, and so they kept talking past each other. It continued until the end of the break; there was no natural resolution. It was polite, it was friendly, and it wasn't very productive. I've heard those discussions many times in my career.

The system dynamics course was fun and intense: three full days of work learning system dynamics and working example problems, intermingled with two homework days in which they worked together as teams to solve some fairly difficult problems using simulation and without my help except by email. It must have been successful, too; while I had hoped to get the opportunity to come back and help their organization address specific problems using system dynamics, they felt comfortable enough afterwards to do it themselves.

I enjoyed working with that group for a number of reasons. One thing particularly caught my attention. In a break on the last day, two or three people, perhaps the same group, again started a discussion about a problem they faced at work. This time, instead of stating and restating their positions without a way to achieve resolution, the first person said, "DEF" (different problem this time) and then drew a stock and flow diagram that helped make that position clearer. The second person said, "UVW" and drew a stock and flow diagram to clarify that view. Then they started talking productively, using those models, about how their world really worked, about specific questions they had about each model, and about what they might need to change in one person's or the other's view to align more closely with reality and to be more useful for their work. While they didn't (yet) agree on answers, they gave every sign that they understood and appreciated each other's thinking and that they could converge on a common answer that was good for the organization, not necessarily one that agreed with their originial position.

Four and a half days of serious thinking on their part had taken them from being a normal workgroup to being one that could express ideas clearly, advocate for them effectively, and engage in serious dialog on ways to test their ideas and come to a resolution about how to proceed. While they didn't create a simulation model in that 15-minute break, they gave every sign they could likely have done so, given a bit more time.

Moreover, I think they had learned how to peer into each others' now rather explicit mental models to find the crux of problems or differences, which should let them focus their simulations on the essence, the core, of the situation and not the entire problem.

I was proud of them. In four and a half days, they had learned and decided to use what Barry Richmond called "operational thinking." I'm not sure they would have done that without the experience of learning to model and of seeing what their simulations could bring.

Do you see people saying "ABC" and "XYZ" in your organization? Do you wish your people had a better language for wrestling with tough problems? What techniques do you use?

While I like Barry's article quite a bit, I do have one point I'd like to take issue with. I'll save that for tomorrow.

Top postings of 2006

Here are the top ten postings on Making Sense With Facilitated Systems for 2006. Technically, this is a bit premature, for the data was captured on December 27, but I don't think things will change much.

Admittedly, there is a statistical problem with this list. Entries made last January have had more time to be viewed than entries made late in December. As I suspect most entries get their heaviest readership shortly after they're posted, I'm ignoring that potential problem and simply ranking postings by the total number of entries received from January 1, 2006 through December 27, 2006. Besides, the most popular item is also one of the most recent.

10. Last year, I had several postings on the use of systems thinking approaches, especially system dynamics, in program evaluation. System dynamics and program theory (evaluation) was number 10 on the list of most popular postings for the year.
    
    
9. Work is changing, and, for many of us, our work locations and time are becoming more flexible. Number 9 on the list of most favorite postings for 2006 was ROWE: Revolution IN work, a description of Best Buy's changed work environment.
   
   
8. People in mature organizations often wish their employees were more entrepreneurial. Number 8 on the list is Becoming more entrepreneurial, which sheds light on what Saras Sarasvathy and others call effectual reasoning. That's the style she finds most prevalent in entrepreneurs. Are you ready for that approach?
   
   
7. I am a fan and user of open source software, as I've explained multiple times. Number 7 on the list is OO.o tips, intended to help those thinking of switching to OpenOffice.org as their office suite.
   
   The next highest essay on the list, The Joy of Thinking Small, was published in November of 2005, and so it doesn't deserve a place on the top ten list for 2006. I'm mentioning it because it did earn the spot through popularity, and it might give you a pointer to some small tool you can use to make youself more productive.
   
   
6. Number 6 on the list concerned growth, another topic I've covered several times. S-curves, growth, and discerning your position was written to help us all think about whether growth was a good goal for our organizations at this point.
   
   
5. Decisions, decisions, decisions, one of several notes comparing Gary Klein's recognition-primed decision model with other approaches, was number 5 on the list. This essay also touched on Saaty's Analytic Hierarchy Process.
   
   
4. Introducing Systems Thinking into Your Organization, number 4 on the list, announced the temporarily free availability of an article by that name that I wrote for Pegasus Communications' Systems Thinker. The article is still good, and it's still available for a small fee from Pegasus Communications.
   
   
3. Number 3 on the list was Thinking systemically: Limits to Growth, my favorable review of Limits to Growth: The 30-Year Update. I found it to be both a good treatise on the subject of growth and a good, non-technical introduction to system dynamics.
   
   
2. Growth also occuppied the number 2 spot with More on growth, perhaps my most reflective essay on the subject in 2006.
   
   
1. By far, the most popular essay for 2006 was the recent Making sense with numbers. It used an example from the business of classical music to show how an esoteric-sounding concept called Bayes' Rule could help us make better sense of the statistics we may hear in meetings or read in reports. Its popularity was no doubt aided by those of you who blogged about it yourselves. Thanks!
   
   

While all of these essays describe ways of making sense of the world, I see four categories as being important to you.

First, four of those entries, numbers 1, 4, 5, and 10, are about making sense of general situations. That's the theme of this blog and the central theme of my work.

Second, the environment and organization's responses are key in numbers 2, 3, and 6. While I suspect not many companies are yet modulating their growth in response to the environment, many of you seem to be thinking about it. I was fortunate to have been able to work in the environmental arena last year, and I enjoyed it immensely.

Third, number 7 and its 2005 partner showcase ways to do things more productively. As I hope that's one of the benefits people get from my work, I'm glad it came out in my essays.

Fourth, numbers 8 and 9 speak to the ways we work in organizations. As I focus on helping people through helping the organizations in which we work, I'm glad that came through, too.

By contrast, which essay was the least popular? That turns out to have been A wake-up call with positive ideas, an essay about Clyde Prestowitz' Three Billion New Capitalists: The Great Shift of Wealth and Power to the East, one of several essays I've written about what I see as the importance of those of us in the U.S.A. connecting better with those of us who live in other parts of the world.



What's the map, you ask? That's where you live—you who voted on these essays by viewing them during the course of the year.

Thank you for reading these essays and for the comments you've made. I'll try to focus even more on issues you care about in 2007, and I'll try to throw in a few new areas, too, just to keep things interesting.

I welcome comments from even more of you in the coming year. Email me, call me, or add a comment to a blog entry. I'd especially appreciate your feedback on what you found helpful in 2006 and what you'd like to see me cover more thoroughly in 2007.

Where do you get your ideas?

Innovation is a big topic these days. Some years ago, I noted that I often learn well through a combination of action, reflection, and the injection of new ideas from outside. Sometimes those ideas come from dialog with others; at other times, they come from reading, perhaps even from seemingly unrelated books.

IBM has published Expanding the Innovation Horizon: The Global CEO Study 2006. Figure 11 on page 22 (section 3) of the printed report indicates that "Companies with higher revenue growth reported using external sources significantly more than slower growers" (40% to 30%).

Where do you get your fresh ideas? What fosters innovation in your organization?

Better thinking

We need our best wits about us in business. The New Scientist has an interesting list of eleven items to help strengthen our mental abilities.

So it's beans for breakfast, exercise, rest, and avoiding interruptions! Read the article for the rest.

Here's a question: if interruptions are bad for you (15 minutes lost work after each interruption) and thus your organization, while accessibility ard open door policies are good for your organization, what do you do?

Those already flooded in email may not like this answer, but I propose that email is a key part of the answer. About 20 years ago, I managed a group of software engineers who had developed a culture that they rarely spoke to the person at the next desk; they'd email that person. That way, they got to transmit their information or request on their timetable, and the recipient got to receive it on their timetable, giving everyone control. Sure, email volume has gone up drastically in 20 years, but technology has improved, too; most email software can help you sort email to make your life easier.

The power of "and"

Sumantra Ghoshal said it. Today, William Raspberry (free subscription required) said it. AT de Lange has said it.

Now I'll say it: we do ourselves and our colleagues a disservice when we act as if there are only two alternatives and they are mutually exclusive: hard-nosed or touchy-feely, profit or people, the environment or the economy, Left or Right. Often when we do that, we follow up by saying that one of the alternatives is good and the other bad. Life is much richer than that, even business life.

I'm beginning to suspect that categorizing everything as mutually exclusive extremes, as in the preceding list, is but one example of the human failing that led to "the map is not the territory." Maps are handy, but only when we realize they are aids, not reality. They help us find our way quickly, but they're less often helpful in puzzling situations. When faced with a puzzle, we're more likely to understand how to proceed when we ignore the map and investigate the reality or, lacking that, when we try multiple maps to see if one (or a combination) can help us make sense of the puzzle we face.

Certainly the best answer is sometimes at one of the extremes, as one might expect if best answers were randomly strewn about the spectrum. I've found we can save ourselves a lot of work and make much useful progress if we stay open to the idea that the best answer to a question might also be found somewhere in the middle, perhaps using bits of one extreme and the other, or perhaps using yet another idea not part of either extreme.