Galen: The problem is that you can't run repeatable experiments in the field of economics. Deducing what is happening from what is known about human behavior is the best that you can do.
Mike: You can't run repeatable experiments in astronomy either, yet that field of study is still a very successful science.
Laboratory experiments are the easiest way to obtain results in science. But one cannot always do things in the lab. Another approach is the natural experiment.
For example guy named Lott did a study of the effect of right to carry laws on crime rates. He took a huge data set of county-level crime data and put it into a statistical model which attempted to explain crime rate using a variety of social variables thought by criminologists to have an impact on crime, including the presence or absence of right-to-carry laws (RTC). He found a very statistically significant effect of the presence of guns and wrote a book about it called "more guns less crime. This study is perfect example of how not to do social science.
I read an article about the study and thought, that's not the way to do this, one should do a perturbation study. So I did one. I identified a set of states that changed their laws over a narrow period, and another set of states that did not change their laws over the entire period of interest. I then constructed a data series of crime rates for both groups and plotted them out. What you have here are two systems, one of which receives RTC and the other that does not. What I found is the two systems show the same basic behavior: both show rising crime rates in the 1980's and declining rates in the 1990's. RTC was added over the 1987-90 period for this set of states. If we look at the degree of change in the rate from the pre-1990 to post-1990 period the crime reduction was larger in the control group, but this result was not statistically significant. There was no evidence that right to carry laws had any favorable impact on crime, if anything the effect was the opposite.
Now if you analyzed all these together as one mass of data (as Lott did) and compared the one quarter of the data in which right-to-carry laws are present (which falls into the 1990's period when crime are rates are falling) you would get a result of falling crime when right to carry is present. In contrast the 3/4's of the data that occurs in periods without right to carry laws (two-thirds of which comes from the 1980's when crime was rising) would show an average result of rising crime. So you get a result of rising crime in the absence of right to carry and falling crime when right to carry is present, giving an apparent result of "more guns less crime", which is wrong. Its an artifact of the fact that the dominant factor (whether it was before or after 1990) was not included in the initial regression model.
The natural experiment is a technique that works to study systems in which laboratory experiments are not possible. It is not as good as the true experiment because one cannot employ randomization to break the lurking variable effects (why correlation is not causation) but it is still very useful. There are also a variety of sophisticated statistical tools evolutionary biologists and ecologists have developed that allow for controlled case studies approaches to be used in social science (Peter Turchin's work is an example). Even among economists there are economic historians who have both feet planted in empirical reality (e.g. Kindleberger, Minsky) who have produced useful descriptive and theoretical models for economic events.
Mike: You can't run repeatable experiments in astronomy either, yet that field of study is still a very successful science.
Laboratory experiments are the easiest way to obtain results in science. But one cannot always do things in the lab. Another approach is the natural experiment.
For example guy named Lott did a study of the effect of right to carry laws on crime rates. He took a huge data set of county-level crime data and put it into a statistical model which attempted to explain crime rate using a variety of social variables thought by criminologists to have an impact on crime, including the presence or absence of right-to-carry laws (RTC). He found a very statistically significant effect of the presence of guns and wrote a book about it called "more guns less crime. This study is perfect example of how not to do social science.
I read an article about the study and thought, that's not the way to do this, one should do a perturbation study. So I did one. I identified a set of states that changed their laws over a narrow period, and another set of states that did not change their laws over the entire period of interest. I then constructed a data series of crime rates for both groups and plotted them out. What you have here are two systems, one of which receives RTC and the other that does not. What I found is the two systems show the same basic behavior: both show rising crime rates in the 1980's and declining rates in the 1990's. RTC was added over the 1987-90 period for this set of states. If we look at the degree of change in the rate from the pre-1990 to post-1990 period the crime reduction was larger in the control group, but this result was not statistically significant. There was no evidence that right to carry laws had any favorable impact on crime, if anything the effect was the opposite.
Now if you analyzed all these together as one mass of data (as Lott did) and compared the one quarter of the data in which right-to-carry laws are present (which falls into the 1990's period when crime are rates are falling) you would get a result of falling crime when right to carry is present. In contrast the 3/4's of the data that occurs in periods without right to carry laws (two-thirds of which comes from the 1980's when crime was rising) would show an average result of rising crime. So you get a result of rising crime in the absence of right to carry and falling crime when right to carry is present, giving an apparent result of "more guns less crime", which is wrong. Its an artifact of the fact that the dominant factor (whether it was before or after 1990) was not included in the initial regression model.
The natural experiment is a technique that works to study systems in which laboratory experiments are not possible. It is not as good as the true experiment because one cannot employ randomization to break the lurking variable effects (why correlation is not causation) but it is still very useful. There are also a variety of sophisticated statistical tools evolutionary biologists and ecologists have developed that allow for controlled case studies approaches to be used in social science (Peter Turchin's work is an example). Even among economists there are economic historians who have both feet planted in empirical reality (e.g. Kindleberger, Minsky) who have produced useful descriptive and theoretical models for economic events.