Misconceptions about Scientific Experiments

I am afraid that the reader has quietly read the last paragraphs about experiments but still hasn't begun to rethink his ideas of what a scientific experiment is. I taught physics laboratory - pretty intensive with four hour labs once a week - and after fifteen labs the students were just beginning to let go of their misconceptions about the scientific method. So now I want to address three popular misconceptions straight on.

The first misconception is just the popular meaning of the word "experiment." "I experimented with adding nutmeg to the pancakes." "He experimented with drugs." In everyday usage "experiment" means doing something new in hope of obtaining some improvement. Often one doesn't know ahead of time what improvement may be obtained, except perhaps in very general intuitive terms, and in fact one often hopes for a surprise. Along the same lines, we could say that early humans experimented with eating various substances, domesticating animals, and sowing seeds; this was long before natural philosophers and has little to do them. Experimentation in this sense belongs instead to the normal range of activities centered on "getting things done" which I contrasted earlier (link) with philosophy. It is oriented toward results, and at a deeper level it is oriented toward human beings and their interests.

In contrast, scientific experimentation is oriented away from ourselves, toward nature, toward observing nature as it is. Compared to normal experimentation, scientific experimentation involves a lot more planning attention to observing what happens during the experiment or what is produced by it. Experimental scientists spend their entire careers inventing and refining new tools and techniques for observing things - seeing them more clearly, more accurately, with more detail, and extending the range of observation. The observational procedures and tools are checked and rechecked in meticulous detail, calibrated, refined, rebuilt, all to ascertain that the new vision the produce is real and not just fancy or mirages. While readying a new experimental tool one plans exactly how to use it, what to measure, so that afterwards one can check and rethink whether your observational procedure really did what you thought it was doing. Also one writes out in detail one's expectations about the results of the observation, in order to be able to contrast one's expectations with the actual observation. It is well known that humans are much better at noticing things that change than things that stay the same, so writing out our expectations ahead of time improves our observational abilities.

A second misconception about scientific experiments comes from our "experiments" in school laboratories. One is given some materials (wire, chemicals, roller carts, etc.), some measurement devices (electrical meters, microscopes, rulers, etc.,) and instructions about about what to do with them. The instructions typically require reading and recording one or more numbers, computing something based on those numbers, and then reporting the results of your computation. The emphasis is on following a procedure and producing some desired numbers.

In one way these school laboratories do teach something about scientific experimentation: depending on the lab format, the students may be encouraged to encounter and manipulate objects, perhaps even with some care and attention, and will typically be asked to use at least one number to describe the "outcome" of the experiment. However these labs can also give entirely the wrong impression. Probably their single biggest defect is that they focus on procedure more than observation. I would like to see labs where the students are taught to observe and think clearly about everything before them, to really see the wires and chemicals, sense them, measure everything about them and then measure everything about the measurement devices. A scientist doing an experiment is really interested in the thing she is measuring, and wants to know everything about it, and finally to understand it. She won't be content with a single number, or a hundred numbers. Preliminary studies of a newly encountered material or phenomenon usually involve many different measuring instruments, each producing large quantities of information. A large part of a scientist's life is sorting through the information, checking it and the way it was produced, processing it, piecing it together to get a full vision of the thing being observed. The end results may be in a synthetic summary form, or may highlight the most striking information, but this conciseness belies the care that has been taken to observe as much as possible, as comprehensively as possible. A good scientist really wants to get know what she is measuring, through and through.

Of course this is not an innate skill: if you tell a typical student that you want him to understand everything about a particular experiment including the measurement apparatus, he will look at it for some minutes, maybe measure a couple of things, write down a few sentences and numbers, and be done. The observational skills, and curiosity to think of things to measure, are not built in. Neither is the ability to check one's own measuring devices and procedures and invent and refine new measurement procedures as necessary. So I haven't figured out yet exactly how to make a school laboratory that really duplicates the experience of a scientific experiment. Probably the best one can do is to help the student to grow in the right direction, beyond procedures and a few numbers toward real observation and real care for what is being observed.

The third misconception about scientific experiments is exemplified in movies, where the scientist spends some time and effort, builds a new device or something, and tada! she discovers a new thing. It's presented almost as a guarantee: tinker around some, build something strange and new, and then - guaranteed! - you'll make a Discovery! I sometimes get the same feeling when we talk about national spending on research - spend x billion dollars and you'll get a proportionate return in discoveries. It's as if you could build science factories: build twice as many factories and you'll get twice as many discoveries. This sort of reasoning is true for engineering and industry where you know how things work already. It's not true about discovery: if it is really a discovery then that means that you did not know ahead of time what you were going to discover, or whether you would discover anything at all. And really the emphasis on Discovery! is inappropriate because of its emphasis on achievement and being right: really a scientist's business is to somehow discover where she is wrong - and the atmosphere is much more one of desperate hope for enlightenment. The more triumphalist one gets, the less one is humble about the mysterious depths one is just beginning to touch, the less one is able to do real science.

Experimental verification part II - Running Into Walls

I am reminded of episodes of Star Trek (I'm most familiar with its first incarnation) where it seemed like in every episode our intrepid explorers encountered an entirely new physical phenomenon. Gazing at his instruments and the starship's view screen, Spock says "Fascinating, Captain! A being of pure energy!" A minute later they begin conversing with the being. This is not faithful to human nature. When we encounter a physical phenomenon which we had not imagined before hand, as a rule we don't even notice it. We presume things are exactly the way we imagined them, and the only uncertainties are the ones we've already thought of. The process of noticing something is different than we imagine it to be requires not minutes but centuries - and the main way that it occurs is by us collectively running our heads into a brick wall many many times until our stubborness releases its clutch for a minute and our minds begin to bend around the newly perceived fact. Humans are not good at being surprised.

Long before the advent of physics many people including philosophers spent lifetimes and many manuscripts reasoning about the physical world and how it must be, developing elegant accounts of motion and inertia, heat and cold, germination and death, the stars and planets. These were smart people and after millenia their reasoning about politics, geometry, math, engineering, architecture, art, literature, and logic continues to be a foundation stone for the human race. However their reasoning didn't do so well when applied to the physical world around us, and after the advent of natural philosophy - physics - their ideas about motion and inertia, heat and cold, etc., have been swept away.

One key to natural philosophy's success was, I believe, that natural philosophers started to try to deliberately force their ideas and themselves into collision with nature. It was as if they realized that the only way to get through human thick-headedness was to have head first collisions with facts, and therefore decided to have as many collisions as possible. Natural philosophers deliberately construct situations where they have a very clear and precise idea of what nature will do, and then check that nature actually does what they expect. The human spirit - individually and collectively - wants to be right in these situations, and to obtain confirmations of one's rightness, and has a hard time recognizing evidence to the contrary. This is really the opposite of scientific progress, which is the process of getting one's wrong-ness smashed into one's intellect until one surrenders to the facts of what one is actually observing. Therefore natural philosophers spend a lot of time and effort - whole careers - trying to outwit their own hunger for rightness. This process is called experiments, or the experimental method. The main idea is to force yourself to actually see what is in front of your eyes, to really observe nature. You make a detailed plan of exactly what parts and chemicals etc. will be used, how they will be put together, and all of this gets written very carefully in a lab notebook. You also think out very meticulously what you expect to see, touch, smell, and why you expect it. You make sure to think of and write out as many details as possible - you are not interested in just one number, but in the entire apparatus, everything you will see not just one detail. If you are not attentive to every detail of what you observe then you can not be a good scientist. Then you actually do the experimental procedure, and keep careful records of everything. Later you compare your records of what you actually observed with your records of what you originally expected - you do this comparison in the spirit of an accountant, sweating over every missing penny. There is a theoretical possibility that you the individual researcher may notice something that clearly teaches you something new about the real world, if you're lucky. But much more likely the things she notices seem to to be matters of refining and perfecting her technique, of needing to think things out more clearly the second time, third time, thousandth time. The real learning occurs as many scientists run headfirst at the same wall over and over again, comparing notes as they go, perfecting their technique, and argueing their reasoning out with each other, until finally the group as a whole begins to catch on to a way they might be wrong, and to develop a feel for the real surprise in what they are observing.

Natural Philosophy

Of course natural philosophers - physicists - have the same philosophical bent. As a case in point, consider the efforts to come up with a Theory Of Everything, or a Grand Unified Theory. In those efforts physicists are trying to understand something that is far beyond human experience, with almost no connection to the real world, so that we have to build billion dollar machines if we want to hear even the slightest whisper from nature about these matters. Not practical at all.

But going back to the natural philosophers, you can imagine how it looked at the time when a person asked whether heavy things really fall faster than light things, or debated the commonplace knowledge that vacua can not exist in nature, or claimed that everything is made out of very small invisible indivisible pieces. This last was proposed by the Indians and Greeks but had no connection with real world evidence until two millenia later. These people clearly had an unworldly bent, the sort of attitude that is epitomized in so many pictures of Einstein.

But there was something a bit different about these natural philosophers, which made them far different from other philosophers, and still lies between the two like a canyon. The natural philosophers emphasized numbers, experimental verification of their ideas against nature's actual behavior, and complete control over experiments. I'll expand on all three.