The way science progresses has been the subject of many theories. Thomas Kuhn popularised the concept of paradigm. In a scientific field, a paradigm is a set of concepts that more or less fit the mass of experimental data with some models and theories. The whole set tends to protect itself from the accidents of new experimental facts and new theories that can occur from time to time. It takes what Thomas Kuhn calls a scientific revolution (overwhelming evidence in theory and/or experiment) to move from one paradigm to another.
Karl Popper also challenged what can be called scientific. His concept of falsifiability is now part of the criteria used to decide what academic field can be called scientific (think creationism, psychoanalysis, homeopathy).
On the other hand, Martin Harwit has shown how scientific discoveries were often made simultaneously and by independent people. For example, we can take Darwin and Wallace, or Einstein and Poincaré, as clear examples of great discoveries being made independently of each other. Nobel Prizes are often awarded to several independent discoverers of the same new scientific fact. For example, the Higgs mechanism has been found at least twice separately. The periodic table of elements was theorized many times, until it stabilized with Mendeleev. His model was superior to the others because it predicted new elements that were eventually discovered.
Here we think that ideas are generated in an environment. They are basically randomly generated then filtered in the brains of different people. Then they are tested by experiments in the environment of the world: General relativity had to explain the whole corpus of gravitational science set up by Newton, and it could explain even more (the precession of Mercury's perihelion for example). Then it predicted that matter could bend light, and Eddington measured this during a famous solar eclipse.
Science progresses by testing hypotheses in a specific way: ideas can lead to predictions in an experimental setting. These tests can be done anywhere at any time. They can be repeated by anyone. Science is the corpus of ideas that have been tested by experiments with a strict methodology, with repeatability, falsifiability, refutability and predictivity. Karl Popper insisted that scientific ideas cannot be proven. Instead, there must be a way to show how they can be refuted. There must be experiments that test the ideas. Otherwise, the ideas are considered outside the scope of science. This is why science has split from other ancient disciplines such as theology and philosophy.
In any case, we see Darwinian evolution at work in the scientific process. The experimental method sifts through all ideas. Ideas that pass the tests are selected and retained.
They are then passed on to the younger generation of scientists and, in the long run, assimilated by the public. The progress of science is now much faster because the corpus of science can be easily written down and transmitted.
This figure shows how new objects were rediscovered in the field of Astronomy (Martin Harwit) |