The 2014 PBS TV series, How We Got to Now is a good miniseries on improvements in glass-making, sewage, water management, etc. that serve as the material/organizational basis for this modern world.
At one point in the series, the host Steven Johnson, a kind of historian of innovation, reveals his idea of how innovation occurs and he focuses on mavericks whose breakthrough is not a sudden “Eureka!” moment, but rather what Johnson calls “a slow hunch.” In other words, the innovators struggle along with a partially understood sense of possibility, very inchoate in the beginning, that comes into better focus with the passage of years and decades, via missteps and boondoggles.
The science writer Arthur Koestler shines a different “flashlight” on this problem of intuitive creativity and its bearing fruit:
Arthur Koestler, CBE (UK: 5 September 1905 – 1 March 1983) was a Hungarian British author and journalist. Koestler was born in Budapest. His masterful book, The Sleepwalkers, is a kind of defense of the way people in the past benefited from a productive sleepwalking on their journeys to scientific advance.
The Sleepwalkers: A History of Man’s Changing Vision of the Universe is a 1959 book by Arthur Koestler. It traces the history of Western cosmology from ancient Mesopotamia to Isaac Newton. He suggests that discoveries in science arise through a process akin to sleepwalking. Not that they arise by chance, but rather that scientists are neither fully aware of what guides their research, nor are they fully aware of the implications of what they discover.
A central theme of the book is the changing relationship between faith and reason. Koestler explores how these seemingly contradictory threads existed harmoniously in many of the greatest intellectuals of the West. He illustrates that while the two are estranged today, in the past the most ground-breaking thinkers were often very spiritual.
Another recurrent theme of this book is the breaking of paradigms in order to create new ones. People—scientists included—hold on to cherished old beliefs with such love and attachment that they refuse to see the wrong in their ideas and the truth in the ideas that are to replace them.
The conclusion he puts forward at the end of the book is that modern science is trying too hard to be rational. Scientists have been at their best when they allowed themselves to behave as “sleepwalkers,” instead of trying too earnestly to ratiocinate.
Add to this overview the “creativity” discussion on The Charlie Rose Show in The Brain Series (2010), where Professor Eric Kandel, the Nobel-prize physiologist, states forthrightly that brain research has no idea about creativity and the prospect of explaining creativity in terms of the brain is very distant indeed.
The arrival of a “slow hunch” (Steven Johnson) and “productive sleepwalking,” as opposed to unproductive kinds of woolgathering (Arthur Koestler), are mind, personality and spirit issues, although they do have brain-chemical “correlations” that cannot be explained mechanistically.
Mysteries all have physical/chemical “correlations” but cannot be simplistically reduced to biochem or genomics.