Simulated Annealing

“Picture a water droplet or a ball. Once it reaches one local minimum, it is stuck there forever unless disturbed by some out outside process. That is, whether hiking uphill towards fitness peaks or downhill toward cost minima, if one can only take steps that improve the situation, one will soon become trapped. But the minima or maxima by which one is trapped may be very poor compared with the excellent minima or maxima. The question is how to escape.”

  • Stuart Kauffman, At Home in the Universe

There is a fascinating branch of heuristic computation called “Simulated Annealing”, wherein a system is forced, excited up into random states, and then “cooled” again so as to “find” certain kinds of basins of “cost minima”. Such repeated “heatings” and “coolings” mimic, or some would argue, carry forth on another substrate, the processes undergone in annealing iron. The same algorithmic unfolding and refolding of steps that occurs molecularly when a smithy is heating, pounding and dipping a sword in water, is being carried out with data by programmers.

As Dennett describes the connection,

“The right level of explanation is the algorithmic level: As the metal cools from its molten state, the solidification starts in many different spots at the same time, creating crystals that grow together until the hold is solid. But the first time this happens, the arrangement of the individual crystals is suboptimal – weakly held together, and with lots of internal stresses and strains. Heating it up again – but not all the way to melting – partly breaks down these structures, so that, when they are permitted to cool the next time, the broken-up bits will adhere to the still-solid bits in a different arrangement. It can be proven mathematically that these arrangements will get better and better, approaching an optimum or strongest total structure, providing that the regime of heating and cooling has the right parameters.”

- Darwin’s Dangerous Idea

and as Stuart Kauffman describes,

[i]“Annealing is just gradual cooling…as the smithy anneals and hammers, the microscopic arrangements of atoms of iron are rearranged, giving up poor, relatively unstable, local minima and settling into lower-energy minima corresponding to harder, stronger metal.”

“…if the deepest energy minima drain the biggest basins, then as the temperature is lowered, the microscopic arrangements will tend to become trapped in the biggest drainage basins…Working iron by annealing will achieve hard, strong metal because annealing drives the microscopic atomic arrangements to deep energy minima.”

“At high temperature, a physical system jostles around in its space of possible configurations, molecules colliding with one another and exchanging kinetic energy. This jostling means that the system does not just flow downhill into local energy minima, but can, with probability that increases with temperature, jump uphill in energy over “energy barriers” into the drainage basins of neighboring energy minima.”
[/i]

What is compelling about this process is that it is about as physical a process – the smithing of metal – abstracted to about as abstract a process – heuristic computation – as can be imagined. Further what is of interest is that the process of annealing has long been a metaphor for personal growth and “man-making”, suffer and you will get stronger. Since the discovery of annealing, the metaphor has been applied to persons and societies, yet now we actually have a kind of analytic bridge between the two.

What happens in annealing is simply that through heating, atoms are forced up from their local minima, and in cooling, allowed to settle into deeper minima, a collective reorganization, a crystallization process that brings the metal closer to overall strength. The process though must be carried out carefully and repeatedly, for the intermediate processes actually make the metal more brittle, less strong, seemingly further from its goal. Through the process though, deeper minima are eventually found and the metal reaches its “optimal” state.

The ungodly hags from Hell in Aeschylus’ Greek Tragedy The Eumenides, in explaining their necessary role in society offer this version of annealing wisdom,

“…ksumpherei sōphronein hupo stenei”,

“It comes together to temper in the narrow.”

Or, as it can also be translated,

“It holds to be sober under woe.”

This bit of wisdom is about the coherence that collects in the basin, in the canals of life, brought on by the heat, the excitation of suffering. Persons and societies, according to the Eumenides, are annealed.

What occurs to me is the wisdom of the basin, the valley, the minima into which atomic parts, data, emotions, behaviors are urged to collect, by virtue of the annealing process, has similarity to the Tao-te-ching’s celebration of the Valley Spirit, the emptiness of the Valley (called the Mysterious Female), that brings all things into being. The very lowness of the land, generates, just as the lowness of minima in annealing - simulated or otherwise - generates the optimal.

“The gate of the Mysterious Female is called the “Earth and Heaven’s root.” On and on (like silk), it does not tire.”

  • Part I, sec 6

What is curious, and perhaps even instructive, is to think that one is always passing through annealing processes, forced up out of local minima, local understandings into which one has settled, only to be cooled and potentially settle into deeper minima. The learning curve of life can be seen this way. As with annealing, the process at any one point may very well produce a more brittle, more unstable state, a “worse” one. As with annealing, the thresholds and the art of cooling and heating are important factors in whether the deeper minima will be reached, whether the sword will be strong. If one would go so far as to stretch to Nietzschean thinking, the Dionysian excitation of oneself (and others) is integral to Apollonian coherence later to be reached. One cannot by purely rational, closer, closer, closer steps from a local perspective arrive at optimal states. The process is global and takes place on the edge of the criticality, a line between the sub-critical and the super-critical. What also can be called the creative.

Further as a note on the epistemological considerations. The nature of the basins, the parameters of the landscape are value driven and ever shifting. The nature of what constitutes a “minima” will determine the kinds of annealing produced. If you change the index under which minima are fixed, different basins will govern the product made. Knowledge is fashioned. Is annealed, but according to use and need.

“The deepest valleys ‘drain’ the largest region of the space of possibilities. If we think of the valleys as real valleys in a mountainous terrain, water can flow downhill to the deepest valleys from the greatest number of initial positions on the landscape.”

  • Stuart Kauffman, At Home in the Universe

The values that we operate under, the use we put our investigations toward, configure the kind of world produced, the kinds of basins into which the “space of possibilities” will drain.

A further thought. If one finds oneself contained in a local minima into which one has settled, consider it perhaps a partial placement within an annealing processes. This may very well be the “wisdom” behind the Tao-te-ching’s passage. Find the lowest place, the bottom of the valley. Excite yourself beyond local energy barriers, local understanding, and find a lower place still. Be annealed. Remember though that in annealing, the metal is never melted, and the cooling must be slow, and the process repeated.

Dunamis

I first came across simulated annealing in the numerical analysis bible “Numerical Recipes in C”. I was also fascinated by its so-literally-physical inspiration.

Your annealing-creativity analogy is (I think) pretty accurate – just as annealing is both a random and a systematic process, so creativity is a peculiar mix of randomness and systematicity. I just have one thing to add to the analogy: while annealing does a better job than most of finding global minima, it is also a slower method computationally. If your problem is simple and without local minima, it’s more efficient to use a logical, downward-step-by-step method. Choosing the best method depends on the problem at hand.

I wonder if in the same way the two contrasting methods of creativity and logic are adapted to different situations in life. Sometimes you need to “get out of your comfort zone” and try new things, just to see if there’s a valley over that next hilltop; and sometimes you need to grasp on to something you’ve found and follow it downward, logically, to its deepest depths.

How do you know which method to use and when? That requires global knowledge of the landscape. Only the gods can tell you…

aporia,

I wonder if in the same way the two contrasting methods of creativity and logic are adapted to different situations in life.

This is an interesting question, and I believe perhaps may come under the local vs. global distinction. The logical step-down may very well work as along as there is a way to “step down”. What may supercede this of course is that given the landscape, such stepping down may find minima that are not “optimal”, but logically appear to be so, from that local understanding. Further, is it not possible that the local logical procedures are also products of annealing processes themselves. The constraint that governs the annealing analogy is the inability, as you say, to see in a God’s eye fashion the global picture, a constraint that also follows the historically contingent, that is “local” nature of one’s language pursuits. If the annealing processes main draw back is its slowness, if applied to real world processes as the evolution of societies and knowledge over vast networks of interaction, this really isn’t a consideration. Or, doesn’t it make more sense to view “logic” as an annealed subsystem?

Dunamis