SoftwarePhysics

FRIDAY, OCTOBER 31, 2008

How to Think Like a Softwarephysicist

In this posting I would like to sum up softwarephysics and explain how you can use it in your daily activities as an IT professional.

The Basic Idea of Softwarephysics
In softwarephysics I have tried to imagine what a physicist would do if he were suddenly transported to another universe with similar, but slightly different, physical laws than the physical Universe we are all used to. This is essentially what happened to me in 1979, when I switched careers from being an exploration geophysicist, exploring for oil in the Gulf of Suez with Amoco, to being a professional IT person supporting Amoco’s production geophysical software. One very scary Monday morning, I suddenly found myself surrounded by all these strange IT people scurrying about in a controlled state of panic, like the characters in Alice in Wonderland. This was a totally alien terrain for me. Granted, I had been programming geophysical models for oil companies since taking a FORTRAN class back in 1972, but that was the full extent of my academic credentials in computer science. Fortunately, back in 1979 most of the other IT professionals I came across were also refugees from the physical sciences, engineering, mathematics, or accounting, but there was also a disturbing influx of young computer science graduates from the recently formed computer science departments of the major universities within the United States. Being totally lost in this new career, I naturally gravitated back to the physics, chemistry, biology, and geology that had served me so well in my previous career. I figured if you could apply physics to geology; why not apply physics to software? So like the Gulf of Suez exploration team at Amoco that I had just left, consisting of geologists, geophysicists, paleontologists, geochemists, and petrophysicists, I decided to take all the physics, chemistry, biology, and geology that I could muster and throw it at the problem of software, in order to help me cope with the daily mayhem of life in IT, by viewing software as a virtual substance whose behavior could be modeled, but not fully understood at the deepest levels. Again, softwarephysics is just an effective theory of software behavior that only makes valid predictions over a limited range of IT conditions and only provides a limited insight into the true nature of software. So in softwarephysics we adopt a very positivistic point of view, in that we do not care what software “really” is, we only care about how software appears to behave, with the intended goal of developing models that reproduce and predict software behavior. In all of my previous postings on softwarephysics, I have tried to illustrate this evolutionary change in the way people approach physics and the other physical sciences. In the 17th, 18th, and 19th centuries physicists mistakenly thought that they were actually discovering the fundamental laws of the Universe, which they thought were based upon real tangible things like particles, waves, and fields. According to Newton, light was a stream of particles, for Maxwell it was an electromagnetic wave, and for Einstein it was both. Now with QED, we think of light as a probability wave that simultaneously follows all possible paths as you observe your image in a mirror. Somehow, each photon seems to instantly perform an infinite number of calculations to determine where and how it will ultimately end its journey. So by the end of the 20th century, most of physics had dissolved into pure mathematics and worse yet, we now realize that all these theories and models are just approximations of reality, if reality even exists. In a similar manner, in softwarephysics I have tried to pattern software behavior after the very successful effective theories of physics, most notably thermodynamics, statistical mechanics, the general theory of relativity, and quantum mechanics. But always keep in mind that that these models are just approximations of software behavior. Do not fall into the same trap that the early physicists did, by confusing models of software behavior with what software “really” is. In fact, nobody “really” knows what software is because if you keep digging deeper you eventually cross over to the hardware upon which the software runs, and then you get into real trouble – just ask any physicist, because nobody “really” knows what the hardware is made of.

In The Fundamental Problem of Software, I laid down the three laws of software mayhem:

1. The second law of thermodynamics tends to introduce small bugs into software that are never detected through testing.

2. Because software is inherently nonlinear these small bugs cause general havoc when they reach production.

3. But even software that is absolutely bug-fee can reach a critical tipping point and cross over from linear to nonlinear behavior, with disastrous and unpredictable results, as the load on software is increased.

So as an IT professional, you have two major pitfalls to avoid – the second law of thermodynamics and nonlinearity. InEntropy – the Bane of Programmers, I described how software can be viewed macroscopically as the functions the software performs, the speed with which those functions are performed, and the stability and reliability of its performance. In The Demon of Software, I also showed how software could also be viewed microscopically, as a large number of characters in a huge number of possible microstates, or versions, and in both of these postings, I showed how the entropy, or disorder, of software tends to increase and its internal information content decrease whenever a change is made because of the second law of thermodynamics – there simply are far more “buggy” versions, or microstates, of a piece of software, than there are correct versions, or microstates, of a piece of software. InSoftwareChemistry, I carried this further by describing a model which depicted software as series of interacting organic molecules (lines of code) consisting of atoms (characters), each of which can be in one of 256 quantum ASCII states. In this model, each character or atom in a line of code is defined by 8 bits or electrons in a spin up ↑ or spin down ↓ state. In SoftwareBiology, I described how living things are faced with exactly the same problem as programmers, in that they must assemble complex organic molecules from individual atoms with very low error rates in order to perform the functions of life in a Universe dominated by the second law of thermodynamics. And inSoftware Chaos, I showed how both living things and software must also contend with the effects of existing in a nonlinear Universe, where very small changes to these organic molecules or lines of code create dramatic and unpredictable results. In Self-Replicating Information, I showed how both software and living things are forms of self-replicating information, and that is why there are so many similarities between biology and IT. So as IT professionals, there is much we can learn from the survival strategies that living things have developed over the past 4 billion years to overcome the second law of thermodynamics and nonlinearity.

Keep an Open Mind and Learn From the Other Sciences
The first thing you need to do is open your mind and learn from the other sciences. In my opinion, computer science is a very isolated science indeed, much more so than the other sciences, which also suffer from this malady. Indeed, inbred thinking is a dire threat faced by all the sciences, which severely limits the pace with which they can make progress. In Self-Replicating Information and the The Fundamental Problem of Everything, I tried to describe how the very conservative nature of meme-complexes severely restricts the adoption of new ideas, or memes, into a meme-complex. To some extent, this is a good survival strategy for a meme-complex, so that it does not pick up mutant memes of a dubious nature, but it seems that nearly all meme-complexes, and especially IT, take this conservative position to an extreme, and that prevents them from making valuable progress. Yes, there is always a risk in adopting a new idea, or meme, and that is why meme-complexes are so conservative. But there is also a risk in not adopting, or at least examining, a new idea or meme too. So the first thing you need to do as a softwarephysicst is to keep an open mind to new ideas from the other sciences beyond what you were taught in computer science classes. Do not fall into the “But we do not think about IT problems like that” trap, without at least thinking the whole thing through. Perhaps the source of all your problems has been precisely because you have not thought about IT problems in those terms!

Google and the Wikipedia are great ways to brush up on your general scientific knowledge. Another thing I like to do is to read the many popular books written by some of the greatest scientists in the world today. I read about one of these books each week. These scientists invested a great deal of their valuable time in writing these books, that could have easily been put to other uses, to help us all gain a better understanding of the Universe, and they certainly did not do it for the money! Finally, I like to visit the district library of my township to check out very heavy 800 page college textbooks on subjects like chemistry, organic chemistry, biochemistry, astronomy, cosmology, geology, molecular biology, cell biology, and the like. When reading these textbooks, I just read through the entire volume, skipping over all the chapter problem sets and not worrying about exams in the least. I really loved the university life of my youth, but being an old man free from the pressures of problem sets and exams, allows me to focus on the essence of the subject matter at hand, and unlike most university courses which only cover perhaps 30% of the volume, I can read the entire textbook at my own leisurely pace. As I said before, I frequently have thoughts of going back to my old physics department to change the plaque hanging on the department wall to read: “I can do the problems; I just don’t understand the material!”.

Use Softwarephysics as Your Secret Weapon
Next, if you are young and upwardly mobile, I would recommend keeping softwarephysics to yourself for your own personal use. As I originally explained in SoftwarePhysics, I decided to wait 25 years until the year 2004, before attempting to approach the IT community with softwarephysics a second time. My hope was that by 2004, the IT and computer science communities would have matured to the point where softwarephysics might be more favorably accepted than it was back in the 1980s. But I did not take into consideration the dramatic deterioration of the scientific curiosity and literacy of the American public over this same 25 year period. Indeed, I have found the marketing of softwarephysics in the 21st century to be even more difficult than it was back in the early 1980s, when most IT professionals came from a background in the physical sciences or mathematics. The formation of a very conservative computer science meme-complex in the 1980s, with its own set of accepted and legitimate ideas that preclude all others, did not help either. So I continue to be frustrated by strange looks whenever I try to explain to my fellow IT colleagues why they are being tormented so by software and how to avoid some of the pitfalls. I am 57 years old, on a glide path to retirement, happy where I am, and with very little to lose in a compassionately frozen career path, but if you are young and on the way up, spouting ideas about softwarephysics in meetings will definitely not enhance your upward mobility – believe me I was once young too. But you can still secretly use the concepts from softwarephysics to steer yourself clear of many of the pitfalls of IT, and that will definitely bode well for your career. So use softwarephysics as your secret weapon in tight situations, when you have to quickly make tough IT decisions. Your fellow IT colleagues will be trying to get by on IT common sense, and I have showed you how unreliable common sense can be in the Universe we live in. So to quote Rudyard Kipling - ”If you can keep your head when all about you are losing theirs”, you are obviously unaware of the situation, or perhaps, you are secretly using softwarephysics to get yourself through it all.

Take a Biological Approach to Software
In SoftwareBiology and Self-Replicating Information, we saw that software is a form of self-replicating information that has entered into a parasitic/symbiotic relationship with the business your IT efforts support. The business provides a substantial IT budget for the care and feeding of software and also for the promotion of its replication, and in return, the software allows the business to conduct its business processes and operations. Smart businesses view this as a symbiotic relationship, while less agile businesses view software in a more parasitic manner. For example, I was in the IT department of Amoco for more than 22 years, and at Amoco, IT was definitely viewed as a parasitic cost - just a necessary evil to its core business of finding, producing, refining, and marketing oil. On the other hand, my current employer, the best company I have ever worked for, views software completely differently. My current employer is in the financial industry, and since the only thing this business does is to store and process information, IT is core to its business processes and is viewed as a means of gaining competitive advantage. On the other hand, there is Amoco, the last of the mighty Standard Oil Trust companies to still use the infamous Standard Oil Torch and Oval emblem, but which is no longer with us, having finally managed to achieve its lifelong ambition of reducing its IT budget to $0.

In SoftwareBiology, we saw that as an IT professional, you are the equivalent of a software enzyme, greasing the wheels for the care and feeding of the software upon which your business runs. But the problem has been that over the past 70 years, we have not learned to be very effective software enzymes. The good news is that you can drastically improve your IT competence and productivity by adopting the same survival strategies that living things have discovered through 4 billion years of innovation and natural selection to overcome the second law of thermodynamics and nonlinearity. That will be the emphasis for the remainder of this posting.

Do Not Fight the Second Law of Thermodynamics
The second law of thermodynamics is really just a scientific restatement of Murphy’s Law – if it can go wrong, it will go wrong. The second law guarantees that all systems will naturally seek a level of maximum entropy, or disorder, minimum information content, and minimum free energy. In my day-to-day activities, I constantly see my fellow IT professionals taunt the second law of thermodynamics with abandon, like poking a sleeping bear with a sharp stick, only to see them gobbled up in the end. Living things on the other hand, do not fight the second law of thermodynamics, but use it instead to fashion the structures out of which they are made. In SoftwareBiology, we saw how living things compartmentalize things by surrounding them with membranes. This includes the external cell membrane itself, which is used to isolate the innards of cells from their external environment, like object instances in the memory of a computer, but also to surround the organelles within, like their mitochondria and chloroplasts, to isolate functions like the methods of an object instance. We saw that these very useful membranes are constructed by simply letting phospholipids form a bilayer which minimizes the free energy of the phospholipid molecules, like a ball rolling down a hill.

Thus, living cells use the second law of thermodynamics to construct these very useful cell membranes one molecule at time, with no construction crew on the payroll. In addition, they embed protein receptors into the phospholipid bilayer of the cell membrane in order to communicate with the outside world. Molecules called ligands that are secreted by other cells can plug into the socket of a receptor protein embedded in the phospholipid bilayer of a cell membrane, causing the protein to change shape as it minimizes its free energy. This shape change of a receptor protein then causes other internal cellular molecules to take actions. The receptor protein itself results from amino acids being strung together into a polypeptide chain by the transcription of a DNA gene into a protein via mRNA, tRNA, and ribosomes. Once the polypeptide chain of amino acids forms, it naturally folds itself up into a complex 3-dimensional structure by minimizing its free energy too.

+++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++@+++@++++++++++++++++++++++++++++
+++++++++++++++++++++@@@++++++++++++++++++++++++++++
OOOOOOOOOOOOOOOO@@@@OOOOOOOOOOOOOOOOOOOOOOOO
||||||||||||||||||||||||||||||||||||@@@@@||||||||||||||||||||||||||||||||||||||||||||||||
||||||||||||||||||||||||||||||||||||@@@@@||||||||||||||||||||||||||||||||||||||||||||||||
OOOOOOOOOOOOOOOO@@@@@OOOOOOOOOOOOOOOOOOOOOOO
++++++++++++++++++++@@@@++++++++++++++++++++++++++++
++++++++++++++++++++@@@@++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++

There is much to learn from this ingenious strategy from an IT perspective. IT professionals normally are in a constant battle with the second law of thermodynamics and we normally take it on in a deadly frontal assault, which gives the second law all the advantages, as it happily machine guns us down from a protected position. As you can see, living things use more of a guerrilla warfare approach to the second law, by sneaking behind the lines and tricking the second law into firing on its own troops. Instead of turning cars into piles of rust, the second law suddenly finds itself constructing beautiful and intricate structures from simple molecules!

One way for IT professionals to trick the second law is to always use templates, like the DNA, mRNA, and tRNA templates used in protein synthesis. My description of BSDE – the Bionic Systems Development Environment is a good example of how I used templates to generate code. In this tool, I went to the extreme of trying to fully replicate the protein synthesis steps used by living cells to generate SQL code from “genes” for the programs within an “embryo” application that developed within BSDE through embryonic growth and differentiation as the “genes” were turned on and off by a programmer.

Another example of using templates is to try to copy/paste code and text as much as possible. For example, I currently approve all the change tickets that Middleware Operations performs, and it is always a challenge for me to get the people who submit these change tickets to include accurate information. In WebSphere you install Applications into a WebSphere Appserver via an .ear file, like Customer.ear, that contains all the code required by the Customer Application. These .ear files are stored on a server controlled by our Source Code Management group. So in the change ticket, the requester has to specify the path to the required .ear files. Quite frequently, the paths and .ear file names are misspelled, which causes us all sorts of grief when we try to install the .ear files in the middle of the night. I keep telling change ticket submitters that all they have to do is navigate in Windows Explorer to the desired .ear file and then highlight the Address textbox showing the path to the .ear file and then copy/paste the path into their change ticket. Right-clicking on the .ear file and displaying the .ear file’s Properties allows them to then copy/paste the exact name of the .ear file too. I am always frustrated by the great reluctance on the part of change ticket submitters to follow this simple fool-proof process.

Another way to trick the second law is to minimize typing with your fingers whenever possible. Every time you type out some software or a Unix command on the command line, there is always the chance of making a mistake. For example, I currently support over 200 production servers, and also a large number of UAT, Integration, and Development servers. Trying to keep track of all those servers and what they do is a real challenge. I have a script on our home server called /home/scj03/p that sets my Unix environmental variables the way I like them and also about 400 aliases that significantly reduce my need to type. Reducing typing also speeds up my ability to handle tough situations. When we have a website outage, I will get paged into a conference call for the problem, and I might end up with 20 windows open to different servers and monitoring tools. Just logging into all those servers and getting to the appropriate directories can take a long time, so using aliases really helps speed things up. For example my “p” script sets:

alias g="grep"
alias ap1=”ssh apw01it.unitedamoco.com”
day=`date +'%C%y%m%d'`
alias apl="echo 'cd to the Apache log files for today';cd /apache/logs; ls -l | grep $day"

So if I want to look at today’s Apache webserver logs on webserver apw01it.unitedamoco.com for the customer instance, instead of typing:

ssh apw01it.unitedamoco.com
cd /apache/logs
ls –l | grep 021409 | custom

I just type:

ap1
apl | g custom

and I see:

cd to the Apache log files for today
-rw-r--r-- 1 apache apache 35599711 Feb 4 01:59 access.customer-public-14000.20090204.00h00m.log
-rw-r--r-- 1 apache apache 29864992 Feb 4 03:59 access.customer-public-14000.20090204.02h00m.log
-rw-r--r-- 1 apache apache 23817935 Feb 4 05:59 access.customer-public-14000.20090204.04h00m.log
-rw-r--r-- 1 apache apache 81466621 Feb 4 07:59 access.customer-public-14000.20090204.06h00m.log

I also have some WebSphere aliases that let me quickly look at the SystemOut.log for a WebSphere Appserver. For example, suppose the WebSphere Appserver is running on dwas99it.unitedamoco.com:

alias dw1="ssh dwas99it.unitedamoco.com”
alias l="echo 'cd to the WAS Appserver log directories';cd /data/WebSphere; ls"
alias clm="echo 'cd to logs directory and more SystemOut.log';. /home/scj03/bin/clm6"

The above clm alias calls a small clm6 script in my bin directory:

/home/scj03/bin/clm6
#!/usr/bin/ksh
#
if [ $# -lt 1 ]
then
echo " Syntax error - need the name of a directory"
echo " Example:"
echo " clm6 appserver1"
echo " Does a:"
echo " cd appserver1/ver6/logs"
echo " more SystemOut.log"
else
cd $1/ver6/logs
dir=`pwd`
echo "more $dir/SystemOut.log"
echo " "
more SystemOut.log
fi

So instead of typing in:

ssh dwas99it.unitedamoco.com
cd /data/WebSphere
cd orders
cd ver6/logs
more SystemOut.log

I just type:

dw1
l
clm orders

and I immediately see the SystemOut.log file for the orders WebSphere appserver:

************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.13 cf130745.06] running with process name UnitedAmocoCell\was6node1\orders and process id 164482
Host Operating System is AIX, version 5.3
Java version = J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20071007 (JIT enabled)
J9VM - 20071004_14218_bHdSMR
JIT - 20070820_1846ifx1_r8
GC - 200708_10, Java Compiler = j9jit23, Java VM name = IBM J9 VM

Since I have over 400 aliases, I have an alias that helps me find other aliases:

alias a="echo ' '; alias | grep -i"

So if I want to find all the aliases that have to do with our Apache webservers, I just type in:

a apache

and I see:

apb='echo '\''cd to the Apache Start/Stop scripts directory'\'';cd /opdata/apache/scripts; ls'
apc='echo '\''cd to the Apache servers directory'\'';cd /opdata/apache/conf; ls -l'
apcon='echo '\''cd to the Apache static content directory'\'';cd /data/content; ls -l'
apl='echo '\''cd to the Apache log files for today'\'';cd /opdata/apache/logs; ls -l | grep '\''Feb 04'\'
aps='echo '\''cd to the Apache servers directory'\'';cd /opdata/apache/conf; ls -l'
sa='echo '\''sudo su - apache'\'';sudo su - apache'

I have my /home/scj03/p script on all the servers I support, and I have a script send_p.sh that scp’s the p script to all the servers when I make changes to it. When I login to a server, I frequently have to sudo to a production ID that has permissions on the production files, such as “sudo su – apache”. After I do that, I press F2, which I have set to “. /home/scj03/p” in my telnet software to set all my aliases for the production ID.

Do Not Fight the Nonlinear Nature of Software
In Software Chaos, I described the nonlinear nature of software, in that small changes to software or the load that software processes can cause dramatic and unpredictable results. I also suggested that IT needs to stop thinking of software in linear terms, constantly looking for root causes, when none may even exist. What is the root cause of a tornado? The sad truth is that because of the nonlinear nature of weather, a tornado might simply be generated by a gentle wind blowing over a hill. IT needs to view software outages in the same way that the civil authorities view severe weather conditions like tornados. The first priority should be reducing the loss of life and the damage to property, by the constant monitoring of conditions and an early warning system that generates alerts for people to take action. You can always go back to look at log files and dumps to try to figure out what happened, but your first action should be to fail production traffic over to your backup software infrastructure, or to start bouncing impacted software components if you have no backup infrastructure. You should think of your IT infrastructure like the ER of a hospital. Your first priority should be to stabilize the patient and get them out of danger, then you can begin to look for root causes. First get the patient’s heart pumping again, then get him breathing, stop the bleeding, and only then start to look for bullets in his log files and core dumps.

In Software Chaos, we also saw that many nonlinear systems, like a simple pendulum, do have regimes in which they behave like nice controllable linear systems. Many times, if you overbuild your software infrastructure sufficiently, you can safely operate your software in such a linear regime. Unfortunately, in IT there has always been a tendency to push the software infrastructure beyond 80% of capacity to get your moneys worth. Many times this simply happens because the growth of the processing load overtakes the capacity of the infrastructure because rigorous capacity planning was not continuously performed.

Follow Standards
All living things use the same 20 amino acids to build proteins, the same nucleic acids in the form of DNA and RNA to store and process genetic information, the same simple sugars to build polysaccharide structures, the same metabolic pathways like the Krebs cycle to oxidize carbohydrates, the same fatty acids to make fats and oils - the list goes on and on. And these standards that all living things follow are not necessarily predetermined things. For example, you can build almost an infinite number of amino acids from their basic structure of:

H H O
| | ||
H-N-C-C-OH
|
R

by simply changing the atoms in the R side group. But of all these millions of possible amino acids, all living things from bacteria to humans, use the same 20 amino acids to build proteins. And they also all use the same bases A, C, T, G, and U for DNA and RNA and the same mapping table to map amino acids to the 3-bit base-pair bytes of genetic information. It even goes further than that. Many of the organic molecules involved in biochemical reactions can come in left and right-handed versions, called stereoisomers or optical isomers. To see how this works, hold up your left and right hands palm down. Notice that they both look completely identical, with the same number of fingers and the same size and shape. But are they? Imagine you lost your right hand in a car accident, and by some miracle of surgery, they replaced it with a donor hand, but the donor hand happened to be a left hand! This would present all sorts of difficulties for you, beyond just trying to put your gloves on. As an IT professional, you can easily simulate these difficulties by simply crossing over your hands on your keyboard. Now try to type some code! You will find that your fingers no longer line up properly with the keys on your keyboard. Similarly, right-handed organic molecules have a hard time lining up with left-handed organic molecules in a lock and key configuration. This is why nearly all of the amino acids used by living things are left-handed, or L amino acids, while nearly all sugars are right-handed or D sugars. Living things could probably have gotten by if they had reversed things by only using D amino acids and L sugars, but there had to be a standard set by the biosphere at some point to go with one or the other. Otherwise predators would not be able to eat their prey, without worrying about the arbitrary coding standard their prey happened to have decided upon!

This is a valuable lesson for all IT professionals – use standards whenever possible. This can be applied to all the artifacts of IT – design documents, variable names, method names, database names, database column names, job names, job schedule names – the list goes on. For nearly 30 years, I have watched IT people suffer the consequences of the expediency of the moment. They plunge ahead in the heat of the moment to get something quickly into production, with the sincere intent of coming back to fix everything to conform to standards, but they never seem to do so. If you learn nothing more from softwarephysics than the value in following standards to overcome the effects of the second law of thermodynamics and nonlinearity, as all living things do, you will find it well worth your effort. Too much of IT is hand-crafted and non-standard.

If It Ain’t Broke, Don’t Fix It!
Living things evolve through small incremental enhancements that always improve their survivability. Richard Dawkins calls this Climbing Mount Improbable (1996). In this book, Dawkins depicts survivability as a mountainous landscape, with foothills and valleys leading up the slope. Living things do not jump straight up the face of a steep cliff in this survivability terrain, instead, they follow a slow arduous upward path in their assent to improved survivability. They also do not descend into valleys of lower survivability, which might have an easier path to higher survivability on the other side of the valley, because living things cannot “foresee” the terrain of the survivability landscape lying ahead. Natural selection only lives for the moment and selects for advantageous mutations and against disadvantageous mutations at every point along the landscape. Consequently, living things can only move upwards in climbing mount improbable, or stand still if nothing else. There is no selective advantage to decreasing your chance of survival by descending into a valley of lower survivability. Now there is both good and bad in this strategy from an IT perspective. As an IT professional, you are not limited to this path of small incremental enhancements in your climb up mount improbable, because you are privy to an expanded view of the survivability terrain of software. This gives you a better view of the hills and valleys leading up the mountain than living things are privy to, but sometimes it is also advantageous to take the more limited view that living things have in formulating IT designs. So when somebody comes up with a brilliant enhancement in a meeting that sounds like a good idea, I frequently put the question to myself – “Would this enhancement naturally evolve in nature? What are the selective advantages of this approach? What are the selective disadvantages?“.

Here is a recent example, in WebSphere there is a plugin-cfg.xml file that is installed on Apache webserver instances that tells Apache where to send WebSphere requests that it receives from browsers over the Internet. These requests are sent to physical WebSphere servers hosting WebSphere Applications installed in WebSphere Appservers. Now a typical physical Apache webserver might have 20 Apache instances serving up webpages for the many websites being hosted by the physical Apache webserver. Prior to WebSphere 6.0 we had just one communal plugin-cfg.xml file for all the Apache instances on a physical Apache webserver. The danger was that when we used WebSphere to regenerate the plugin-cfg.xml file for changes to Application A, there was a slight chance that it could mess up the plugin-cfg.xml file section for Application B. However, in 5 years of WebSphere administration, I do not recall a single instance of this actually happening. To overcome this non-existent problem, in WebSphere 6.0 IBM came up with an enhancement that allows for separate plugin-cfg.xml files for each of the 20 Apache instances we might currently have on an Apache webserver, providing complete isolation between the Applications being serviced by the Apache webserver. Now on the face of it, this sounded like a great idea to all, except to me. I immediately thought to myself, “Would this enhancement have a chance to evolve in nature? Would it be viewed as a positive mutation, enhancing survivability, or as a genetic disease to be ruthlessly eliminated from the gene pool?” My conclusion was that this enhancement would cause all sorts of problems because of the second law of thermodynamics and nonlinearity, with no offsetting advantages. My fear was that the developers who created our change tickets would forget to mention which of the 20 Apache instances we should map their WebSphere Applications to and that our Offshore team members doing the installs in the middle of the night, would occasionally get mixed up and accidentally copy the wrong plugin-cfg.xml file from one of the 20 directories storing plugin-cfg.xml files on the WebSphere Deployment Manager server to its corresponding directory on the 8 Apache webservers. You see, the WebSphere Deployment Manager server now has 20 directories storing plugin-cfg.xml files and each of the 8 Apache webservers also have 20 directories storing plugin-cfg.xml files, so the number of permutations of messing this up are huge. With WebSphere 5.0 there was only one plugin-cfg.xml file stored in one directory on the WebSphere Deployment Manager server and only one directory on each of the 8 Apache webservers storing the single plugin-cfg.xml file. This was pretty hard to mess up and it never happened over the course of five years. Now with the WebSphere 6.0 enhancement, we had 20 plugin-cfg.xml files to worry about on the WebSphere Deployment Manager server and 20 plugin-cfg.xml files on each of the 8 Apache webservers! Just do the math to come up with the ways you could mess that up! And sure enough, one pager cycle when I was Primary, we got hundreds of calls into our call center about people getting intermittent 404s when trying to login to our website. Naturally I could not reproduce the problem. It turns out that our install team accidentally copied the wrong plugin-cfg.xml file into one of our 8 Apache webservers, causing a 1 in 8 chance of generating a 404. This was a very difficult problem to troubleshoot because I never had this problem when there was only one plugin-cfg.xml file used by all the Apache instances, so I never even suspected a bad plugin-cfg.xml file. Now do I find fault with our install team for this mishap? Not in the least. Our install team does a marvelous job with the very difficult task of doing production installs. Too many times in IT we needlessly taunt the second law of thermodynamics when we tell people to avoid mistakes by simply “being more careful”. This is just setting people up for failure, like poking the second law of thermodynamics with a sharp stick. To alleviate this problem we now have a script that automatically pushes out the plugin-cfg.xml file from a menu. All you have to do is pick the correct plugin-cfg.xml file listed on the menu of 20 plugin-cfg.xml files. Even so, this requires a huge amount of intellectual energy to do correctly. The developers have to specify the correct plugin-cfg.xml file to regenerate for the Apache instance their Application uses on their change tickets and our Offshore team has to pick the correct plugin-cfg.xml file from the menu at install time. My contention is that this “enhancement” would be viewed in nature as a genetic disease, with a great deal of selection pressure against it because it consumes energy and resources, but provides no competitive survival advantage.

Follow the Lead of Bacteria for Day-To-Day Activities
As I pointed out in SoftwareBiology, prokaryotic bacteria are the epitome of good IT design, consequently, bacteria are found to exist under much harsher conditions and under a much wider range of conditions than multicellular organisms. We saw that bacteria block their genes to minimize I/O operations and do not load up their genomes with tons of junk DNA and introns, as do the “higher” forms of life. The simplicity, resourcefulness, and frugality of bacteria are to be admired from an IT perspective. So for the mundane day-to-day issues of IT, I frequently ask myself “What would a bacterium do in this situation?”.

Follow the Lead of Multicellar Organisms and the Biosphere for Architectural Issues
For higher level IT issues look to the superb organization found in multicellular organisms and the biosphere. For example, we are currently going through the SOA – Service Oriented Architecture revolution in IT. With SOA, we develop a large number of common services, like looking up the current status of a customer, which can then be called by any number of applications, rather than have each application individually code the same identical functionality. As we saw in SoftwareBiology, SOA is just the IT implementation of the dramatic eruption of multicellular architecture during the Cambrian Explosion 530 million years ago. Consider the plight of a single, lonely, aerobic bacterium which is challenged to find its own sources of food, water, oxygen, and warmth and must also find an environment in which it can dump its wastes without poisoning its immediate surroundings, and do all this while it tries to avoid all the dangers of a hostile microbial world. Contrast that with the relatively fat and easy life of the 100 trillion cell instances in your body that find themselves bathed in all the food, water, oxygen, and warmth they could desire, without the least worry about dumping their wastes into their immediate environments because their wastes are briskly flushed away, and without the least worry of being taken out to lunch by a predator. The poor little bacterium must have code that handles all these difficult situations, while the cells in a multicellular organism just rely on the services of other cell instances residing in SOA service machines, or organs, known as hearts, lungs, intestines, livers, kidneys and spleens to handle all these common services. In Software Symbiogenesis and Self-Replicating Information we saw how the biosphere has taken SOA to an even higher level. Not only do the cells in a multicellular organism rely upon the services of cells within the organs of its body, they can also rely upon the services of cells in other bodies via parasitic/symbiotic relationships between organisms. We are beginning to see this in IT too, as companies begin to expose services to other companies over the Internet to allow B-to-B applications to form symbiotic relationships between businesses in a supply chain.

I became convinced of the value of SOA many years ago at United Airlines, before SOA had even appeared on the IT scene. At the time, I was supporting the middleware for www.united.com running on Tuxedo. Through a process of innovation and natural selection, the middleware group ended up writing a large number of Tuxedo services to access things like customer data from a series of databases and reservation data stored in the Apollo reservation system to support www.united.com. We then opened up these Tuxedo services for other internal applications at United to use via a JOLT call from their java applications. Over a period of a year or two, we accidentally evolved a pretty sophisticated SOA architecture. This SOA architecture was put to the test one day, when my boss got a call from our CIO. It turns out that our marketing department had published a “Fly Three, Fly Free” campaign in hundreds of newspapers that day. The campaign explained that if you flew three flights with United in any given month, you were then eligible to fly free to any destination within the continental United States. All you had to do was to register for the campaign to obtain your flight benefit. Now that was the problem. Our marketing department had forgotten to tell IT about the new campaign, and in the newspaper ads, it stated that the promotion was going into effect that day at 12:00 midnight! So we had 15 hours to design, develop, test, and implement an application to be used by our call centers and that could also be run from www.united.com. We immediately formed a number of crash workgroups in applications development, database, QA testing, and call center training to field this application as quickly as possible by working in parallel as much as possible. And because we could sew together an application quickly by calling our already existing SOA Tuxedo services, we actually made the 15 hour deadline! After that experience, I was totally sold on the idea of SOA, before SOA even existed.

Let Softwarephysics Guide Your Career Path
SOA has the advantage of reducing the amount of redundant code that a corporation has to maintain and allows developers to quickly throw together new Applications because they can just assemble them from a collection of existing components or services. The problem that SOA presents is governance - who decides how and when to change components? SOA can also be a problem for Operations when troubleshooting problems because many applications draw upon the same services, and a bug in one service can impact many applications. So is SOA a good thing or a bad thing? And how do you decide?

Well there was this same controversy about object-oriented programming in the early 1990s. Was object-oriented programming a good thing or a bad thing? And was it just a flash in the pan, or something that would catch on? There is a lot of hype in IT, and you can easily get misdirected down the wrong architectural path. For example, in the early 1980s they told us to start looking for a different job because IT would soon be vanishing. You see, in the early 1980s, the theory was that 4GL languages (4th Generation Languages), like Mark IV and Focus, would soon allow end-users to simply write their own software, and the Japanese were also working on a 4th Generation Computer, due to be completed by 1990, that would be based upon an operating system using Artificial Intelligence. End-users would interact with this new operating system like the folks in 2001 - A Space Odyssey, with no need for pesky IT people. But none of that ever happened! So in the early 1990s, the COBOL/CICS/DB2 mainframe folks had a dim view of object-oriented languages like C++ because they thought they were just another flash in the pan. There was also a lot of FUD (Fear, Uncertainty, and Doubt) in the early 1990s because of the impending architectural change to a distributed computing platform which made matters even more confusing. We saw folks throwing together LANs with cheap PCs, and then there were those nasty Unix people bringing in Unix servers to replace the mainframes too. Mainframes were supposed to be gone by 1995, and IBM nearly did go bankrupt in those days. So maybe Unix and C++ might be something good to learn - or maybe not?

So how do you make sense of all this? How do you make good IT decisions about your career path when IT architectural changes seem to be happening, but there is all this hype? Fortunately, I had softwarephysics back in the early 1990s to help me make decisions. Based upon softwarephysics, I taught myself Unix, C++ and Java (1995) when it came out, because softwarephysics told me that object-oriented programming was a keeper and would have a big impact on IT. Sofwarephysics also predicts that SOA is the way to go too. Softwarephysics claims that object-oriented programming and SOA are really just implementations of multicellular organization in software. An object instance is just a cell. The human body has 210 cell-types or Classes. Each of the 100 trillion cells in your body is an object instance of one of these 210 Classes. Each cell is isolated by a cell membrane that protects it from other cells, so that other cells cannot directly change the cell's "data". In object-oriented languages, this is called encapsulation. Each cell contains many internal methods(), used by the cell to conduct biological operations. Each cell also has a set of public or exposed methods() that can be called by other cells, via the secretion of organic molecules, that bind to receptors on the cell membrane and induce the cell to perform biological processes. The cells in your body also use the object-oriented concepts of inheritance and polymorphism. The 100 trillion cell instances in your body use the SOA services of other cell instances, like the cell instances in your lungs, liver, kidneys, heart and intestines. These organs, or service machines, provide your cell instances with oxygen, food, and water, and also dispose of the wastes they excrete. So given the success of the multicellular architecture found in the biosphere, softwarephysics predicts that object-oriented programming and SOA will also be successful and worthy of investing some time in learning.

Conclusion
As you can see, softwarephysics is not a magic bullet to cure all of your IT woes. It is really more of a way to think about software. Because it provides an effective theory of software behavior, softwarephysics can be a very robust and flexible tool in your IT arsenal. All you have to do is use your imagination!


Comments are welcome at scj333@sbcglobal.net

To see all posts on softwarephysics in reverse order go to:
http://softwarephysics.blogspot.com/