Software is like everything else
The following was originally written in my notes app at 2:00 AM on a sleepless night about a year ago.
I have just rediscovered it. It is published here in its original form.
–
an organism is made of organs which are made of tissues made of cells, powered by organelles, machinery defined by genetic codes behaving according to chemical processes which abide by the basic laws of physics
when studying the natural sciences, you can choose a specialization at any of these various layers of abstraction. ecology, animal behavior, medicine, cellular and molecular biology, genetics, chemistry, and so on
computer systems have similar depth. a complex web of microservices with data flowing to and fro, each composed of a program composed of libraries implementing algorithms, derived from abstract mathematics, composed of code, compiled to instructions which control the operation of logic gates composed of transistors which may as well be magical runes imprinted on a silicon crystal, by fine lasers composed of their own magic, ultimately directing the flow of electrons below this pile of abstractions.
while the study of the natural world focuses on discovering the unknown, which mankind did not create, just as much mystery lies within the world of software. despite its relative youth and the fact that it has all been engineered by humans from the ground up, no single individual may yet understand its full breadth and depth.
while no scientist is expected to fully grasp concepts from two specializations as unrelated as, say, population ecology and molecular pharmacology, demanding similar breadth of a software engineer is commonplace. you must know how to properly architect a system of dozens of components, maintain all their dependencies, debug data races and performance issues, choose the correct data structures and algorithms, and manage resource constraints, all while satisfying whatever functional demands are being made of the system, in addition to implementing it securely and testing and monitoring the whole thing.
there was a time when software was simple enough that a system could be understood in its entirely by a single human mind, but not anymore. what was once a homunculus in a jar is now frankenstein’s monster. it is no longer wrought from nothing but dust, and instead assembled from myriad heterogenous parts. and as such it is something we cannot control. what was once inherently loyal, as limited in function as it may have been, is now corruptible. this is the price paid in exchange for the promise of limitless potential.
i regret to observe that the industry currently seems to be focused on building systems that we are even less capable of understanding or controlling. we are told to ignore the risk. instead demagogues suggest we follow the monster’s advice, to let it help us build even more complex systems we cannot understand. ignore its ulterior motive. let it perform its occult ritual.
–
a software project is also like a house. a home, if you’re lucky. the similarities between software and a building are engrained in borrowed vocabulary; for instance, “architecture.” though we do not carry the same professional qualifications of, say, a civil engineer, we call ourselves engineers nonetheless.
a building must have functional features. a door to enter through, windows for light and fresh air. a kitchen with hookups for the appliances needed to store and prepare food. restrooms with the fixtures and infrastructure needed for hygiene and sanitation.
a building must withstand its own weight, as well as the weather, and wear. it must meet certain standards. fire safety. codes. it should be maintainable and repairable.
a building must also be beautiful. it is not lost on me that this fact has been lost on many builders (and even buyers) in recent years (or even decades), but to many this is still essential. call me a romantic. from the architectural style to the paint color and fixtures, aesthetic choices are made at every step through construction.
finally, a building has a je ne sais quoi. it must be livable. it must feel like it was made by a human being. a special touch must tie it all together. this art has also been diminished in recent years but not all have lost the ability to sense it.
the architect must take all of these aspects into account. so must the software engineer. the relationship between the two fields has been discussed ad nauseum. the functionality and overall structure of the code are obvious parallels. it must be maintainable and stable, of course. less obvious are aesthetic choices - I’m not talking about ui/ux, but rather code style, naming convention, and the like. and finally, there’s the je ne sais quoi (which has been mentioned before too [1]).
the difference between the architect and the software engineer is that the former is not an electrician, and is not a city planner. meanwhile a software engineer plays these roles plus those of the carpenter, the plumber, so on and so forth, up to and including the inspector, exterminator, and maid.
a block of five condoized townhomes occupying the lot of a former single family home lacks the je ne sais quoi. it checks all of the boxes for features but there’s no room to live, let alone for any change. there may be a parking spot, but you don’t park in it because it’s too tight and you need to make an 8-point turn to access it in the first place. maybe the driveway is too steep so you scrape the bottom of your car. there’s no bathroom on the main floor, the bedroom upstairs gets too hot in the summer, and there’s nowhere to let your dog out to pee. the builder does not care because they do not have to live there.
software too can be built without regard to habitability, or without humanity. however, the difference is that the builder is indeed the inhabitant. having a human builder does not guarantee that they’ll instill these qualities into their product, but removing from the equation guarantees they will be missing.
–
technical debt is a swear word. it has at times been dismissed as myth but i maintain that it is a real thing. it accrues whenever you make a decision for short term gain, while making compromises that need to be addressed in the future.
the financial parallel here is obvious but it is not one I see discussed at length. the headline feature of any debt note is its interest rate.
we can assume that a competent software team can produce some amount of tangible value given some amount of time. this is the basis through which the entire industry operates. though obscured, underlying this relationship is some function between time spent and value added.
if a shortcut is taken today for a small profit, it has cost us nothing at all. when we take the time to fix it shortly later, it costs us a little but the leverage afforded by taking out debt would have still allowed the transaction to be profitable. if instead we add five new features which also depend on this shortcut, we have accrued debt.
the more likely a debt-ridden change is to become interwound with other components, the higher its interest rate. the work it would take to address it grows more quickly with time the more coupled it is. needless to say, it would be unwise to make a huge compromise in a central part of a system and leave it unaddressed for a long time. when left to rot, it becomes more expensive to repair.
many financial entities exist by virtue of accruing debt and using it to generate a positive carry. that is, a bank may pay you interest for your deposit, but they can loan it back out with more than a few points of spread and thereby earn a profit. despite the temptation, it is not sustainable to do this with software.
a bank collects more from its debtors than it has to pay its depositors because when it funds loans, it is taking on risk. that is, there’s a chance someone might not be able to pay them back. as a result, everyone has to pay the bank a little bit more. if one or two loans default, the bank remains solvent and it’s not a big deal. they have many other loans that they will continue to collect on.
it may be tempting to try to create a similar investment vehicle in a software project. keep borrowing time to launch new features. as long as the features are successful, the total value earned exceeds the cost of paying off the debt. this creates the illusion that the debt can be carried and allowed to grow without bound.
most software teams do not have the luxury of as diversified portfolio of investments as your average lending institution. the time saved today by taking on debt can likely only be invested into a single feature effort. if that feature isn’t successful, that debt still needs to eventually be paid off.
it only takes one failed launch for this structure to become unsustainable. eventually you’re taking on more debt for each feature than the value it can provide.
of course, if you can borrow time from yourself you can loan it to yourself too. taking the time to fix things now is cheaper than doing it in the future. these investments earn interest instead of accruing debt. you’ll be able to deliver more value per unit of time than before.
–