Matt Legend GemmellModesty is Lying

As a freelance software engineer, I very frequently need to take a client’s specifications or requirements, and design a new piece of software (whether it’s a component or a full application) to meet those specs. Having to turn a feature-list into actual software architecture on a roughly weekly basis gives me some perspective on that process, and there’s an interesting pattern which always emerges. I call it the perceived complexity curve.

The Perceived Complexity Curve for software.

The curve may look slightly different from where you’re sitting, but the general shape is the important thing. The graph above shows the developer’s perception of how complex his current piece of software is, whilst he’s developing it. There are some interesting properties to note.

• Almost every app starts off at some nominal “average” level of complexity, or apparent “hardness”. Software varies massively in its nature and actual complexity, but at the idea and features stage, there’s a “normal” level. As the developer’s experience in software engineering increases, projects fall closer and closer to this “normal” level.
• As we first start expanding and refining a spec, and start thinking about an architecture, the apparent complexity of the project skyrockets; this is region A. I call this the Zone of Despair. It’s why most hobbyist projects never even reach what these days we’d call beta. Seeing the project’s actual scope bloom right in front of your eyes is a very intimidating experience, but we have to push through it.
• If we do push through, we reach the first inflection point on the curve; point 1. At this point, you’re firmly into software architecture design mode. You’re thinking about entities and relationships, maybe a bit about data models, possibly about the technologies you’ll use. This is your first real foot-hold; your first piece of firm ground. You have a well-defined discipline to apply that chips away at the apparent complexity of the problem and begins to reduce it to familiar patterns and tasks. Here we gather speed, and start to believe we might even finish this one early.
• Before we know it, we hit inflection point 2. The actual problems of our project are well understood at this point, and our chosen architecture has divided the intimidating spec into manageable, logically discrete chunks. The problem is, there are dependencies. Interconnections, relationships; coupling. Perceived complexity starts to ramp back up at this point, not in the same amorphous way as in the Zone of Despair, but rather in a broad-and-shallow landscape of all the hundreds of little things you have to do in order to make your architecture actually work. You’re in region B, the unavoidable fallout area of your design; I call this the Zone of Consequence. This is an incredibly energy-sapping, frustrating period, but we’re professionals: we must push forward.
• Thankfully, things never became as seemingly complex as they did at the start of the project, and at last we reach inflection point 3. We’ve designed and built the broad architecture, we’ve put in the plumbing and the electrical wiring, and now it’s a matter of running through our (lengthy) list of fairly clean and discrete tasks; the interior decorating, to continue the metaphor. This is the fill-in-the-blanks period where you’re rewarded based on the quality of your initial design and architecture – you’ve reached that comfortable, normal level of actual complexity which defines our entire type of work. The road ahead isn’t short, but it’s ten times clearer than it was before. If you can get your second wind, this could be the home stretch.

I encounter this pattern constantly in my own work; it’s a very interesting phenomenon, and probably applies to all tasks of more than trivial actual complexity. You have a task which starts off seeming simple, then upon further thought and exploration it becomes considerably more complex (with several nebulous unanswered questions). Then, upon still further examination, it becomes clear that the level of refinement of thought has lead to discovering once again that the problem is simpler than it seemed to be.

This cycle can continue of course, probably with ever-decreasing amplitude, but eventually it narrows in on a roughly average level of complexity which pretty much all non-trivial APIs (or problems) share. I find that fascinating, because it creates the proposition that expertise isn’t a superior ability to grasp complex concepts, but rather a high enough level of familiarity with the subject that the simplicity underlying the apparent complexity can be seen, and dealt with on that level. I suppose that the universe itself is the ultimate example.

There are important lessons to be found here about maintaining motivation and enthusiasm during the initial (often stalling) phases of difficult projects, and also about the importance of not starting to write code too soon, before you have a clear vision.

There are always exceptions, and your experience will be subtly different, but I’ve found this same pattern of perceived complexity appearing again and again in my work. The more projects you work on, the more you’ll start to feel the natural rhythm of software architecture design and engineering, and the more you’ll be able to feel where you are on the curve for your current project.

Any unforeseen or radical deviation from that rhythm is usually a sign that something is wrong, and is cause for taking a step back and allowing time for high-level review – before it’s too late.

(Footnote: A very interesting property of the curve is that it also holds true when you’re familiarising yourself with an existing codebase – even one of your own that you’ve not touched in a while.)

Tweet This entry was posted on Friday, June 11th, 2010 at 11:08 pm and is filed under Development. You can follow any comments to this entry through the RSS 2.0 feed. You can leave a comment, or trackback from your own site.

Tags: complexity, difficulty, implementation, programming, software