- An Interesting Problem
- An Uninteresting Problem
- A Knowledge Question
- The Engineer Growth Curve
- Slope of Productivity
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Slope of Productivity
When you finally do find your feet as an engineer, and really grow in skill and productivity, that's when you start to encounter the NIH syndrome again. This time it's in a new way: in the balance between finding an existing solution to a problem, and inventing a new solution. In other words, there's a balance between the cost of looking for the right solution to a problem—understanding its architecture, flaws, assumptions, and testing its performance and fit for your particular need—and the cost of building your own, very specific solution from scratch or from existing tools that you know and understand well.
This very real problem arises time and again. Striking the right balance between these two costs is an art form. I continually struggle with it myself. Only in hindsight do we realize that the effort spent on a custom solution was far too expensive and that an existing solution would have sufficed.
Let's consider some common symptoms of this problem and how to approach them.
As I mentioned earlier, there's generally a tradeoff between the amount of time it takes you to find, learn, and integrate a new API, versus simply writing it from scratch for your own very specific purpose. If this is the only motivation, I tend to err on the side of caution, usually putting aside a time box (say three hours), in which I make a cursory survey of available options and brief pros and cons. After this, a short list naturally emerges, which I can then spend more time on vetting exhaustively. In general, you shouldn't be doing the detailed investigation for more than one solution—or two at most.
It may even help to create a small experimental project that is a microcosm of your problem. For example, if you're investigating Java frameworks for WebSocket, you could create a really simple chat client that simply pings messages back to the browser. Generally this is enough to tell whether you're happy with the solution or whether adapting it to your needs is more trouble than it's worth.
Sometimes you might be tempted to consider writing from scratch, if that would take less time than the time box itself (a math formulas library, for example). I caution against this approach for a number of reasons. We always overestimate our speed in writing software. And generally we don't account for the time we'd need to clear all the bugs and corner-cases that are bound to pop up when actually tackling the problem.
Another important factor in your consideration is performance. Does the library do what you need to do, and do it fast? If writing by hand would yield a significant performance increase, it may be worth the effort. The issue also depends on whether the problem you're solving is in a critical path. The time box technique works well here, too. My approach is generally twofold: First, when using an open source library, I investigate the source code for any lurking gremlins (an overuse of thread synchronization in library code is definitely a gremlin). Then I write a small performance benchmark in a unit test that mimics the real-world scenario that interests me.
It's not always necessary to choose the solution that won the performance race. Many times it might be enough to know that there's little difference in performance between your options. For example, recently I was choosing between MongoDB and PostgreSQL as data storage options. MongoDB outperformed PostgreSQL by about 20% in my benchmark, but I was happy to select PostgreSQL because it was fast enough. Plus, my primary need was data durability, for which I was more confident of PostgreSQL.
As you can see, there are no clear rules for how to label something "bad" NIH or "good" NIH. Mostly it's a matter of trading off time versus maintainability, and balancing the effort to learn something new against the impulse to create an interesting solution of your own. There's too much chagrin against NIH in engineering environments. At the same time, it's not surprising, given that junior and senior engineers alike indulge liberally in rewriting perfectly good software, for no reason other than that it's fun.
So the next time you embark on a project and encounter a client or a fellow engineer who has been bitten with the NIH bug, take a moment to think about all the angles that have led him or her to this point. It can be a rewarding—or punishing—decision, more so than most you'll make in software.