Allan Clarke

Mastering Agile Development: Debunking the Field of Dreams

Mastering Agile Development: Debunking the Field of Dreams

Over my decades of a career as developer and software architect, I have seen many projects, products, and companies come and go. I have always been struck by a repeated mistake that seems to be both widespread and pernicious. Not only have I suffered this in the past, but I’ve have heard similar laments of my colleagues in the industry.

I call this mistake the “Field of Dreams” or the “build-it-and-they-will-come” belief. The concept is that if you have a clever, technically cool, or highly utile concept, then commercial success is only a formality. The greatness of your concept will be sufficient to attract customers. One of these clever products I’ve worked on was a decade ahead of the market (not really a position you want to be in).

This aberration seems to be realized in one of two ways: The Great Idea or The Platform.

The Great Idea

The great idea often originates from academia, a research group, or a sole inventor. The idea can be described in terms of its implementation prowess to excite technical types. The idea can be described by its cleverness to potential customers who clearly agree that it is a cool idea. The great idea often solves some difficult technical problem and potential customers agree that the solution could be useful. Management eagerly takes that agreement as a validation of commercial viability.

The Great Idea – Redux

The abstract problem with The Great Idea is that there is no correlation to its business value. Sometimes even the customer does not know how much a problem costs them, so cannot validate commercial viability. To complicate things further, technology always shifts, causing changes to the boundaries of the problem, and hence the value.

An often-overlooked issue is understanding of the personal cost to potential customers. To use the product, does a potential customer have expertise on staff? Do they have to hire another DB admin or another network guy?

When customers are shown The Great Idea and asked if it is valuable, they rarely push back because they don’t have any “skin in the game” – it doesn’t cost them a thing. For every ten companies that have The Great Idea, nine will fail and be forgotten. The 10th will succeed and others will point to that success as proof of the righteousness of taking the path of The Great Idea.

The Platform

This second variant manifests when a potential product tries to be all things to all people. Its primary selling point is its utility, its flexibility. It is a high-tech box of Legos. It can be configured, scripted, or programmed to solve not just one customer problem, but a multitude. The ostensible leverage is that it will, therefore, payback multifold. Customers can either use their own staff or can rent expertise from the vendor to reconfigure the product.

The Platform – Redux

There are several problems with The Platform. The cost of hiring expertise is under-appreciated. Too late, customers realize that The Platform solution has really become a long-term project. The Platform must support use cases that can vary greatly between customers. The wider the variance in use cases, the more generic the implementation must be – sometimes to the detriment of solving specific, value-dense problems. Customers don’t want a box of Legos dumped on their desk – they want a problem solved.

As an afterthought, you are better off focusing on a few use cases as you can manage. Usually the more use cases you cover, the more diluted your efforts will be, and the less value there will be to customers. Unless you have deep pockets, this dilution is a real risk.

Backwards Thinking

This whole “build-it-and-they-will-come” is a glaring instance of backwards thinking. If you want to maximize your chances of commercial success, you *must* talk to customers and identify pain points as a first step. You will need to match those needs with your domain of expertise, then you will need to go back to potential customers. You need confirmation that they will spend X dollars with you now and are good for Y more dollars in the future. Only then will you have validation of commercial viability.

To learn more about how Lumina helps our customers build innovation-ready platforms, visit

5G NetDev Services


Mastering Agile Development: Hercules in the Age of Agility

Mastering Agile Development: Hercules in the Age of Agility

I have been in the software development business for well over twenty-five years and have seen many successful and unsuccessful projects. The past ten years have been on teams following Agile principles to varying degrees. The teams have always been composed of different levels of experience and expertise, and I have been very fortunate to have been on teams that were mostly experienced developers, and importantly open-minded developers.

Over this span, I have seen patterns of behaviour emerge that, on the surface, seem beneficial to the team, the product, and the company, but in reality come with a lot of unrecognized costs. These costs are not planned for, not quantified – they are unmanaged. Oftentimes they are never historically reconciled.

There is one particular behavioural offender in this developer ecosystem and it might surprise you. I’m talking about the hero developer.


Meet Hercules, a member of our mythical development team. Hercules fits fairly well into the team, communicates reasonably, and is a competent developer. He gets his work done consistently. But where he really shines is when there are looming, seemingly impossible deadlines. He is the one that stays up into the wee hours, throwing himself across the finish line just-in-time.

He is the hero. He is the one who gets the adoration of managers and that good ole back slap with an “at-a-boy”. Honestly, he really loves the attention and approval from his manager and the awe from the junior developers who hope to grow up someday to be like Hercules.

Managers see the deeds of Hercules and know that this is how successful software is created. If only they had a bunch developers just like Hercules, their software problems would be solved.

A Deeper Analysis

First, let’s acknowledge that being dedicated to your team and working hard are assets and that these assets are not found in all developers. But when we look at Herculean deeds from an Agile point-of-view, we start to see issues that can undo all of the gains that came from the providence of Hercules.

Hercules specializes in cranking out code in a short amount of time. Sometimes it seems the more pressure, the better. Hercules knows that code. Due to time pressure, he didn’t get to do a lot of the unit or integration tests, but through sheer ability to focus, he delivered today’s functionality, and that’s what’s important, right?

Hercules sure knows that code, but the other developers do not. The agile developer wants group code ownership. They want group input on design choices. They want the most well thought out, flexible code that the company and project can afford (girded by time and money). The agile development team wants to de-silo the implementation, meaning that multiple developers are familiar with the code so future changes occur at a reasonable cost. But given the supreme, single source effort, the odds go against this.

A good hero sees heroics as an aberration – something to be avoided if possible. But I have seen cases where heroes use heroics as a path to job security. They hoard information by being the only custodians of said code, such that others recognize their area expertise. They want to make sure that there is a large painful hole in the development team should they ever leave. Naturally, job security like this is good for them and really bad for the company.

Another issue involves scaling the team. If you have a Hercules, it is almost impossible to clone him/her. You should be able to find developers, but if you need one to fit into a chaotic, barely-managed environment, you need another Hercules, and they are both harder to find and cost more money.

The final drawback is a bit more subtle. Tom Demarco’s book “Slack” ( does a good job of discussing the issue. In a nutshell, if you have heroics as your normal means of delivering software, your heroes are 110% busy – and this means time for creativity and giving back (like mentoring) fall by the wayside. This cost is never recognized by management but can have a very large impact on the business over time.

Some Advice

Hire as many heroes as you can and avoid heroics at all costs.

5G NetDev Services

Top 10 Things For New OpenDaylight Developers

I recently started contributing upstream to the Open Daylight project (ODL) as a developer. I mostly followed the ODL developer documentation on how to get started and how to use the tools. Through ambiguous documentation or through hubris (every developer always thinks they know what they are doing), I made some mistakes and had to learn some things the hard way. This article is an enumeration of some of those hard knocks, just in case it can help the next person following this path.

First, this list is more than ten items. It is just that “top 10” has a catchy sound to it that just isn’t there for “top 17”.

Gerrit Workflow

Gerrit has a different workflow that you are likely using in downstream coding with other tools. There is a general tutorial and an ODL coding introduction to working with Gerrit that is very helpful.

Coding Conventions

We followed our own coding guidelines and those did not match to the Google coding conventions used upstream. Differences we ran into were:

  • Lower-case method names, even for JUnit test methods
  • Using the most restrictive visibility for access qualifiers (I won’t push back the philosophy of library design)
  • Use Java 8 lambdas everywhere possible

Gerrit Commands

Gerrit (plugin) has some non-obvious controls, namely that you can kick off another build by putting “recheck” as your comment. Others are “rerun”, “verify” as in here.


Upstream coders usually add the prefix “WIP: ” to their bug report to let other developers know things are not ready for prime time reviews yet. I have been putting the “WIP:” prefix as a new comment right after my new patch set.


Mechanically you can review the issues by using “Diff against:” drop list to pick a revision to start at then go into the first file and then use the upper right arrows to move between files.

Broken Master

The master branch on ODL can and does break, causing lots of down time with broken builds (days even). Full builds take 10 hours and verifying-before-pushing is an evolving process. Have patience.

Git Review

If you are using “git review” and you forgot the “- – amend” option on your commit, you will be prompted to squash and will end up with a new branch. You can recover using “git rebase -i HEAD~2” then using the force option and then abandoning the new branch.


Along with master breaking, master can also produce stale artifacts, so don’t assume that clearing your m2 cache, creating a subproject repo and building will give you up-to-date code.

Searching Jars

You can search jar files using find . -type f -name ‘*.jar’ -print0 |  xargs -0 -I ‘{}’ sh -c ‘jar tf {} | grep “YourClassHere” &&  echo {}’ for helping verify your m2 has the latest.

Patch Workflow

The patch workflow is not very good at all for having multiple people working on an item; traditional git branches and management are superior in this respect so expect some pain.

Broken Master Again

If your patch won’t build because master is broken, you can rebase it to pick up a fix temporarily. When you do a “git review” to push it, you will be prompted because there are two commits (your new patch revision and the fix). You do not want to merge that fix into your patch.

Skip Tests

There may be a bug here or there in dependencies, so you should always to a full build within a subproject the first time after you create the repo. In my case, I was in netconf and saw a difference between starting off with a “-DskipTests” or not. The former lead to a failed build, while starting with a full build (and then doing any number of skip tests), seemed to work.


If you are a developer who works with meaningful coding standards, you will find yourself clashing with the pedantic nature of ODL’s use of check style. Although it probably varies from project to project, your code reviewer might decide that your code is the perfect opportunity to enforce check style.

Bug Report

Put the URL to the gerrit issue in “external references” on the bug report and put the bug as the first part of the commit message like “BUG-8118: ….”

Gerrit Patch Set

Any replies that you make to the patch set discussion use the current patch set as an annotation. Be sure to move your “current patch set” by using the upper right drop list “Patch sets 8/8” (for example).

Easy Debug

You can do development on a project like netconf and start a controller with your code: git clone, mvn clean install, go to netconf/karaf/target/assembly/bin, do ./karaf, at console install your feature, like feature:install odl-netconf-callhome-ssh (and maybe odl-mdsal-apidocs). Viola!


If you work on the docs, then you need to know there are three kinds: contributor (on wiki), developer, and user (both on

Getting Started Upstream in ODL

By Allan Clarke and Anil Vishnoi



Getting started as an upstream contributor to OpenDaylight is not easy. The controller ecosystem is big, there are many projects, and there are millions of lines of code. What is a new ODL developer to do? Here is some pragmatic advice on where to begin to become an active contributor.

Fix Bugs

One of the easiest ways to get to know a code base is to start fixing bugs. Peruse the ODL bugs list on Bugzilla, say with the NETCONF project. You want to find bugs that aren’t likely being worked on and are of limited scope (to match your limited understanding of the project). Ideally bugs will have an owner assigned to indicate that they are actively being worked on, but it is not always a great indicator. In particular, someone may run across a bug, file a report, then jump into fixing it—and forget to assign it to themselves. This is most likely with the project contributors, so figure out who are the project contributors and look at the date of the report. If it was a project contributor and a newish date, then that bug might be being worked on. You should read through the report and try to decide how much domain knowledge is needed—as a newbie, smaller is better.

Once you have selected a bug to work on, click on the “take” link. Also add a comment to the bug. If someone already is working on it, they should get a notice and respond. You can also try the ODL mailing lists and give notice there. You mainly want to avoid duplicate work, of course.

Review Patches

Reviewing patches is a great way to contribute. You can access patches via Gerrit, and we’ll use the NETCONF patches as an example. Doing code reviews is a great way to not only see existing code but also to interact with other developers.

  • If you have some domain expertise and know the code, you can review the functionality that is being pushed.
  • If you have neither of these, you can do the review based on Java best practices and good software engineering practice.

Address Technical Debt

ODL uses Sonar for analytics of the upstream project. Here is an example for the NETCONF issues. Note that the ODL project has coding conventions, and the Sonar Qube has some best practices. This list shows violations that should be addressed. As a newbie, you can work on these with little domain knowledge required. You can also see that the code coverage varies for the NETCONF coverage, so adding NETCONF unit tests to boost the coverage in the weakest areas would be very helpful.

Sonar has a lot of interesting metrics. You can explore some of them starting here including coverage, tech debt, etc. If you look at the Sonar dashboard, it will point out a lot of available work that does not require a large span of time to invest. Doing some of this work is a great step towards getting your first patch submitted.

Follow Best Practices

With well over a million lines of code and many contributors from many companies, the ODL project has quite a girth. To manage the code entropy, ODL has some best practices that you should become familiar with. These cover a diverse set of topics, including coding practices, debugging, project setup and workflow. We strongly recommend that you carefully read these. They will save you a lot of time and will pay back your investment quickly. They will help you skate through code reviews. These practices are really time-tested advice from all the ODL developers, so don’t ignore them.

Support Attribution

Attribution is an important insight into most if not all open source projects. Attribution allows stakeholders to see who is contributing what, from the individual up through sponsoring companies. It allows both a historical and current view of the project. You can see an example of why attribution is illuminating here. You need to sign up for an ODL account, and a part of that process will be to associate yourself with a company (if applicable). You can also see breakdowns by authors on the ODL Spectrometer.

That’s all for now. Happy trails, newbie.

Watch for Allan’s blog next week where he will share his Top 10 learnings as a new developer contributing to ODL.

Pin It on Pinterest