The service provider market is undergoing earth-shaking changes. These changes impact the way that applications for consumers and business users are deployed in the network and cloud as well as the way that the underlying data transport networks are built.
At Lumina, we’ve had the chance to work with several large service providers on their software transformation initiatives and get an up-close look at what works and what doesn’t. Three factors are particularly favorable in setting up successful projects for frictionless progress from the lab through prototype and proof of concept and into the production network.
Our first observation is that top-down initiatives and leadership work better than bottom-up or “grass roots” approaches. The experience of AT&T strongly illustrates the advantage. While a few of the hyperscale cloud providers had already launched open source initiatives and projects, the first big move among the established service providers was AT&T’s Domain 2.0, led by John Donovan in 2013. Domain 2.0 was not a precise description of everything that AT&T wanted to do, but through that initiative, the leadership created an environment where transformative projects are embraced and resistance to change is pushed aside.
While lack of top down support is not a showstopper, it is extremely helpful to get past obstacles and overcome organizational resistance to change. If top-down support in your organization is lacking or weak, it is worth your effort to influence and educate your executives. In engaging executives focus on the business value of open software networking. The advantages of open source in software networks include eliminating lock-in and spurring innovation. As our CEO, Andrew Coward, wrote in his recent blog, Why Lumina Networks? Why Now?: “Those who build their own solutions—using off-the-shelf components married to unique in-house developed functionality—build-in the agility and options for difference that are necessary to stay ahead.”
Although it may create a slower start, from what we have seen, taking the time to do early PoCs to onboard executive support so that they deeply attach to the value is time well worth spent. Sometimes a slow start is just what is needed to move fast.
Collaboration through Open Source
The second observation is that industry collaboration can work. I read an interesting comment by Radhika Venkatraman, senior vice president and CIO of network and technology at Verizon, in her interview with SDxCentral. She said, “We are trying to learn from the Facebooks and Googles about how they did this.” One of the best ways to collaborate with other thought leaders in the industry is to join forces within the developer community at open source projects. The Linux Foundation’s OpenDaylight Project includes strong participation from both the vendor community and global service providers including AT&T, Alibaba Group, Baidu, China Mobile, Comcast and Tencent. Tencent, for one, has over 500 million subscribers that utilize their OpenDaylight infrastructure, and they are contributing back to the community as are many others.
A great recent example of industry collaboration is the newly announced ONAP (Open Network Automation Platform) project. Here, the origins of the project have roots in work done by AT&T, China Mobile and others. And now, we have a thriving open source developer community consisting of engineers and innovators who may not have necessarily collaborated in the past.
These participants see benefits of collaboration not only to accelerate innovation but also to build the software run time across many different types of environments and use cases so as to increase reliability. Providers recognize that in their transformation to software networks there’s much they can do together to drive technology, while using how they define and deliver services to stand out from each other in the experiences created for customers.
What about your organization? Do you engage in the OpenDaylight community? Have you explored how ONAP can help you? Do you use OpenStack in your production network? And importantly, do you engage in the discussions and share back what you learn and develop?
Pursuit of Innovation
A third observation is the growing ability for service providers to create and innovate at levels not seen before. A prime example here is the work done by CenturyLink to develop Central Office Re-architected as a Datacenter platform to deliver DSL services running on OpenDaylight. CenturyLink used internal software development teams along with open source and Agile approaches to create and deploy CORD as part of a broad software transformation initiative.
One might have thought that you would only see this level of innovation at Google, Facebook or AWS, but at Lumina we are seeing this as an industry-wide trend. The customer base, business model, and operations of service providers vary widely from one to another based on their historical strengths and legacy investment. All have an opportunity to innovate in a way that advances their particular differences and competitive advantages.
So we encourage you to get on the bandwagon! Don’t stand on the sidelines. A combination of leadership, collaboration and innovation are the ingredients you need to help your organization drive the software transformation needed to stay competitive. There is no other choice.
Stay tuned for my next blog where we will discuss some of the specifics of the advantages, development and innovation using open source.
When my colleague Jon Castro told me early this year that the next OpenStack summit was going to be in Barcelona and that we should go I thought, yeah right, keep dreaming! How could we possibly land a valid excuse to travel halfway across the world to eat Paella and drink Sangria, oh and attend some talks of course 😃 . So we submitted an abstract for the work we had done with NTT West around the OpenDaylight controller and OpenStack, and thankfully we were selected!
Being my first OpenStack summit I had no preconceived notions as to what to expect (so hopefully this recap is bias free), and if you’re a fan of TLDRs like I am, then here is mine: Lots of NFV, SFC, Orchestration talk, OpenStack is no longer a science project but an enterprise solution, and Barcelona is good fun. Now into the details.
As I was at the summit for the reason of presenting, I thought it’s best to go over this topic first. My talk was focused on a project we did with NTT West around providing programmatic/API access to a traditional firewall that could only be configured via CLI, and then driving this API through a new Horizon dashboard. The end goal being an operator (tenant) could be assigned a firewall and be able to administer the device without having to be experienced in the specifics of the command line interface. If this sounds interesting to you, you can watch the presentation on YouTube to get a better understanding of how we achieved this.
Just by looking at the Summit schedule it is clear that NFV is a key use case for many companies using and contributing to OpenStack. There were 24 presentations and/or demos focused around NFV, diving into particular aspects such as orchestration, service function chaining (SFC) and use cases such as virtual evolved packet core (vEPC). I can’t talk about NFV without mentioning OPNFV (Open Platform for NFV) which is a reference NFV platform built by the open source community under the linux foundation. OPNFV is primarily a testing and integration project that brings together all the bits and pieces (OpenStack, OpenDaylight, OVS, DPDK, etc) you would need to launch a NFV cloud ready platform, that has been thoroughly tested, automated and put through a CI/CD pipeline, all packaged in a nice installer(s) with documentation. You can learn more about OPNFV by visiting the following site. One OPNFV presentation at the summit was highlighting the advancements in neutron that benefit NFV use cases such as VLAN aware VMs and also showing the current shortfalls of neutron and how other networking solutions such as OpenDaylight can play nicely with Neutron to fulfil these shortfalls.
SFC was another hot topic, there were a couple of talks that showcased the benefits of using SFC and how it was being implemented in OpenStack. The following presentation was one of those, showing how traffic could be redirected based on a classification such as optimising IPSec, or video traffic. This is achieved through using Network Service Headers (NSH) a SFC encapsulation protocol still in a IETF draft but quickly becoming reality and some aspects can already be used in the Mitaka release of OpenStack.
Other great network-centric talks included the OVN presentation which shows some powerful features if you’re operating a OpenStack cloud using OVS and OVN to perform L2 and L3 functions (networking-ovn plugin). It allows for extended OVS scaling (distributed DHCP on the OVS agent), L3 performance becomes on-par with L2 performance through flow caching, and pre-calculation of network hops. New debugging features called ovn-trace allows for “what-if” analysis on packet classifications so you can see how a packet will traverse the network and flow table(s). They also spoke about BPF Datapath which provides a sandboxed environment in the linux kernel allowing new functionality to be inserted at runtime without having to write new linux kernel modules which can cause headaches not only to write, but to maintain and be supported in various linux distributions. This means new network and tunnelling protocols developed for a particular use case can be created and potentially be portable across Linux distributions.
Orchestration was also a big topic at the summit, with many presentations from the Tacker team, Cloudify and others. All of whom seem to be converging on the use of TOSCA as the modelling language for NFV services. I recommend the following presentations for those who are interested in all things orchestration:
Other Stuff and Conclusion
Stepping back from the technical aspects of the summit and reading between the lines it can be said that OpenStack has really matured, the distributors such as Mirantis, Ubuntu and RedHat have really gone to lengths to ease the pain of installation through project such as Ironic, OpenStack Ansible, Fuel, Packstack etc. The CI/CD system in place for OpenStack has also meant that code that is pushed upstream is properly reviewed (through gerrit), tests are written (Unit + Integration) and run automatically by the CI system (Jenkins) the results of this process is a stable system made up of many different components with hundreds of contributors from around the world, quite an achievement in itself. I believe the maturity and stability of the product is driving more adoption into telcos who generally set a high bar when it comes to production ready software, so this can only be a good sign for the project.
Last but not least, the city of Barcelona is an amazing place, and the perfect setting to catch up with colleagues and meet new ones all over some fantastic food, drinks and laughs. 10/10 would do it again.
Originally Published on the [https://netdevservice.atlassian.net] website on 4/27/174
I’ve spent the last few months working closely with the OpenDaylight and OpenStack developer teams here at Brocade and I’ve gained a heightened appreciation for how hard it is to turn a giant pile of source code from an open source project into something that customers can deploy and rely on.
Not to criticize open source in any way – it’s a great thing. These new open source projects in the networking industry, such as OpenDaylight, OpenNFV and OpenStack are going to do great things to advance networking technology.
No, it’s just the day to day grind of delivering a real product that challenges our team every day.
On any given day, when we are trying to build the code, we’ll get new random errors and in many cases it’s not immediately obvious where the problem is. In another test we’ll get unexpected compatibility problems between different controller elements. Again, somebody made a change and you can’t trace the problem. On some days, certain features will stop working for no known reason. Because of the above, we need to continuously update and revise test automation and test plans – that is also done daily.
When it comes to debugging a problem, unless you’re working with the source code and regularly navigating to find problems, diagnosis is difficult. Some of the controller internals are extremely complex, for example the MD-SAL. Digging into that to make either enhancements or fixes is not for the faint of heart.
The OpenDaylight controller is actually several projects that must be built separately and then loaded via Karaf. This can be non-intuitive.
Another area of complexity is around managing your own development train. If you’re going to have a non-forked controller that stays very close to the open source, you cannot risk being totally dependent upon the project (for the above reasons and others), and so you basically have to manage a parallel development thread. At the same time, you find problems or want to make minor enhancements that you need in service, but cannot contribute immediately back to the project (that takes some review and time). So you’re left with this problem of branching and re-converging all the time. Balancing the pace of this with the projects pace is a challenge every day.
Then there’s all the maintenance associated with managing our own development thread, supporting internal teams, maintaining and fixing the documentation etc. Contributing or committing code back to the project, when needed, is not a slam dunk either. There is a commit and review process for that. It takes some time and effort.
I think we’ll find the quality of the new Helium release to be significantly better than Hydrogen. Lithium will no doubt be an improvement over Helium and so on. The number of features and capabilities of the controller will also increase rapidly.
But after going through this product development effort the last few months I have a real appreciation for the value that a commercial distribution can bring. And that’s just for the controller itself – what about support, training and so on? Well, I leave those things for another blog.
Originally Published on the Brocade Community on 10/9/2014