Department of Defense
High Performance Computing Modernization Program

The materials collected here document the experiences of those who have:

  • deployed a new or updated an existing networking infrastructure including Software-Defined Networking (SDN) or associated software architectures such as Network Functions Virtualization (NFV).
  • developed or updated networking devices, software, or procedures including support for SDN or associated software architectures such as NFV and Virtual/Virtualized Network Functions (VNF).

While not limited to the topic of deploying SDN, the article 8 Lessons from 20 Years of Hype Cycles still provides lessons that are relevant to the topic of deploying SDN technology.

Training and testing materials for those already using or planning to use SDN or associated software architectures are also provided.

No attempt has been made to extract lessons learned. Instead, original documents are referenced so that the lessons learned can be understood in context. A best practices document describes actions or practices that are known to produce good outcomes when followed..

Materials listed below come from a wide variety of sources, including conference presentations, research papers, industry studies, commercial provider publications, and other websites. Material was included because of its informational content, rather than to promote interest in SDN and associated software architectures or any particular company. The lessons learned material is organized as follows:

  1. Networking customers (e.g., government, industry, educational or research organizations)
  2. Service providers (e.g., Internet Service Providers, internet exchange operators, or wireless carriers)
  3. Data center and cloud service operators, content providers (e.g., Internet Content Providers or Over-The-Top content providers)
  4. Commercial providers (i.e., companies that manufacture, sell or support networking devices, functions, software or services)
  5. Groups (open source communities, standards development organizations and trade groups)
  6. Websites, Small-Medium Businesses [SMB], and individuals (researchers and developers)

and the training and testing materials are organized as follows:

  1. Training Resources (presentations, papers, books, hands-on tutorials, software tools)
  2. Testing Resources (measurement, methodology, test beds, testing, troubleshooting)

In alphabetical order within each category, each entry provides a source, date of creation and title with a link to the web address of the presentation, paper, video, website, or book.


1. Networking customers. 

Cornell University 2013 Experience with 3 SDN controllers in an enterprise setting
Department of Energy 2015 ESnet’s (100G) SDN Testbed
    Software Defined Exchanges: The New SDN?
Duke University 2015 Duke's SDN Journey
  2016 ConCERNing SDN (review of lessons learned designing, building, and operating an intercampus network)
Florida International University 2015 FELIX: FEderated Test-beds for Large-scale Infrastructure eXperiments
Global Environment for Networking Innovation (GENI) 2014 GENI: A federated testbed for innovative network experiments (NOTE: Registration required)
Google 2012 Openflow@Google (video available here)
  2013 B4: Experience with a Globally-Deployed Software Defined WAN (video available here)
  2015 Software Defined Networking at Scale
  2016 Maglev: A Fast and Reliable Software Network Load Balancer
  2016 A Purpose-built Global Network: Google's Move to SDN
  2016 Lessons Learned from B4, Google's SDN WAN (video available here)
Internet2 2012 Internet2 Innovation Platform FAQ
  2015 Challenges of supporting SDN in production
  2015 Internet2 Implements Open Source SDN Networking OS
Internet 2 & Open Network Operating System (ONOS) 2015 Global ONOS and SDN-IP deployment
  2015 Internet2 Implements First Large-scale Deployment of ONOS in Live Network
Lawrence Berkeley National Laboratory 2015 SDN Theory and Best Practices
Microsoft 2014 Real-World SDN: 5 Lessons:
Lesson 1: Management Up Front
Lesson 2: Conquer the Enemy Within
Lesson 3: Focus on the App
Lesson 4: Plan for a Hybrid Cloud
Lesson 5: Demand More From Vendors
Stanford University 2013 Maturing of OpenFlow and Software-Defined Networking through Deployments


2. Service providers.

Accedian Networks 2016 NFV and SDN Lessons from vCPE Deployments (video available here)
 American Telephone and Telegraph (AT&T) 2013 AT&T Domain 2.0 Vision White Paper
  2017 OpenContrail as SDN controller for NFV infrastructure in AT&T network
CenturyLink, Inc. 2014 Real World Lessons of SDN and NFV
Ciena Corporation 2020 Overcoming Network Virtualization Barriers
Deutsche Telekom AG 2013 TeraStream – A Simplified Service Delivery Model
Defense Information Systems Agency (DISA) 2018 Why DISA has embraced SDN for the Pentagon
Fujitsu Network Communications, Inc. 2014 Software-Defined Networking for the Utilities and Energy Sector
Global Environment for Networking Innovations (GENI) 2014 Workshop on Prototyping and Deploying Experimental Software Defined Exchanges (SDXs) (in particular, the Workshop Outbrief)
Huawei 2015 Upgrading Your System – a Telco User Perspective
  2016 Lessons Learned - Cloud Transformation in the Enterprise
Network Operations and Internet SEcurity (NOISE), Princeton on-going SDX: Software Defined Internet Exchange
NTT America - An NTT Communications Company 2013 SDN: Lessons Learned from a Top IP Provider
Openwave Mobility 2017 6 lessons for mobile operators moving to NFV
Pacific Northwest National Laboratory (PNNL) 2022 Software-defined Networking for Energy Delivery Systems (SDN4EDS): An Architectural Blueprint – Final Report
Research and Education Advanced Network New Zealand Ltd (REANNZ) & Wellington Internet Exchange (WIX) 2014 Cardigan: SDN Distributed Routing at an Internet Exchange
Rothenberg, Chua, et al 2014 Cardigan: When Open Source Meets Network Control Planes
SKT, Viva Kuwait, and ZTE 2018 Migration from Physical to Virtual Network Functions – Best Practices and Lessons Learned
Verizon 2016

Lessons Learned: Deploying NFV infrastructure at Verizon (video)

 Notes on the video

  2016 Verizon Network Infrastructure Planning: SDN-NFV Reference Architecture (technical details)


3. Data center and cloud service, content providers.

Alibaba 2015 Hybrid Cloud Networks: OpenFlow/VxLAN Practice (video)
ACMQueue 2006 A Conversation with Amazon CTO Werner Vogels
High Scalability blog entry 2007 Amazon Architecture
eBay inc. 2015 SDN Networks at web scale (video)
Facebook 2015 What Facebook's newest data center can teach us
  2015 Facebook completes Chef DevOps migration, shares lessons
Google 2014 Enter the Andromeda zone – Google’s Cloud Platform (video available here)
  2015 A look inside Google’s Data Center Networks (video available here)
Intel Corporation 2014 Adopting Software-Defined Networking in the Enterprise
LinkedIn 2016 Project Falco: Decoupling Switching Hardware and Software
Microsoft 2013 Brain-Slug: A BGP-only SDN for Large Scale Data Centers
  2013 SDN in the Public Cloud: Windows Azure
  2014 Windows Azure: Scaling SDN in the Public Cloud (video)
  2015 Microsoft Azure: SDN in a Hyperscale Cloud (video)
  2016 Software for Open Networking in the Cloud (SONiC)
  2017 VFP: A Virtual Switch Platform for Host SDN in the Public Cloud
  2018 Azure Accelerated Networking: Smart NICs in the Public Cloud
Network Heresy 2013 Network Virtualization and the End-to-End Principle
Openstack 2015 Building a Secure Multi-tenant Cloud for SaaS Applications (video)
PayPal 2015 PayPal VP on OpenStack deployment benefits and burdens
VMWare and International Computer Science Institute (ICSI) 2014 Network Virtualization in Multi-tenant Datacenters
Yahoo! 2012 SDN in Warehouse Scale Datacenters v2.0 (video available here)


4. Commercial providers.

Affirmed Networks (a Microsoft company) 2018 Lessons Learned on the NFV Front Lines
DataYard 2014 Building Mission Critical Cloud Infrastructure: Lessons Learned At Scale
Deloitte Development LLC. 2015 Operationalizing SDN and NFV Networks
Ericsson 2014 CenturyLink: Competitive Innovation
Nicira Networks 2012 SDN: What I’ve Learned
Cisco, NetApp, & Red Hat 2014 Considerations and Lessons Learned Deploying OpenStack
HP, Intel, & WindRiver 2013 Practical Implementation of SDN & NFV in the WAN
Metaswitch Networks (A Microsoft company) 2016 Summary Lessons from Deploying NFV
Packet Design 2015 How we failed at OpenStack
Red Hat 2016 Lessons Learned from a Large-Scale Telco OSP+SDN Deployment (video available here, CI means "continuous integration"))
  2017 Best practices for successfully deploying NFV
Sandvine 2015 From hardware to NFV: lessons learned in deploying an OpenStack reference at scale (video)


5. Groups. 

Alliance for Telecommunications Industry Solutions (ATIS) 2013 Operational Opportunities and Challenges of SDN/NFV Programmable Infrastructure © 2013 by Alliance for Telecommunications Industry Solutions. ALL RIGHTS RESERVED.
American Council for Technology-Industry Advisory Council (ACT-IAC) 2017 Software Defined Networking and Network Function Virtualization
  2018 Cloud Migrations – Lessons Learned (cloud migration case studies from 11 representative Federal government agencies) 2018 Why 70% of NFV and Digital Transformation Projects Fail
Groupe Spécial Mobile (GSM) Association (GSMA) 2020 Migration from Physical to Virtual Network Functions – Best Practices and Lessons Learned Version 0.1
ICSI and UC Berkeley 2012 NOX, POX, and Lessons Learned
Institute of Electrical and Electronics Engineers (IEEE) Future Directions Committee (FDC) 2014 Software-Defined Networks for Future Networks and Services: Main Technical Challenges and Business Implications
  2014 A Survey of Software-Defined Networking: Past, Present, and Future of Programmable Networks
Metro Ethernet Forum (MEF) 2014 Carrier Ethernet and SDN
Part 1: An Industry Perspective
Part 2: Practical Considerations
Network Functions Virtualization (NFV) European Telecommunications Standards Institute (ETSI) Industry Specifications Group (ISG) 2015 ETSI Group Specification (GS) NFV-EVE 005 V1.1.1 (2015-12) NFV; Ecosystem; Report on SDN Usage in NFV Architectural Framework
Open Data Center Alliance, Inc. (ODCA) 2013 ODCA Master Usage Model: Software-Defined Networking Rev. 1.0 © 2013 Open Data Center Alliance, Inc. ALL RIGHTS RESERVED.
Open Networking Foundation 2013 Migration Tools and Metrics
  2013 Migration Use Cases and Methods (3 case studies: Google Inter-Datacenter WAN, NTT Provider Edge, and Stanford Campus Network)
Optical Internetworking Forum 2014 Global Transport SDN Prototype Demonstration


Additional materials may be available. See the Points of Contact article for links to websites of several open source communities and standards development organizations.


6. Websites, SMB, and individuals.

SourceDateTitle, a subsidiary of IDG Enterprise 2015 CIOs Share Lessons Learned From the Journey to the Cloud
Shawn Ennis 2017 How Vendor Maturity Challenges NFV Adoption
Harvard Business Publishing 2015 Cloud Adoption Gets Serious: 2½ Years In
InformationWeek 2011 4 Lessons Learned From Virtualization Masters
Koybayashi, Seetharaman, et al 2013 Maturing of OpenFlow and Software-defined Networking through deployments
Latin School of Chicago 2010 Lessons Learned about Consolidation, Virtualization and Disaster Recovery
Levin, Canini, Schmid, et al 2013 Incremental SDN Deployment in Enterprise Networks (video available here)
Lin & Hart 2013 Seamless Interworking of SDN and IP
  2014 Seamless Interworking of SDN and IP
Mishra & Dasari 2014 GENI Deployment and Research at US Army Research Laboratory (NOTE: Registration required)
Sagar Nagare 2018 Evaluating Container Based VNF Deployment For Cloud Native NFV
Naqvi 2015 Technical and Non-Technical lessons learned from SDN Implementations
Packet Pushers 2015 Software Defined WAN – Night of Nerdery – Live From New York (podcast featuring Bloomberg, Gap, and Visa panel members)
    SDN: What Small and Mid-Sized Businesses Need to Know in 2015
Prayson Pate 2013-2021 A series of articles on Linkedin about aspects of NFV and SDN
Pepelnjak 2015 What NFV Means for Enterprise
SDxCentral 2016 4 Lessons the Telecom Industry Should Learn From the Enterprise
  on-going SDN Webinars – a series of webinars on current industry deployments of SDN
University of Pittsburgh Medical Center 2014 SDN meets the real-world: implementation benefits and challenges (used by permission)


7. Training Resources.

Acadia Technology Group 2018 11 Rules for Software-Defined Networking Integration
Donovan & Prabhu 2017 Building the Network of the Future: Getting Smarter, Faster, and More Flexible with a Software Centric Approach (book)
Goransson & Black 2014 Software Defined Networks: A Comprehensive Approach (book)
Matt Oswalt blog 2014-2015 SDN Protocols in 5 parts:
Part 1: OpenFlow Basics
Part 2: OpenFlow Deep-Dive
Part 3: OVSDB
Part 4: OpFlex and Declarative Networking
  2015 Open-Source Routing: QUAGGA, ExaBGP, and BIRD
Mininet – GitHub 2015 Introduction to Mininet
Mininet – SIGCOMM 2014 Tutorial: Teaching Computer Networking with Mininet
Nadeau & Gray 2013 SDN: Software Defined Networks. An Authoritative Review of Network Programmability Technologies (book)
NS-3 Network Simulator on-going Discrete event network simulator with OpenFlow 1.3 support
Open Networking Foundation (ONF) 2014 Migration Use Cases and Methods (guidelines, methods and recommendations to migrate network services from a traditional network to SDN) 2015 Five SDN starter kit options you should know
  2015 SDN starter kits remove some do-it-yourself aspects of an SDN deployment
Tittel (Tom's IT Pro) 2017 Best Free Software Defined Networking (SDN) Training and Materials


8. Testing Resources.

BISmark – Georgia Tech 2011 Managing the Home Network
BISmark – Project BISmark 2014 Project BISmark - Broadband Internet Service Benchmark
Canini, Enzano, et al 2012 A NICE Way to Test OpenFlow Applications
Hongyi 2014 Automatic Data Plane Testing
Kurshid, Zou, Zhou, et al 2013 VeriFlow: Verifying Network-Wide Invariants in Real Time
Microsoft 2022 Troubleshoot the Windows Server Software Defined Networking Stack
ONF on-going OpenFlow Conformance Certification
SDN Testing 2015 blog of various open source and commercial SDN controller test results
University of New Hampshire InterOperability Laboratory (UNH-IOL) ongoing SDN Test Plans: Certification, Conformance, Interoperability
  ongoing SDN and Routing Testing Services


Additional materials may be available from commercial testing service and equipment providers and open source testing software developers on their respective websites. A partial list includes Accedian Networks, European Advanced Networking Test Center AG, Keysight Technologies, Paragon Active AssuranceSpirent Communications plc, UNH-IOL and Vivai Solutions, Inc.

Other listings of open source testing software developers are available in the Points of Contact article, here and in Table 1 of this paper.