Department of Defense
High Performance Computing Modernization Program


Updating an existing networking infrastructure to include Software-Defined Networking (SDN) or associated software architectures, or deploying a new networking infrastructure using SDN or associated software architectures, does not require any Internet Protocol (IP) changes or cause any packet format incompatibilities. As a result, such updates and deployments are already taking place and will continue to occur across the United States and around the world at all organizational levels.

Unless otherwise noted, specific software products mentioned in this Overview have been informally determined, based on users’ experiences and developers’ statements, to support Internet Protocol version 6 (IPv6). (Such lack of support for IPv6 by SDN software products is rare.) Consequently, an elaborate decades-long coordination effort across all levels of the world-wide networking community will not be required (as is the case for the on-going IPv4 to IPv6 transition). An added benefit of such updated or newly-deployed infrastructure is that it will usually be able to support IPv6.

This Software-Defined Overview briefly discusses the software architecture, or emerging architecture, for several Software-Defined technologies. A software architecture is an architecture that has been adopted by at least one standards development organization (SDO).  An emerging architecture is an architecture that has been adopted by at least one software development association (SWDA) but has not yet been standardized. An SWDA is an association formed by one or more interested individuals, non-commercial organizations and commercial companies. SDOs are similar to SWDAs in many organizational aspects, but their purposes are quite different. The purpose of an SDO is to develop, publish and maintain standards within a defined scope of responsibility. One purpose of an SWDA is to develop, publish, maintain and/or promote use of software within a defined scope of responsibility, and (for a few SWDAs) another purpose is to function as an SDO for that software. An SWDA may have additional purposes that are not limited to software development. The scope of an SDO or SWDA may be self-defined or it may be defined by a higher level organization.


What is a Software-Defined technology?

Consider any technology that involves both software and the virtualization of some physical computing, storage or networking/communications device. The nature and the name of that technology will change over time. For example, when a physical server was virtualized, the result could be have been called a “Software-Defined Server” rather than a “virtual machine” but it was not (until years later). When the concept of a physical radio access network was virtualized, it was called a virtual RAN (vRAN), then a “Software-Defined Radio Access Network (SD-RAN)”, then a “Cloud RAN (C-RAN)”, and it will probably continue to evolve.

A Software-Defined technology can also involve both software and the virtualization of data or programs apart from the physical device or devices on which the data is stored or the programs run. For further details, see “What Makes Something Software-Defined?”.


What is Virtualization?

Virtualization is an integral part of any Software-Defined technology. Virtualization can be used to implement some (typically, but not necessarily, hardware-based) features of that technology in software. Virtualization can be used to create an abstraction layer in software which conceals implementation details of one part of the technology from other parts of that technology, or from whomever or whatever is using that technology. For example, early in the evolution of the Software-Defined Networking Architecture, the control plane was an abstraction layer which concealed implementation details of the forwarding table residing in the data plane from the management plane.

Virtualization is older than any of the software or emerging architectures discussed in this Overview. For further details, see this review of virtualization technology evolution. As this article on technology trends observes:

The trend of virtualization is one of the most powerful forces in technology.

During the last 60 years more and more physical information technology (IT) infrastructure resources have been virtualized. Physical computing resources have been virtualized, including memory, processors, displays, sensors and entire servers. Physical storage resources have been virtualized, including tape and disk drives, channels, storage subsystems and entire storage systems. Physical communications/networking resources have been virtualized, including switches, routers, gateways and entire networks. All communications/networking hardware (not just the active resources such as switches, routers and gateways, but also the interconnection resources) can be virtualized. Even data has been virtualized.

More recently, virtually all categories of software resources have become virtualized resources, including applications software, operating systems software, virtual desktop infrastructure (VDI) software and computing center software environments.

Cloud computing is another example of virtualization. Any collection of IT infrastructure resources that is implemented in such a way that it can be used upon demand by the user without regard to the configuration details and physical and/or virtual locations of those resources can be configured for use as a cloud computing resource. Cloud computing resources are variously called

hybrid or
multi cloud.

Once something is virtualized, it along with other virtualized resources typically:

(This article compares and contrasts the characteristics of virtual machines and containers.)


Topics in this Overview:


Software-Defined Networking Architecture

While the timing is everything adage does not always hold true, it has been true during development of the SDN architecture. While there have been previous efforts to change router and switch architectures, the SDN architecture effort has been the most successful to date. The SDN architecture as originally envisioned included:

  • A data plane: Hardware that actually receives and forwards packets, that could be physically separated from
  • A control plane: Software in this abstraction layer monitors the forwarding table residing in the data plane and propagates changes in routing policies to the forwarding table, and
  • A management plane: Software that maintains the routing policies.

This introduction of an intermediate control plane to propagate changes in the routing policies to multiple forwarding tables was a significant addition during the early evolution of the SDN architecture. It made it possible for multiple (possibly dissimilar) data planes to perform distributed packet handling, while a centralized management plane maintained the routing policies. The separation of mechanism (packet handling) from policy (routing policies) is a well-established design principle in computer science.

Early SDN technology (circa 2007)   More recent SDN architecture (circa 2012)

Early SDN technology (circa 2007)                        More recent SDN architecture (circa 2012)


For further details, see this review of the SDN architecture evolution through 2014. Note that this evolution continues – see Emerging Software-Defined Everything Architectures. As shown by the number of SDN open source communities in the Points of Contact article, this continuing evolution can lead to confusion when implementing SDN. The SDN Architecture for the Internet-of-Things developed by the Multinetwork INformation Architecture (MINA) research project proposes an extension of the SDN architecture to support the Internet of Things (IoT).

The SDN architecture currently includes the following elements:

  • A management plane that became part of the Application Layer: Software called Business/Network Applications control the behavior and performance of a network. One way that Applications do this is by using the services of modules to maintain the routing policies. An Application does not offer any APIs to other Applications or to modules. The functionality of Network Applications included in different SDN implementations varies widely. The functionality of Business Applications is determined by the customer.
  • A control plane that became the Control Layer: Software called Network Service modules in this abstraction layer performs one or more services, providing one or more Application Program Interfaces (APIs), allowing software in the Application or Control Layers to make use of said services and returning one or more results. Some of the services these modules perform include monitoring the forwarding tables residing in multiple networking devices, and propagating changes in routing policies to those forwarding tables. Modules are logically centralized, even though they may be installed on one or more physical or virtual SDN controllers. Refer to this list of open source controllers and this list of commercial SDN controllers for further details.
  • A data plane that became the Infrastructure Layer: Upon matching a conditional entry in a forwarding table for a received packet, a networking device performs one or more actions, possibly including forwarding the packet. The forwarding tables residing in each device are structured to meet that device’s specific requirements. A device may be either physical or virtual. A device manufactured by an original equipment manufacturer may be referred to as a white box. Prior to SDN, the operating system (O/S) for a device was usually provided by the manufacturer of that device. In SDN, the O/S for an “open network” (also called an “open networking”) device may be provided by an SWDA rather than the manufacturer.

In the SDN architecture:

  • The Control and Application Layers communicate by means of “Northbound APIs” (NBIs). RESTful and RESTCONF are some examples. This diagram shows an example of communications between the Control and Application Layers using the RESTful NBI. The Open Networking Foundation (ONF) initiated the OpenSourceSDN initiative in 2015 (no longer active, but website archived in 2017). NBIs developed by that initiative are available on GitHub.
  • The Control and Infrastructure Layers communicate by means of “Southbound APIs” (SBIs). OpenFlow, P4, or NETCONF are some examples.
  • Data modeling languages such as Yet Another Next Generation (YANG) can facilitate the use of NBIs and SBIs.
  • For communications within the Infrastructure Layer (originally called the Data Plane), several APIs and API implementations have been standardized, including the OpenDataPlane (ODP) and the Data Plane Development Kit (DPDK).

Different SWDAs may use different terminology to describe the elements of their SDN architecture. A Layer in one may be analogous to a Plane or a Tier in another, or even have no analogue. For additional details about terms in the SDN architecture, refer to this glossary. In the future, the scope of the SDN architecture could expand to include security – see Software-Defined Security Architecture.



Network Functions Virtualization Software Architecture

The realization that physical network function components (PNFCs) (e.g., firewalls, load balancers or other network elements) could be virtualized was the foundation for the “Network Functions Virtualization (NFV)” software architecture developed by the European Telecommunications Standards Institute (ETSI) Industry Specification Group for NFV (ISG for NFV). For further details, see this three part report on the NFV software architecture:

Part 1: Foundations of NFV: NFV Infrastructure (NFVI) and VIM,
Part 2: Rising Roles of MANO, LSO, and Assurance, and
Part 3: State of the VNF Ecosystem.

Some industry observers worry that NFV promises too much too soon, as in this Does NFV Stand For Not Financially Viable? article. 



NFV software architecture (circa 2013)


An NFV software architecture includes the following elements:

  • “NFV Infrastructure (NFVI)”: An interconnected collection of IT infrastructure resources including virtual and physical computing devices, storage devices, networking devices, network functions and sensors. This collection need not be collocated. If the collection resides at more than one location (whether physical or virtual), the virtual and physical networking resources interconnecting those locations are included in the NFVI. It is neither assumed nor required that the NFVI uses Software-Defined technologies. In 2019 the Linux Foundation (The LF) and the Groupe Spécial Mobile (GSM) Association (GSMA) SWDA developed an NFVI reference architecture, called Botrange.
  • Virtual/Virtualized Network Function (VNF)” or "Cloud-Native Network Function (CNF)": A collection of VNF Components (VNFCs) and PNFCs implementing an external network service delivery policy for an NFVI. The properties of each resource in the collection are specified in a VNF Descriptor (VNFD). A VNFD is a template that specifies the operational capabilities, deployment requirements and management characteristics of a VNFC or PNFC,
  • NFV MANagement and Orchestration (NFV MANO)” or Cloud NFV includes:
  1. Orchestrator: configures and maintains an internal resource allocation policy, an external network service delivery policy and the VNFDs for an NFVI. For example, when multiple NFVIs are running in physically or virtually separate Docker containers, an orchestrator can be used to manage them, as explained by this articleWhat an Orchestrator is not is also important.
  2. VNF Manager(s): abstraction layer that instantiates, manages and terminates VNFs or CNFs in accordance with an external network service delivery policy and VNFDs contained in the VNFs and CNFs of the NVFI.
  3. Virtualized Infrastructure Manger(s) (VIM): abstraction layer that manages the IT infrastructure resources of the NFVI in accordance with an internal resource allocation policy.

Approaches to controlling interconnected collections of IT infrastructure resources are proliferating. One example is the emerging Software-Defined Interconnect (SD-IX, sometimes called SDI) architecture. (The emerging SD-IX architecture should not be confused with an implementation of the emerging SDDC architecture called SDI mentioned in the Software-Defined Data Center topic.)

SWDAs and researchers typically apply the use case methodology of systems analysis to identify and clarify implementation requirements and discuss anticipated benefits of specific cases of SDN architecture use or of NFV software architecture use in a detailed, ever growing SDN and NFV use cases taxonomy. Some may apply the user story methodology.

For further details about terminology in the NFV software architecture, refer to the ETSI Group Specification NFV 003 Terminology for Main Concepts in NFV. In the future, the scope of the NFVI software architecture could expand to include security – see Software-Defined Security Architecture.


Software-Defined Internet Exchange Point

One network function that can be virtualized is an “Internet Exchange Point (IXP)”. An IXP using SDN and possibly other Software-Defined or emerging architectures is called a “Software-Defined IXP (SDX)” or an industrial-scale SDX (iSDX).


Software-Defined Security Architecture

Software-Defined Security (SDSec)” is an emerging architecture under consideration for adoption by SDOs and SWDAs. As envisioned by the ONF in an SDN context, the scope of Applications could be expanded to include configuring and maintaining an external security policy, while Network Service modules could implement that security policy. As envisioned by ETSI in an NFV context, the scope of the Orchestrator in the NFV MANO element could be expanded to include configuring and maintaining an external security policy for an NFVI, while VNFs could implement that security policy. SDSec is also under consideration as an element of the SDx Infrastructure emerging architecture.

Alternative SDSec software and emerging architectures include Software-Defined Perimeter, Software-Defined Protection, and MicroSegmentation (originally in the context of the Software-Defined Data Center) and related emerging architectures such as NanoSegmentation, which is mentioned in passing by this Gartner blog entry.

In SDSec software and emerging architectures, security policy implementation could be separated from security policy management in a manner analogous to the distributed routing policy execution, with centralized maintenance of the routing policies in the SDN architecture.


Other Network Virtualization Architectures

In conjunction with the ongoing evolution of the NFV architecture, many “Network Virtualization (NV)” software and emerging architectures are appearing with abstraction layers that conceal implementation details of the network resources from the network configuration, management or security software. Just which implementation details are hidden and the functionality of the network software vary widely, depending on the specific software or emerging architecture.

Some Wide Area Network (WAN) NV software and emerging architectures include “Intent Based Networking (IBN)”, “Network-as-a-Service (NaaS)”, “Virtual WAN (VWAN)”, “Software-Defined WAN (SD-WAN)” (sometimes referred to as an “overlay-based SD-WAN” or “tunnel-based SD-WAN”, in contrast to another emerging architecture, the “tunnel-free SD-WAN”), "universal SD-WAN (uSD-WAN)" and "Secure Access Service Edge (SASE)". Open source software implementations of SD-WAN architectures are also beginning to emerge, and in July 2019 the Metro Ethernet Forum (MEF) SWDA released an SD-WAN Services Standard document, defining standard Edge, Controller and Orchestrator functions.

Implementations of virtual/virtualized “Evolved Packet Core (vEPC)” and virtual/virtualized “Internet Protocol Multimedia Subsystem (vIMS)” WAN NV, together with network slicing and dynamic network slicing emerging architectures are offered by members of the “3rd Generation Partnership Project (3GPP)”. Additional WAN NV emerging architectures are listed in this article.

Originating at the Local Area Network (LAN) level, but not confined to that level, “virtualized Customer Premises Equipment (vCPE)” and "universal CPE (uCPE)" NV emerging architectures are discussed in this SD-WAN and virtualized CPE ReportSD-Branch NV emerging architectures are a further evolution of SD-WAN for various use cases. SD-LAN emerging architectures are an alternative approach. LAN NV architectural elements such as VNFCs and PNFCs may also be found in other emerging architectures such as the Software-Defined Data Center.

Independently of the ongoing evolution of the NFV architecture, virtualization is being incorporated into LAN architectures. Examples include “Virtual LAN (VLAN)”, “Virtual eXtensible LAN (VXLAN)”, “Network Virtualization using Generic Routing Encapsulation (NVGRE) LAN” software architectures, and the many “wireless LAN (WLAN)” emerging architectures.

Emerging architectures for both physical and virtual networking switch fabrics continue to evolve, as part of a quixotic quest for an SD-WAN that is an all-encompassing Fabric for Universal Networking (FUN). Seriously.


IT Infrastructure Virtualization Architectures

Abstraction layers can be introduced anywhere in an IT infrastructure. If an abstraction layer is introduced to conceal implementation details in an IT infrastructure from the customer interface to that IT infrastructure, it may result in a Software-Defined Data Center or a Software-Defined Workspace. If an abstraction layer is introduced to conceal implementation details of some applications in an IT infrastructure from the customer interface to those applications, it may result in a Software-Defined Architecture. If an abstraction layer is introduced to conceal details of some storage resources in an IT infrastructure from the application interface to those storage resources, it may result in Software-Defined Storage. If an abstraction layer is introduced to conceal details about the attributes of data or the storage resources in an IT infrastructure where data resides from the applications that access that data, it may result in Software-Defined Data.


Software-Defined Data Center

Various SWDAs are developing “Software-Defined Data Center (SDDC)” emerging architectures. An SDDC is a data center implemented using virtualization together with one or more Software-Defined software or emerging architectures.

There is no consistent nomenclature for SDDCs. SDDCs are variously called Software-Defined Infrastructures (SDIs), Software-Led Infrastructures (SLIs), Platform-as-a-Service (PaaS), or composable infrastructures by the SWDAs developing them. A composable infrastructure can even be built with composable fabric, as this article comparing PaaS with other cloud computing models describes.

An SDDC may include a combination of both physical and virtual: discrete, converged and hyper-converged infrastructure (HCI) resources. An SDDC may even include disaggregated HCI resources. Multi cloud using SD-WAN is another emerging architecture. According to this article, based on the IT resources included in an SDDC emerging architecture, an SDDC can be categorized as either:

  1. discrete, physical servers (bare metal) plus other physical IT resources,
  2. virtualized servers (running on hypervisors) plus physical IT resources,
  3. HCI (running on hypervisors), or
  4. HCI (running on hypervisors, but adding containers to the mix of IT resources included, called an Application-Defined Infrastructure (ADI) by some SWDAs).

The customer interface is often called “Infrastructure-as-a Service (IaaS)”. An SDDC is implemented by some combination of local IT infrastructure, fog computing, edge computing and a refinement of edge computing called multi-access edge computing (MEC, originally called mobile edge computing) resources.

This Open Glossary of Edge Computing defines terms used when discussing edge computing. For further details about fog-computing and MEC, see this article. The ETSI ISG for MEC and ISG for NFV are working to enable deploying MEC resources in NFV infrastructures.

An abstraction layer is introduced to conceal various details (which vary with the SWDA) of the internal resource allocation policy and the intrinsic (possibly device-specific) orchestration, management, and provisioning characteristics of the resources in the SDDC.

Different SWDAs may use different terminology to describe their different SDDC emerging architectures.


Software-Defined Workspace/Workplace

Software-Defined Workspace (SDW)” (also called Workspace-as-a-Service or an integrated workspace platform) is an emerging architecture. It can be difficult to differentiate from another emerging architecture, Software-Defined Workplace (also called Workplace-as-a-Service). These emerging architectures employ multiple abstraction layers in order to support the use of any type of application on any platform from anywhere via any network.

Different SWDAs may use different terminology when describing the elements of these emerging architectures.


Software-Defined Architecture

Software-Defined Architecture (SDA)” is an emerging architecture in which an abstraction layer introduces outer APIs (NBIs) and inner APIs (SBIs) for selected applications. The NBIs conceal the implementation details of those applications and their run-time environment from external customers/users. The SBIs conceal these details from other applications. This makes it possible to transparently modify or replace the applications or their environment. The SDA emerging architecture is mentioned in this International Data Corporation article.

SDA is a further evolution of the earlier “Service-Oriented Architecture (SOA)” software architecture. While SDA applications are sometimes called microservices by SWDAs, not every SOA application should be called a microservice for the reasons given in this article. Different SWDAs may use different terminology to describe the abstraction layer of their SWDA emerging architecture.


Software-Defined Storage

Software-Defined Storage (SDS)” emerging architectures separate virtual and physical data storage resources from the software that manages those storage resources, in a manner similar to the separation of the data plane from the management plane in the early evolution of SDN architecture. This article compares and contrasts SDS emerging architectures with the earlier software architectures: Storage Area Networks (SAN) and Network-Attached Storage (NAS). Various SWDAs are developing SDS emerging architectures, such as this initiative by the Storage Networking Industry Association and the Virtual SAN (VSAN) offerings by Cisco Systems, Inc.

An SDS implementation is typically transparently modifiable while on-line and in-use. For example, without disrupting availability or performance, storage capacities can be increased or decreased, or individual storage resources can be repaired, upgraded or replaced.

This article argues that 5 key attributes must be part of any SDS software or emerging architecture. Different SWDAs may use different terminology to describe their SDS emerging architectures which may not always include all 5 attributes.


Software-Defined Data

Could “Software-Defined Data (SDD)” become the next step in the evolution of SDS? This emerging architecture uses a combination of data virtualization and structural metadata to create an abstraction layer that separates data from some of its attributes. These data attributes can include format, storage media, access protocol and physical locality. In the SDD emerging architecture, data would be accessible at any time; appear to reside on any storage media; and appear to be accessible using any protocol, whatever the actual location of the data, the storage medium it resides on, the protocol used to access it, the intended use of that medium (such as on-line, archival or disaster recovery), or the format of the data on the storage resource where it resides (all of which could transparently and automatically be changed over time in response to data access patterns). Implementations of this emerging architecture by SWDAs, including DataSphere (ceased operation in 2018) and ViPR, are mentioned in this International Data Corporation Community article.


Emerging Software-Defined Everything Architectures

Software-Defined technologies continue to evolve. The “Software-Defined Everything" (SDx, originally called “Software-Defined Anything”) emerging architecture is the basis for the inclusive SDx Infrastructure (SDxI). For further details, refer to this eight-part series of articles. Network performance management (NPM) elements of the SDx architecture are defined in this article and application performance management (APM) elements of the SDx architecture are defined in this article.

Note that the SDx emerging architecture should not be confused with the software-defined network function SDX.