<?xml version="1.0" encoding="utf-8"?><?xml-model href="rfc7991bis.rnc"?><!--Required for schema validation and schema-aware editing --> <!-- <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>pre-edited by ST 03/22/24 --> <!--This third-party XSLT can be enabled for direct transformationsdraft submitted inXML processors, including most browsersxml v3 --> <!DOCTYPE rfc [ <!ENTITYfilename "draft-ietf-nvo3-encap-12"> <!ENTITYnbsp " "> <!ENTITY zwsp "​"> <!ENTITY nbhy "‑"> <!ENTITY wj "⁠"> ]><!-- If further character entities are required then they should be added to the DOCTYPE above. Use of an external entity file is not recommended. --> <?rfc strict="yes" ?> <?rfc toc="yes"?><rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="info"docName="&filename;"docName="draft-ietf-nvo3-encap-12" number="9638" consensus="true" obsoletes="" tocInclude="true" ipr="trust200902" updates="" submissionType="IETF" xml:lang="en"version="3"> <!-- * docName should be the name of your draft * category should be one of std, bcp, info, exp, historic * ipr should be one of trust200902, noModificationTrust200902, noDerivativesTrust200902, pre5378Trust200902 * updates can be an RFC number as NNNN * obsoletes can be an RFC number as NNNN --> <!-- ____________________FRONT_MATTER____________________ -->version="3" symRefs="true" sortRefs="true"> <front> <title abbrev="NVO3 Encapsulation Considerations">Network VirtualizationOverlaysover Layer 3 (NVO3) Encapsulation Considerations</title><!-- The abbreviated title is required if the full title is longer than 39 characters --><seriesInfoname="Internet-Draft" value="&filename;"/>name="RFC" value="9638"/> <author initials="S." surname="Boutros" fullname="Sami Boutros" role="editor"> <organization>Ciena Corporation</organization> <address> <postal><country>USA</country><country>United States of America</country> </postal> <email>sboutros@ciena.com</email> </address> </author> <author fullname="Donald E. Eastlake 3rd" initials="D."surname="Eastlake"surname="Eastlake 3rd" role="editor"> <organization>Futurewei Technologies</organization> <address> <postal> <street>2386 Panoramic Circle</street> <city>Apopka</city><region>Florida</region><region>FL</region> <code>32703</code><country>USA</country><country>United States of America</country> </postal> <phone>+1-508-333-2270</phone> <email>d3e3e3@gmail.com</email> </address> </author> <date year="2024"month="2" day="19"/> <area>Routing</area> <workgroup>NVO3 Working Group</workgroup> <!-- "Internet Engineering Task Force" is fine for individual submissions. If this element is not present, the default is "Network Working Group", which is used by the RFC Editor as a nod to the history of the RFC Series. --> <keyword></keyword> <!-- Multiple keywords are allowed. Keywords are incorporated into HTML output files for use by search engines. -->month="August"/> <area>RTG</area> <workgroup>nvo3</workgroup> <abstract> <t>The IETF Network Virtualization Overlays (NVO3) Working Group developed considerations for a common encapsulation that addresses various network virtualization overlay technical concerns. This document provides a record, for the benefit of the IETF community, of the considerations arrived at by the NVO3 Working Group starting from the output ofanthe NVO3 encapsulationdesign team.Design Team. These considerations may be helpful with future deliberations by working groups over the choice of encapsulation formats.</t> <t>There are implications of having different encapsulations in real environments consisting of both software and hardware implementations and within and spanning multiple data centers. For example,OAMOperations, Administration, and Maintenance (OAM) functions such as path MTU discovery become challenging with multiple encapsulations along the data path.</t> <t>Based on these considerations, the NVO3 Working Group determined thatGeneveGeneric Network Virtualization Encapsulation (Geneve) with a few modificationsasis the common encapsulation. This document provides more details, particularly inSection 7.</t><xref target="Recommendations"/>.</t> </abstract> </front><!-- ____________________MIDDLE_MATTER____________________ --><middle> <section><!-- 1. --><name>Introduction</name> <t>The NVO3 Working Group is chartered to gather requirements and develop solutions for network virtualization data planes based on encapsulation of virtual network traffic over an IP-based underlay data plane. Requirements include due consideration for OAM and security. Based on theserequirementsrequirements, the WG was to select, extend, and/or develop one or more data plane encapsulationformat(s).</t>formats.</t> <t>This led to WGdraftsInternet-Drafts and an RFC describing three encapsulations as follows:</t> <ul><li><xref target="RFC8926"/> Geneve:<li>"Geneve: Generic Network VirtualizationEncapsulation</li> <li><xref target="ietf_intarea_gue"/> GenericEncapsulation" <xref target="RFC8926"/></li> <li>"Generic UDPEncapsulation</li> <li><xref target="nvo3_vxlan_gpe"/> GenericEncapsulation" <xref target="I-D.ietf-intarea-gue"/></li> <li>"Generic Protocol Extension for VXLAN(VXLAN-GPE)</li>(VXLAN-GPE)" <xref target="I-D.ietf-nvo3-vxlan-gpe"/></li> </ul> <t>Discussion on the list and in face-to-face meetings identified a number of technical problems with each of these encapsulations. Furthermore, there was a clear consensus at the 96th IETF meeting in Berlinthat, to maximize interoperability,that the working group should progress only one data planeencapsulation.encapsulation, to maximize interoperability. In order to overcome a deadlock on the encapsulation decision, the WG consensus was to form a Design Team <xref target="RFC2418"/> to resolve this issue and provide initial considerations.</t> </section> <section><!-- 2. --><name>Design Team and Working Group Process</name> <t>The Design Team was to select one of the proposed encapsulations and enhance it to address the technical concerns. The goals were simple evolution of deployed networks as well as applicability to all locations in the NVO3architecture were goals.architecture. The Design Team was to specificallyavoid selectingselect a design that allows for future extensibility but is not burdensome on hardwareimplementations but should allow future extensibility.implementations. The selected design also needed to operate well withICMPthe Internet Control Message Protocol (ICMP) and inEqual Cost Multi-PathEqual-Cost Multipath (ECMP) environments. If further extensibility is required, then it should be done in such a manner that it does not require the consent of an entity outside of the IETF.</t> <t>The output of the Design Team was thenprcoessedprocessed through the workinggroupgroup, resulting in a working group consensus for this document.</t> </section> <section><!-- 3. --><name>Terminology</name><t>The<t> The key words"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY","<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>", "<bcp14>MAY</bcp14>", and"OPTIONAL""<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as described inBCP 14BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when, they appear in all capitals, as shownhere.</t>here. </t> </section> <section><!-- 4. --> <name>Abbreviations<name>Abbreviations, Acronyms, andAcronyms</name>Definitions</name> <t>The following abbreviations and acronyms are used in this document:</t> <dl><dt>ACL </dt><dd>- Access<dt>ACL:</dt><dd>Access Control List</dd><dt>DT</dt><dd>- NVO3 encapsulation Design Team</dd> <dt>ECMP</dt><dd>- Equal Cost Multi-Path</dd> <dt>EVPN</dt><dd>- Ethernet<dt>ECMP:</dt><dd>Equal-Cost Multipath</dd> <dt>EVPN:</dt><dd>Ethernet VPN <xref target="RFC8365"/></dd><dt>Geneve</dt><dd>- Generic<dt>Geneve:</dt><dd>Generic Network Virtualization Encapsulation <xref target="RFC8926"/></dd><dt>GPE </dt><dd>- Generic<dt>GPE:</dt><dd>Generic Protocol Extension</dd><dt>GUE </dt><dd>- Generic<dt>GUE:</dt><dd>Generic UDP Encapsulation <xreftarget="ietf_intarea_gue"/></dd> <dt>HMAC</dt><dd>- Hash based keyedtarget="I-D.ietf-intarea-gue"/></dd> <dt>HMAC:</dt><dd>Hash-Based Message Authentication Code <xref target="RFC2104"/></dd><dt>IEEE</dt><dd>- Institute<dt>IEEE:</dt><dd>Institute for Electrical and Electronic Engineers(www.ieee.org)</dd> <dt>NIC</dt><dd>- Network(<eref brackets="angle" target="https://www.ieee.org/"/>)</dd> <dt>NIC:</dt><dd>Network Interface Card (refers to network interface hardwarewhichthat is not necessarily a discrete "card")</dd><dt>NSH </dt><dd>- Network<dt>NSH:</dt><dd>Network Service Header <xref target="RFC8300"/></dd><dt>NVA </dt><dd>- Network<dt>NVA:</dt><dd>Network Virtualization Authority</dd><dt>NVE </dt><dd>- Network<dt>NVE:</dt><dd>Network Virtual Edge(device)</dd> <dt>NVO3</dt><dd>- Network(refers to an NVE device)</dd> <dt>NVO3:</dt><dd>Network VirtualizationOverlaysover Layer 3</dd><dt>OAM </dt><dd>- Operations,<dt>OAM:</dt><dd>Operations, Administration, and Maintenance <xref target="RFC6291"/></dd><dt>PWE3</dt><dd>- Pseudowire<dt>PWE3:</dt><dd>Pseudowire EmulationEdge to Edge</dd> <dt>TCAM</dt><dd>- TernaryEdge-to-Edge</dd> <dt>TCAM:</dt><dd>Ternary Content-Addressable Memory</dd><dt>TLV </dt><dd>- Type, Length, and Value</dd><dt>TLV:</dt><dd>Type-Length-Value</dd> <dt>Transitdevice</dt><dd>- Underlaydevice:</dt><dd>Refers to underlay network devices betweenNVE(s).</dd> <dt>UUID</dt><dd>- UniversallyNVEs.</dd> <dt>UUID:</dt><dd>Universally Unique Identifier</dd><dt>VNI </dt><dd>- Virtual<dt>VNI:</dt><dd>Virtual Network Identifier</dd><dt>VXLAN</dt><dd>- Virtual<dt>VXLAN:</dt><dd>Virtual eXtensibleLANLocal Area Network <xref target="RFC7348"/></dd> </dl> </section> <section><!-- 5. --><name>Encapsulation Issues and Background</name> <t>The following subsections describe issues with current encapsulations as discussed by the NVO3 WG. Numerous extensions and options have been designed for GUE and Genevewhichthat may help resolve some of theseissuesissues, but these have not yet been validated by the WG.</t> <t>Also included are diagrams and information on the candidate encapsulations. These are mostly copied from other documents. Since each protocol is assumed to be sent over UDP, an initial UDPHeaderheader is shownwhichthat would be preceded by an IPv4 or IPv6Header.</t>header.</t> <section><!-- 5.1 --><name>Geneve</name> <t>The Geneve packet format, taken from <xref target="RFC8926"/>, is shown in <xref target="GeneveHeader"/> below.</t> <figure anchor="GeneveHeader"> <name>Geneve Header</name> <artwork type="ascii-art"align="center"> <![CDATA[align="center"><![CDATA[ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 Outer UDP Header: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Dest Port = 6081 Geneve | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UDP Length | UDP Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Geneve Header: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Ver| Opt Len |O|C| Rsvd. | Protocol Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Virtual Network Identifier (VNI) | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Variable-Length Options ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+]]> </artwork>]]></artwork> </figure> <t>The type of payload being carried is indicated by an Ethertype <xreftarget="RFC7042"/>target="RFC9542"/> in the Protocol Type field in the GeneveHeader;header; Ethernet itself is represented by Ethertype 0x6558. See <xref target="RFC8926"/> for details concerning UDP header fields. The O bit indicates an OAM packet. The Geneve C bit is the "Critical"bitbit, which means that the options must be processed or the packet discarded.</t> <t>Issues with Geneve <xref target="RFC8926"/> are as follows:</t> <ul><li>Can't<li>Geneve can't be implemented cost-effectively in all use cases becausevariable lengththe variable-length header and order of the TLVsmakesmake it costly (in terms of number of gates) to implement in hardware.</li><li>Header<li>The header doesn't fit into the largest commonly available parse buffer (256 bytes in a NIC).Cannot justifyThus, doubling the buffer size can't be justified unless it is mandatory for hardware to process additional option fields.</li> </ul><t>Selection<t>The selection of Geneve despite these issues may be the result of the Geneve designefforteffort, assuming that the Geneve header would typically be delivered to a server and parsed in software.</t> </section> <section><!-- 5.2 --><name>Generic UDP Encapsulation (GUE)</name> <figure anchor="GUEHeader"> <name>GUE Header</name> <artwork type="ascii-art"align="center"> <![CDATA[align="center"><![CDATA[ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 UDP Header: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SourceportPort | DestportPort = 6080 GUE | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UDP Length | UDP Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GUE Header: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0 |C| Hlen | Proto/ctype | Flags | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Extensions Fields (optional) ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+]]> </artwork>]]></artwork> </figure> <t>The type of payload being carried is indicated by an IANAInternetprotocol number in the Proto/ctype field. The GUE C bit (Control bit) indicates aControlcontrol packet.</t> <t>Issues with GUE <xreftarget="ietf_intarea_gue"/>target="I-D.ietf-intarea-gue"/> are as follows:</t> <ul> <li>There were a significant number of objections to GUE related to the complexity of its implementation in hardware, similar to those noted for Geneve above, such as the variable length and possible high maximum length of the header.</li> </ul> </section> <section><!-- 5.3 --><name>Generic Protocol Extension (GPE) for VXLAN</name> <figure anchor="GPEHeader"> <name>GPE Header</name> <artwork type="ascii-art"align="center"> <![CDATA[align="center"><![CDATA[ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 Outer UDP Header: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Dest Port = 4790 GPE | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UDP Length | UDP Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ VXLAN-GPE Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|R|Ver|I|P|B|O| Reserved | Next Protocol | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |VXLANVirtual Network Identifier (VNI) | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+]]> </artwork>]]></artwork> </figure> <t>The type of payload being carried is indicated by the Next Protocol field using aVXLAN-GPE-specific registry.registry specific to VXLAN-GPE. The I bit indicates that the VNI is valid. The P bit indicates that the Next Protocol field is valid. The B bit indicates that the packet is an ingress replicated Broadcast, Unknown Unicast, or Multicast packet. The O bit indicates an OAM packet.</t> <t>Issues with VXLAN-GPE <xreftarget="nvo3_vxlan_gpe"/>target="I-D.ietf-nvo3-vxlan-gpe"/> are as follows:</t> <ul> <li>GPE is notday-1day one backwards compatible with VXLAN <xref target="RFC7348"/>. Although the frame format is similar, it uses a different UDP port, so it would require changes to existing implementations even if the rest of the GPE frame were the same.</li> <li>GPE is insufficiently extensible. It adds a Next Protocol field and some flag bits to the VXLAN header but is not otherwise extensible.</li><li>Security, e.g., of the VNI, as<li>As discussed in <xref target="SecExt"/>, security (e.g., of the VNI) has not been addressed by GPE. Although a shim header could be added for security and to support other extensions, this has not been defined yet. More study would be needed to understand the implication of such a shim on offloading in NICs.</li> </ul> </section> </section><section> <!-- 6. --><section anchor="CommonEncapsulationConsiderations"> <name>Common Encapsulation Considerations</name> <section><!-- 6.1 --><name>Current Encapsulations</name><t>Appendix A<t><xref target="EncapsulationComparison"/> includes a detailed comparison between the three proposed encapsulations. The comparison indicates several common properties but also three major differences among the encapsulations:</t> <ul> <li>Extensibility: Geneve and GUE were defined with built-in extensibility, while VXLAN-GPE is not inherently extensible. Note that any of the three encapsulations can be extended using the Network Service Header(NSH(NSH) <xreftarget="RFC8300"/>).</li>target="RFC8300"/>.</li> <li>Extension method: Geneve is extensible usingType/Length/ValueType-Length-Value (TLV) fields, while GUE uses a small set of possibleextensions,extensions and a set of flags that indicate which extensions are present.</li> <li>Length field: Geneve and GUE include a Length field, indicating the length of the encapsulation header, while VXLAN-GPE does not include such a field.ThusThus, it may be harder to skip the encapsulation header with VXLAN-GPE</li> </ul> </section><section> <!-- 6.2 --><section anchor="ExtensionsUseCases"> <name>Useful Extensions Use Cases</name><t>Non-vendor specific extensions,<t>Extensions that are not vendor-specific, such as TLVs,MUST<bcp14>MUST</bcp14> follow the standardization process. The following use cases for extensions show that there is a strong requirement to supportvariable lengthvariable-length extensions with possible different subtypes.</t> <section><!--6.2.1 --><name>Telemetry Extensions</name> <t>In severalscenariosscenarios, it is beneficial to make information available to the operator about the path a packet took through the network or through a network device as well asassociated telemetryinformationavailable to the operator.</t>about associated telemetry.</t> <t>This includes not only tasks like debugging, troubleshooting, and network planning and optimization but also policy or service level agreement compliance checks.</t> <t>Packet scheduling algorithms, especially for balancing traffic acrossequal costequal-cost paths or links, often leverage information contained within the packet, such as protocol number, IP address, orMACMessage Authentication Code (MAC) address.ProbeThus, probe packets wouldthus eitherneed to be either sent between the exact same endpoints with the exact sameparameters,parameters orprobe packets would need to beartificially constructed as "fake" packets and inserted along the path. Both approaches are often not feasible from an operational perspective because access to theend-systemend system is not feasible or the diversity of parameters and associated probe packets to be created is simply too large. An extension providing an in-band telemetry mechanism <xref target="RFC9197"/> is an alternative in those cases.</t> </section> <section anchor="SecExt"><!-- 6.2.2 --><name>Security/Integrity Extensions</name> <t>Since the currently proposed NVO3 encapsulations do not protect their headers, a single bit corruption in the VNI field could deliver a packet to the wrong tenant. Extension headers are needed to use any sophisticated security.</t> <t>The possibility of VNI spoofing with an NVO3 protocol is exacerbated by using UDP. Systems typically have no restrictions on applications being able to send to any UDPportport, so an unprivileged application can trivially spoof VXLAN <xref target="RFC7348"/>packets for instance, includingpackets, using arbitraryVNIs.</t>VNIs, for instance.</t> <t>One can envision support of an HMAC-like Message Authentication Code (MAC) <xref target="RFC2104"/> in an NVO3 extension to authenticate the header and the outer IP addresses, thereby preventing attackers from injecting packets with spoofed VNIs.</t> <t>Another aspect of security is payload security.EssentiallyEssentially, this makes packets that look like the following:</t><sourcecode><artwork><![CDATA[ IP|UDP|NVO3 Encap|DTLS/IPsec-ESP Extension|payload.</sourcecode>]]></artwork> <t>This is desirablesince webecause:</t> <ul> <li>we still have the UDP header forECMP, theECMP,</li> <li>the NVO3 header is in plain text so it can be read by network elements,and differentand</li> <li>different security or other payload transforms can be supported on a single UDP port (we don't need a separate UDP port forDTLS/IPsecDTLS/IPsec; see <xref target="RFC9147"/> and <xreftarget="RFC9147"/>/<xref target="RFC6071"/>).</t>target="RFC6071"/>, respectively).</li> </ul> </section> <section><!-- 6.2.3 --> <name>Group Based<name>Group-Based Policy</name> <t>Another use case would be to carry theGroup BasedGroup-Based Policy (GBP) source group information within a NVO3 header extension in a similar manner as has been implemented for VXLAN <xreftarget="VXLANgroup"/>.target="I-D.smith-vxlan-group-policy"/>. This allows various forms of policy such as access control and QoS to be applied between abstract groups rather than coupled to specific endpoint addresses.</t> </section> </section><section> <!-- 6.3 --><section anchor="HardwareConsiderations"> <name>Hardware Considerations</name> <t>Hardware restrictions should be taken into consideration along with future hardware enhancements that may provide more flexible metadata (MD) processing. However, the set of options that need to and will be implemented in hardware will be a subset of what is implemented insoftware, sincesoftware. This is because software NVEs are likely to grow features, and hence option support, at a more rapid rate.</t> <t>It is hard to predict which options will be implemented in which piece of hardware and when. That depends on whether the hardware will be in the formof</t>of:</t> <ul> <li>a NIC providing increasing offload capabilities to softwareNVEs,</li> <li>or aNVEs, or</li> <li>a switch chip being used as an NVE gateway towards non-NVO3 parts of thenetwork,</li> <li>or even anetwork, or even</li> <li>a transit device that participates in the NVO3dataplane,data plane, e.g., for OAM purposes.</li> </ul> <t>A result of this is that it doesn't look useful to prescribe some orderofto the options so that the ones that are likely to be implemented in hardware comefirst; wefirst. We can't decide such an order when we define theoptions, howeveroptions; however, a control plane can enforce such an order for some hardwareimplementation.</t>implementations.</t> <t>We do know that hardware initially needs toinitiallybe able to efficiently skip over the NVO3 header to find the inner payload. That is needed both for NICs implementing various TCP offload mechanisms and for transit devices and NVEs applying policy or ACLs to the inner payload.</t> </section> <section><!-- 6.4 --><name>Extension Size</name> <t>Extension header length has a significant impact on hardware and software implementations. A maximum total header length that is too small will unnecessarily constrain software flexibility. A maximum total header length that is too large will place a nontrivial cost on hardware implementations. Thus, the DT recommends that there be a minimum and maximum total available extension header length specified. The maximum total header length is determined by the size of the bit field allocated for the total extension header length field. The risk with this approach is that it may be difficult to extend the total header size in the future. The minimum total header length is determined by a requirement in the specifications that all implementations must meet. The risk with this approach is that all implementations will only implement support for the minimum total headerlengthlength, which would then become the de facto maximum total header length.</t> <t>The recommended minimum total available header length is 64 bytes.</t> <t>The size of an extension header should always be4 byte4-byte aligned.</t> <t>The maximum length of a single option should be large enough to meet the different extension use case requirements, e.g., for in-band telemetry and future use.</t> </section> <section><!-- 6.5 --><name>Ordering of Extension Headers</name> <t>To support hardware nodes at the target NVE or at a transit device that can process one or a few extension headers in TCAM, a control plane in such a deploymentcancould signal a capability to ensure that a specific extension header will always appear in a specific order, forexample theexample, that such a specific extension header appear firstonein the packet.</t> <t>The order of the extension headers should be hardware friendly for both the sender and the receiver and possibly some transit devicesalso.as well. This mayrequrerequire that the extension headers and their order bedynamicallydetermined dynamically based on the hardware of those devices.</t> <t>Transit devices don't participate in control plane communication between theend pointsendpoints and are not required to process the extension headers; however, if they do, they may need to process only a small subset of the extension headers that will be consumed by target NVEs.</t> </section> <section><!-- 6.6 --><name>TLV versus Bit Fields</name> <t>If there is a well-known initial set of options thatareis likely to be implemented in software and in hardware, it can be efficient to use the bit fields approach to indicate the presence of extensions as in GUE. However, as described insection 6.3,<xref target="HardwareConsiderations"/>, if options are added over time and different subsets of options are likely to be implemented in different pieces of hardware, then it would be hard for the IETF to specify which options should get the early bit fields. TLVs are a lot more flexible, which avoids the need to determine the relative importance of different options. However, general TLVs of arbitrary order, size, and repetition are difficult to implement in hardware. A middle ground is to use TLVs with restrictions on their size and alignment, observing that individual TLVs can have a fixed length, and to support via the control plane a method such that an NVE will only receive options that it needs and implements. The control plane approach can potentially be used to control the order of the TLVs sent to a particular NVE. Note that transit devices are not likely to participate in the control plane; hence, to the extent that they need to participate in option processing, some other method must be used. Transit devices would have issues with future GUE bit fields being defined for future options as well.</t> <t>A benefit of TLVs from a hardware perspective is that they are self describing, i.e., all the information is in the TLV. In a bit field approach, the hardware needs to look up the bit to determine the length of the data associated with the bit through some separate table, which would add hardware complexity.</t> <t>There are use cases where multiple modules of software are running on an NVE. These can be modules such as a diagnostic module by one vendor that does packet sampling and another module from a different vendor that implements a firewall. Using a TLV format, it is easier to have different software modules process differentTLVs, which could be standard extensions or vendor specific extensions defined by the different vendors,TLVs without conflicting with each other. Such TLVs could be standard extensions or vendor-specific extensions. This can help with hardware modularity as well. There are some implementations with options thatallowsallow different software modules, like MAC learning and security, to process different options.</t> </section> <section><!-- 6.7 --><name>Control Plane Considerations</name> <t>Given that we want to allow considerable flexibility andextensibility, e.g.,extensibility (e.g., for softwareNVEs,NVEs), yet want to be able to support important extensions in less flexible contexts such as hardware NVEs, it is useful to consider the control plane. By control plane in this section we meanbothprotocols, such as EVPN <xref target="RFC8365"/> and others, anddeployment specific configuration.</t>deployment-specific configurations.</t> <t>If each NVE can express in the control plane that it only supports certain extensions (which could be a single extension, or a few), and the source NVEs only include supported extensions in the NVO3 packets, then the target NVE canbothuse a simpler parser (e.g., a TCAM might be usable to look for a single NVO3 extension) and the depth of the inner payload in the NVO3 packet will be minimized. Furthermore, if the target NVE cares about a few extensions and can express in the control plane the desired order of those extensions in the NVO3 packets, then the deployment can provide useful functionality with simplified hardware requirements for the target NVE.</t> <t>Transit devices that are not aware of the NVO3 extensions somewhat benefit from such an approach, since the inner payload is less deep in the packet if no extraneous extension headers are included in the packet. In general, a transit device is not likely to participate in the NVO3 control plane. However, configuration mechanisms can take into account limitations of the transit devices used in particular deployments.</t> <t>Note that with thisapproachapproach, different NVEs could desire different extensions or sets of extensions, which means that the source NVE needs to be able to place different sets of extensions in different NVO3 packets, and perhaps in a different order. It also assumes that underlay multicast or replication servers are not used together with NVO3 extension headers.</t> <t>There is a need to consider mandatory extensions versus optional extensions. Mandatory extensions require the receiver to drop the packet if the extension is unknown. A control plane mechanism can prevent the need for dropping unknown extensions, since they would not be included to target NVEs that do not support them.</t> <t>The control planes defined today need to add the ability to describe the different encapsulations. Thus, perhaps EVPN <xref target="RFC8365"/> and any other control plane protocol that the IETF defines should have a way to indicate the supported NVO3 extensions and theirorder,order for each of the encapsulations supported.</t> <t>Developing a separatedraftdocument on guidance for option processing and control plane participation should be considered. This should provideexamples/guidanceexamples and guidance on the range of usage models anddeploymentsdeployment scenarios for specificoptions andoptions. It should also provide examples of option ordering that are relevant for that specific deployment. This includesend pointsendpoints andmiddle boxesmiddleboxes that are using the options. Having the control plane negotiate the constraints is the most appropriate and flexible way to address these requirements.</t> </section> <section><!-- 6.8 --><name>Split NVE</name> <t>If there is a need for hosts to send and receive options in a split NVE case <xref target="RFC8394"/>, this is possible using any of the existing extensible encapsulations(Geneve,(GPE with NSH, GUE,GPE+NSH)or Geneve) by defining a way to carry those over other transports. An NSH can already be used over different transports.</t> <t>If this is needed with otherencapsulationsencapsulations, it can be done by defining an Ethertype so that it can be carried over Ethernet and IEEE Std 802.1Q <xref target="IEEE802.1Q"/>.</t> <t>If there is a need to carry other encapsulations over MPLS, it would require an EVPN control plane to signal that other encapsulationheader +headers and options will be present in front of theL2Layer 2 (L2) packet. The VNI can be ignored in the header, and the MPLS label will be the one used to identify the EVPN L2 instance.</t> </section> <section anchor="LargerVNI"><!-- 6.9 --><name>Larger VNI Considerations</name> <t>Whether we should make the VNI32-bits32 bits or larger was one of the topics considered. The benefit of a 24-bit VNI would be to avoid unnecessary changes with existing proposals and implementations that are almost all, if not all, using a 24-bit VNI. If we need a larger VNI, perhaps for a telemetry case, an extension can be used to support that. </t> </section> </section><section> <!-- 7. --><section anchor="Recommendations"> <name>Recommendations</name> <t>The Design Team(DT)reported that Geneve was most suitable as a starting point for a proposed standard for network virtualization, for the following reasons given below. This conclusion was supported by the NVO3 Working Group.</t> <ol> <li>On whether the VNI should be in the base header or in an extension header and whether it should be a 24-bit or 32-bit field (see <xref target="LargerVNI"/>), it was agreed that the VNI is critical information for network virtualization andMUST<bcp14>MUST</bcp14> be present in all packets. It was also agreed that a 24-bit VNI, which is supported by Geneve, matches the existing widely used encapsulation formats, i.e., VXLAN <xref target="RFC7348"/> andNVGRENetwork Virtualization Using Generic Routing Encapsulation (NVGRE) <xref target="RFC7637"/>, and hence is more suitable to use going forward.</li> <li>The Geneve header has the total optionslengthlength, which allows skipping over the options for NIC offload operations andwill allowtransit devices to view flow information in the inner payload.</li> <li>The option of using an NSH <xref target="RFC8300"/> with VXLAN-GPE wasconsideredconsidered, but given that an NSH is targeted at service chaining and contains service chaining information, it is less suitable for the network virtualization use case. The other downsideforof VXLAN-GPE was the lack of a header length in VXLAN-GPE, which makes skipping over the headers to process innerpayloadpayloads more difficult. ATotal Option Lengthtotal options length is present in Geneve. It is not possible to skip any options in the middle with VXLAN-GPE. Inprincipleprinciple, a split between a base header and a header with options is interesting (whether that options header is an NSH or some new header without ties to a service path). Whether it would make sense to either use an NSH forthis,this or define a new NVO3 options header was explored. However, this makes it slightly harder to find the inner payload since thelengthLength field is not in the NVO3 header itself. Thus, one more field would have to be extracted to compute the start of the inner payload. Also, if the experience with IPv6 extension headers is a guide, there would be a risk that key pieces of hardware might not implement the options header, resulting in future calls to deprecate its use. Making the options part of the base NVO3 header has less of those issues. Even though the implementation of any particular optioncan notcan't be predicted ahead of time, the option mechanism and ability to skip the options is likely to be broadly implemented.</li> <li>The TLV style and bit field style of the extension mechanisms were compared. It was deemed that parsing either TLVs or bit fields isexpensive and,expensive, and while bit fields may be simpler to parse,it isthey are also more restrictive andrequiresrequire guessing which extensions will be widely implementedso they canin order to get early bit assignments. Given that half the bits are already assigned in GUE, a widely deployed extension may appear in a flag extension, and this will require extraprocessing,processing to dig the flag from the flag extension and then look for the extension itself.AlsoAlso, bit fields are not flexible enough to address the requirements from OAM,Telemetry,telemetry, and securityextensions,extensions forvariable length optionvariable-length options and different subtypes of the same option. While TLVs are more flexible, a control plane can restrict the number of option TLVs as well as the order and size of the TLVs to limit this flexibility and make the TLVs simpler for adataplanedata plane implementation to handle.</li> <li>The multi-vendor NVE case was briefly discussed, as was the need to allow vendors to put their own extensions in the NVE header. This is possible with TLVs.</li> <li>It was agreed that the C(Critical)bit (Critical bit) in Geneve is helpful. This bit indicates that the header includes optionswhichthat must beparsedparsed, or else the packet must be discarded.ItThe bit allows a receiver NVE to easily decide whether or not to process optionsor not, for example(such as aUUID basedUUID-based packettrace,trace) and decide how an optional extensionsuch as thatcan beignored byignored. Thus, areceiver NVE and thus makeCritical bit makes it easy for the NVE to skip over theoptions.options not marked with such a bit. Thus, the C bit should remain as defined in Geneve.</li> <li>There are already some extensions of varying sizes that are being discussed (seesection 6.2) of varying sizes.<xref target="ExtensionsUseCases"/>). By using Geneveoptionsoptions, it is possible to get in-band parameters like switch id, ingress port, egress port, internal delay, and queue size using TLV extensions for telemetrypurposepurposes from switches. It is also possible to add security extension TLVs like HMAC <xref target="RFC2104"/> and DTLS/IPsec (see <xreftarget="RFC9147"/>/<xref target="RFC6071"/>target="RFC9147"/> and <xref target="RFC6071"/>, respectively) to authenticate the Geneve packet header and secure the Geneve packet payload by software or hardware tunnel endpoints. AGroup BasedGroup-Based Policy extension TLV can be carried as well.</li> <li>There are already implementations of Geneve options deployed in production networks. There isas wellnew hardware supporting Geneve TLVparsing.parsing as well. In addition, an In-band Telemetry (INT) specification <xref target="INT"/>specificationis being developed by P4.org that illustrates the option of INTmeta datametadata carried over Geneve.OVN/OVSOpen Virtual Network (OVN) and Open vSwitch (OVS) <xref target="OVN"/> have also definedsomeone or more optionTLV(s)TLVs for Geneve.</li> <li>Usage requirements (seeSection 6)<xref target="CommonEncapsulationConsiderations"/>) have been addressed while also consideringtherequirements and implementations in generalincluding(including those for software andhardware.</li>hardware).</li> </ol> <t>There seems to be interest in standardizing some well-known secure option TLVs to secure the header and payload to guarantee encapsulation header integrity and tenant data privacy. The working group should consider standardizing such option(s).</t> <t>The following enhancements to Geneve are recommended to make it more suitable to hardware and yet provide flexibility for software:</t> <ul> <li>The following sort of text isrecommended:recommended in Geneve documents: while TLVs are more flexible, a control plane can restrict the number of option TLVs as well as the order and size of the TLVs to make it simpler for a data plane implementation in software or hardware to handle. For example, there may be some critical information such as a secure hash that must be processed in a certain order at lowest latency.</li> <li>A control plane can negotiate a subset of option TLVs and certain TLV ordering, as well as limiting the total number of option TLVs present in the packet, for example, to allow for hardware capable of processing fewer options. Hence, the control plane needs to have the ability to describe the supported TLVs subset and their order.</li> <li>The Geneve documents should specify that the subset and order of option TLVsSHOULD<bcp14>SHOULD</bcp14> be configurable for each remote NVE in the absence of a protocol control plane.</li> <li>Geneve should follow fragmentation recommendations in overlay services like PWE3 and the L2/L3 VPN recommendations to guarantee largerMTUMTUs for the tunnel overhead (<xreftarget="RFC3985"/> Section 5.3).</li> <li>Genevetarget="RFC3985" sectionFormat="comma" section="5.3"/>).</li> <li>The Geneve documents should provide a recommendation forcriticalC bitprocessing -(Critical bit) processing. This text could specify how critical bits can be used with controlplane specifyingplanes and specify the critical options.</li> <li>Given that there is a telemetry option use case for a length of 256 bytes, it is recommended that Geneve increase theSinglesingle TLV option length to 256.</li> <li>Geneve address requirements for OAM considerations for alternate marking and for performance measurements that need a2 bit2-bit field in the header should be considered and the need for the current OAM bit in the GeneveHeaderheader should be clarified.</li> <li>The WG should work on security options for Geneve.</li> </ul> </section><section anchor="Acknowledgements"> <!-- 8. --> <name>Acknowledgements</name> <t>The authors would like to thank Tom Herbert for providing the motivation for the Security/Integrity extension, and for his valuable comments, T. Sridhar for his valuable comments and feedback, Anoop Ghanwani for his extensive comments, and Ignas Bagdonas.</t> </section><section><!-- 9. --><name>Security Considerations</name> <t>This document does not introduce any additional security constraints; however, <xref target="SecExt"/>discusessdiscusses security/integrity extensions and this document suggests, inSection 7,<xref target="Recommendations"/>, that thethe nvo3NVO3 WG work on security options for Geneve.</t> </section><!-- end Security Considerations --><section anchor="IANA"> <name>IANA Considerations</name> <t>This documentrequireshas no IANA actions.</t> </section> </middle><!-- ____________________BACK_MATTER____________________ --><back> <displayreference target="I-D.ietf-intarea-gue-extensions" to="GUE-EXTENSIONS"/> <displayreference target="I-D.ietf-intarea-gue" to="GUE"/> <displayreference target="I-D.ietf-nvo3-vxlan-gpe" to="VXLAN-GPE"/> <displayreference target="I-D.smith-vxlan-group-policy" to="VXLAN-GROUP"/> <displayreference target="I-D.hy-nvo3-gue-4-nvo" to="GUE-ENCAPSULAION"/> <references> <name>References</name> <references> <name>Normative References</name> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.2119.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.8174.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/> </references> <references> <name>Informative References</name><reference anchor="ietf_gue_extensions" target="https://datatracker.ietf.org/doc/draft-ietf-intarea-gue-extensions/"> <front> <title>Extensions for Generic UDP Encapsulation</title> <author initials="T." surname="Herbert"/> <author initials="L." surname="Yong"/> <author initials="F." surname="Templin"/> <date year="2019" month="March" day="8"/> </front> <seriesInfo name="work in" value="progress"/> </reference> <reference anchor="ietf_intarea_gue" target=""> <front> <title>Generic UDP Encapsulation</title> <author initials="T." surname="Herbert"/> <author initials="L." surname="Yong"/> <author initials="O." surname="Zia"/> <date year="2019" month="October" day="26"/> </front> <seriesInfo name="work in" value="progress"/> </reference><!-- [I-D.ietf-intarea-gue-extensions] IESG state: Expired as of 08/28/24 --> <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-intarea-gue-extensions.xml"/> <!-- [I-D.ietf-intarea-gue] IESG state: Expired as of 08/28/24 --> <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-intarea-gue.xml"/> <reference anchor="IEEE802.1Q"> <front><title>Bridges<title>IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks</title><author initials="IEEE" surname="802.1 WG" fullname="IEEE 802.1 Working Group"> <organization>Institute for Electrical and Electronic Engineers</organization><author> <organization>IEEE</organization> </author> <dateyear="2014" month="November" day="3"/>year="2022" month="December"/> </front> <seriesInfo name="IEEE Std"value="802.1Q-2014"/>value="802.1Q-2022"/> <seriesInfo name="DOI" value="10.1109/IEEESTD.2022.10004498"/> </reference> <reference anchor="INT" target="https://p4.org/p4-spec/docs/INT_v2_1.pdf"> <front> <title>In-band Network Telemetry (INT) Dataplane Specification</title><author fullname="P4.org"/><author> <organization>P4.org Applications Working Group</organization> </author> <date year="2020" month="November"/> </front> </reference> <!-- [nvo3_vxlan_gpe] [I-D.ietf-nvo3-vxlan-gpe] IESG state: Expired as of 08/28/24. Entered the long way to display editor roles--> <referenceanchor="nvo3_vxlan_gpe" target="https://datatracker.ietf.org/doc/draft-ietf-nvo3-vxlan-gpe/">anchor="I-D.ietf-nvo3-vxlan-gpe" target="https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-vxlan-gpe-13"> <front> <title>Generic Protocol Extension for VXLAN (VXLAN-GPE)</title> <author fullname="Fabio Maino" initials="F."surname="Maino"/>surname="Maino" role="editor"> <organization>Cisco Systems</organization> </author> <author fullname="Larry Kreeger" initials="L."surname="Kreeger"/>surname="Kreeger" role="editor"> <organization>Arrcus</organization> </author> <author fullname="Uri Elzur" initials="U."surname="Elzur"/>surname="Elzur" role="editor"> <organization>Intel</organization> </author> <dateyear="2023"day="4" month="November"day="04"/>year="2023"/> </front> <seriesInfoname="work in" value="progress"/>name="Internet-Draft" value="draft-ietf-nvo3-vxlan-gpe-13"/> </reference> <reference anchor="OVN" target="https://www.openvswitch.org/"> <front><title></title> <author fullname="Open Virtual Network"/><title>Open vSwitch</title> <author> <organization>Linux Foundation</organization> </author> </front> </reference> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.2104.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2104.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.2418.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2418.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.3985.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.3985.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.6071.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6071.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.6291.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.6291.xml"/> <!--Note: RFC 7042 was obsoleted by RFC 9542--> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.7042.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9542.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.7348.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7348.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.7637.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7637.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.8300.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8300.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.8365.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8365.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.8394.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8394.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.8926.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8926.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.9147.xml"/>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9147.xml"/> <xi:includehref="https://www.rfc-editor.org/refs/bibxml/reference.RFC.9197.xml"/> <reference anchor="VXLANgroup" target="https://datatracker.ietf.org/doc/html/draft-smith-vxlan-group-policy-05"> <front> <title>VXLAN Group Policy Option</title> <author initials="M." surname="Smith"/> <author initials="L." surname="Kreeger"/> <date year="2018" month="October" day="22"/> </front> <seriesInfo name="work in" value="progress"/> </reference>href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9197.xml"/> <!-- [I-D.smith-vxlan-group-policy] IESG state: Expired as of 08/28/24--> <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.smith-vxlan-group-policy.xml"/> <!--[draft-hy-nvo3-gue-4-nvo-04] Added during AUTH48. IESG state: Expired as of 8/30/24--> <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.hy-nvo3-gue-4-nvo.xml"/> </references><section></references> <section anchor="EncapsulationComparison"> <!-- Appendix A --> <name>Encapsulation Comparison</name> <section> <!-- A.1 --> <name>Overview</name> <t>This section presents a comparison of the three NVO3 encapsulationproposals,proposals: Geneve <xref target="RFC8926"/>, GUE <xreftarget="ietf_intarea_gue"/>,target="I-D.ietf-intarea-gue"/>, and VXLAN-GPE <xreftarget="nvo3_vxlan_gpe"/>.target="I-D.ietf-nvo3-vxlan-gpe"/>. The three encapsulations use an outer UDP/IP transport. Geneve and VXLAN-GPE use an 8-octet header, while GUE uses a 4-octet header. In addition to the base header, optional extensions may be included in the encapsulation, as discussed inSection A.2<xref target="Extensibility"/> below.</t> </section><section><section anchor="Extensibility"> <!-- A.2 --> <name>Extensibility</name> <section> <!-- A.2.1 --><name>Native<name>Innate Extensibility Support</name> <t>The Geneve and GUE encapsulations both enable optional headers to be incorporated at the end of the base encapsulation header.</t> <t>VXLAN-GPE does not providenativeinnate support for header extensions. However, as discussed in <xreftarget="nvo3_vxlan_gpe"/>,target="I-D.ietf-nvo3-vxlan-gpe"/>, extensibility can be attained to some extent if the Network Service Header (NSH) <xref target="RFC8300"/> is used immediately following the VXLAN-GPE header. The NSH supports either a fixed-size extension (MD Type1),1) or a variable-size TLV-based extension (MD Type 2). Note that NSH-over-VXLAN-GPE implies an additional overhead of the8-octets NSH header,8-octet NSH, in addition to the VXLAN-GPE header.</t> </section> <section> <!-- A.2.2 --> <name>Extension Parsing</name> <t>The GeneveVariable Length Optionsvariable-length options are defined asType/Length/ValueType-Length-Value (TLV) extensions. Similarly, VXLAN-GPE, when using an NSH, can include NSH TLV-based extensions. In contrast, GUE defines a small set of possible extension fields (proposed in <xreftarget="ietf_gue_extensions"/>),target="I-D.ietf-intarea-gue-extensions"/> and <xref target="I-D.hy-nvo3-gue-4-nvo"/>), and a set of flags in the GUE header that indicate for each extension type whether it is present or not.</t> <t>TLV-based extensions, as defined in Geneve, provide the flexibility for a large number of possible extension types. Similar behavior can be supported in NSH-over-VXLAN-GPE when using MD Type 2. The flag-based approach taken in GUE strives to simplify implementations by defining a small number of possible extensions used in a fixed order.</t> <t>The Geneve and GUE headers both include alength field, definingLength field that defines the total length of the encapsulation, including the optional extensions. ThislengthLength field simplifies the parsing by transit devices that skip the encapsulation header without parsing its extensions.</t> </section> <section> <!-- A.2.3 --> <name>Critical Extensions</name> <t>The Geneve encapsulation header includes the'C'C field, which indicates whether the current Geneve header includes critical options, that is to say, options which must be parsed by the target NVE. If the endpoint is not able to process a critical option, the packet is discarded.</t> </section> <section> <!-- A.2.4 --> <name>Maximal Header Length</name> <t>The maximal header length in Geneve, including options, is 260 octets. GUE defines the maximal header to be 128 octets. VXLAN-GPE uses a fixed-length header of 8 octets, unless NSH-over-VXLAN-GPE is used, yielding an encapsulation header of up to 264 octets.</t> </section> </section> <section> <!-- A.3 --> <name>Encapsulation Header</name> <section> <!-- A.3.1 --> <name>Virtual Network Identifier (VNI)</name> <t>The Geneve and VXLAN-GPE headers both include a 24-bit VNI field. GUE, on the other hand, enables the use of a 32-bit field called VNID; this field is not included in the GUEheader,header but was defined as an optional extension in <xreftarget="ietf_gue_extensions"/>.</t>target="I-D.hy-nvo3-gue-4-nvo"/>.</t> <t>The VXLAN-GPE header includes the'I'I bit, indicating that the VNI field is valid in the current header. A similar indicator is defined as a flag in the GUE header <xreftarget="ietf_gue_extensions"/>.</t>target="I-D.ietf-intarea-gue-extensions"/>.</t> </section> <section> <!-- A.3.2 --> <name>Next Protocol</name> <t>All three encapsulation headers include a field that specifies the type of the next protocol header, which resides after the NVO3 encapsulation header. The Geneve header includes a 16-bit field that uses the IEEE Ethertype convention. GUE uses an 8-bit field, which uses the IANAInternetprotocol numbering. The VXLAN-GPE header incorporates an 8-bit Next Protocol field, using aVXLAN-GPE-specific registry,registry specific to VXLAN-GPE, defined in <xreftarget="nvo3_vxlan_gpe"/>.</t>target="I-D.ietf-nvo3-vxlan-gpe"/>.</t> <t>The VXLAN-GPE header also includes the'P'P bit, which explicitly indicates whether the Next Protocol field is present in the current header.</t> </section> <section> <!-- A.3.3 --> <name>Other Header Fields</name> <t>The OAM bit, which is defined in Geneve and in VXLAN-GPE, indicates whether the current packet is an OAM packet. The GUE header includes a similarfield,field but uses different terminology; the GUE'C-bit'C bit (Control bit) specifies whether the current packet is a control packet. Note that the GUEcontrolC bit can potentially be used in a large set of protocols that are not OAM protocols. However, the control packet examples discussed in <xreftarget="ietf_intarea_gue"/>target="I-D.ietf-intarea-gue"/> areOAM-related.</t>related to OAM.</t> <t>Each of the three NVO3 encapsulation headers includes a 2-bit Version field, which is currently defined to be zero.</t> <t>The Geneve and VXLAN-GPE headers include reserved fields; 14 bits in the Geneveheader,header and 27 bits in the VXLAN-GPE header are reserved.</t> </section> </section> <section> <!-- A.4 --> <name>Comparison Summary</name> <t>The following table summarizes the comparison between the three NVO3 encapsulations. In somecasescases, a plus sign ("+") or minus sign ("-") is used to indicate that the header is stronger or weaker in anareaarea, respectively.</t><figure anchor="ComparisonChart"> <name>NVO3 Encapsulations Comparison</name> <artwork type="ascii-art"<table anchor="EncapsulationsComparisonTable" align="center"><![CDATA[ +----------------+----------------+----------------+----------------+ | | Geneve | GUE | VXLAN-GPE | +----------------+----------------+----------------+----------------+ | Outer transport| UDP/IP | UDP/IP | UDP/IP | |<name>Encapsulations Comparison</name> <thead> <tr> <th></th> <th>Geneve</th> <th>GUE</th> <th>VXLAN-GPE</th> </tr> </thead> <tbody> <tr> <td>Outer transport UDP PortNumber| 6081 | 6080 | 4790 | +----------------+----------------+----------------+----------------+ | Base header | 8 octets | 4Number</td> <td>UDP/IP 6081</td> <td>UDP/IP 6080</td> <td>UDP/IP 4790</td> </tr> <tr> <td>Base header length</td> <td>8 octets</td> <td>4 octets</td> <td>8 octets| 8 octets | | length | | |(16 octets| | | | |usingNSH) | +----------------+----------------+----------------+----------------+ | Extensibility |Variable length |Extension fields| No native ext- | | | options | | ensibility. | | | | |an NSH)</td> </tr> <tr> <td>Extensibility</td> <td>Variable-length options</td> <td>Extension fields</td> <td>No innate extensibility. Might useNSH. | +----------------+----------------+----------------+----------------+ | Extension | TLV-based | Flag-based | TLV-based | |an NSH.</td> </tr> <tr> <td>Extension parsingmethod | | |(usingmethod</td> <td>TLV-based</td> <td>Flag-based</td> <td>TLV-based (using an NSH with| | | | |MD Type2) | +----------------+----------------+----------------+----------------+ | Extension | Variable | Fixed | Variable | | order | | |2)</td> </tr> <tr> <td>Extension order</td> <td>Variable</td> <td>Fixed</td> <td>Variable (usingNSH) | +----------------+----------------+----------------+----------------+ | Length field | + | + | - | +----------------+----------------+----------------+----------------+ | Max Header | 260 octets | 128 octets | 8an NSH)</td> </tr> <tr> <td>Length field</td> <td>+</td> <td>+</td> <td>-</td> </tr> <tr> <td>Max header length</td> <td>260 octets</td> <td>128 octets</td> <td>8 octets| | Length | | |(264(264 usingNSH) | +----------------+----------------+----------------+----------------+ | Critical exte- | + | - | - | | nsion bit | | | | +----------------+----------------+----------------+----------------+ | VNIan NSH)</td> </tr> <tr> <td>Critical extension bit</td> <td>+</td> <td>-</td> <td>-</td> </tr> <tr> <td>VNI fieldsize | 24size</td> <td>24 bits</td> <td>32 bits| 32(extension)</td> <td>24 bits</td> </tr> <tr> <td>Next Protocol field</td> <td>16 bits| 24Ethertype registry</td> <td>8 bits| | | | (extension) | | +----------------+----------------+----------------+----------------+ | NextInternet protocol| 16 bits | 8registry</td> <td>8 bits| 8 bits | | field | Ethertype | Internet prot- |Newregistry | | | registry | ocol registry | | +----------------+----------------+----------------+----------------+ | Nextregistry</td> </tr> <tr> <td>Next protocol| - | - | + | | indicator | | | | +----------------+----------------+----------------+----------------+ | OAMindicator</td> <td>-</td> <td>-</td> <td>+</td> </tr> <tr> <td>OAM /control | OAM bit |Controlbit | OAM bit | | field | | | | +----------------+----------------+----------------+----------------+ | Version field | 2 bits | 2 bits | 2 bits | +----------------+----------------+----------------+----------------+ | Reserved bits | 14 bits | none | 27 bits | +----------------+----------------+----------------+----------------+ ]]> </artwork> </figure>field</td> <td>OAM bit</td> <td>Control bit</td> <td>OAM bit</td> </tr> <tr> <td>Version field</td> <td>2 bits</td> <td>2 bits</td> <td>2 bits</td> </tr> <tr> <td>Reserved bits</td> <td>14 bits</td> <td>none</td> <td>27 bits</td> </tr> </tbody> </table> </section> </section> <section anchor="Acknowledgements" numbered="false"> <name>Acknowledgements</name> <t>The authors would like to thank <contact fullname="Tom Herbert"/> for providing the motivation for the security/integrity extension and for his valuable comments; <contact fullname="T. Sridhar"/> for his valuable comments and feedback; <contact fullname="Anoop Ghanwani"/> for his extensive comments; and <contact fullname="Ignas Bagdonas"/>.</t> </section> <section anchor="Contributors" numbered="false"> <name>Contributors</name> <t>The followingco-authorscoauthors have contributed to thisdocument:.</t>document:</t> <contact fullname="Ilango Ganga"> <organization>Intel</organization> <address> <email>ilango.s.ganga@intel.com</email> </address> </contact> <contact fullname="Pankaj Garg"> <organization>Microsoft</organization> <address> <email> pankajg@microsoft.com</email> </address> </contact> <contact fullname="Rajeev Manur"> <organization>Broadcom</organization> <address> <email>rajeev.manur@broadcom.com</email> </address> </contact> <contact fullname="Tal Mizrahi"> <organization>Huawei</organization> <address> <email>tal.mizrahi.phd@gmail.com</email> </address> </contact> <contact fullname="David Mozes"> <address> <email>mosesster@gmail.com</email> </address> </contact> <contact fullname="Erik Nordmark"> <organization>ZEDEDA</organization> <address> <email>nordmark@sonic.net</email> </address> </contact> <contact fullname="Michael Smith"> <organization>Cisco</organization> <address> <email>michsmit@cisco.com</email> </address> </contact> <contact fullname="Sam Aldrin"> <organization>Google</organization> <address> <email>aldrin.ietf@gmail.com</email> </address> </contact> </section> </back> </rfc>