.. SPDX-FileCopyrightText: 2021 Open Networking Foundation .. SPDX-License-Identifier: Apache-2.0 .. _onos_network_config: Network Configuration ===================== SD-Fabric uses several different types of network configurations. We only focus on ``devices`` and ``ports`` configuration in this section, which are used to provide basic L2/L3 connectivity. See :ref:`advanced-features` for advanced feature configurations. Device Configuration -------------------- Each switch in SD-Fabric requires a device config. .. code-block:: json { "devices" : { "device:leaf1" : { "segmentrouting" : { "ipv4NodeSid" : 101, "ipv4Loopback" : "192.168.0.201", "ipv6NodeSid" : 111, "ipv6Loopback" : "2000::c0a8:0201", "routerMac" : "00:00:00:00:02:01", "isEdgeRouter" : true, "adjacencySids" : [] }, "basic" : { "name": "Leaf1", "managementAddress": "grpc://10.128.100.51:9339?device_id=1", "driver": "stratum-tofino", "pipeconf": "org.stratumproject.fabric-upf-int.montara_sde_9_5_0" } } } } - ``device:leaf1``: arbitrary globally unique identifier of the device, must always be prefixed with ``device:``. - ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in forwarding IPv4 traffic. Can be arbitrary and should be globally unique. - ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally unique and should not be part of the same subnet(s) defined on the data plane ports (see port config). - ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only required when using IPv6. - ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally unique and should not be part of the same subnet(s) defined on the data plane ports (see port config). Only required when using IPv6. - ``routerMac``: Router MAC address. Can be arbitrary and should be globally unique. This MAC address will be used to reply the ARP request for the loopback IP or the Interface IP that will be introduced later. (We recommend using the MAC address of the device's management interface as the router MAC.) - ``isEdgeRouter``: True for leaf switches. False for spine switches. - ``adjacencySids``: Deprecated. Always use an empty array. - ``name``: Human friendly name used in the ONOS UI. - ``managementAddress``: gRPC endpoint of the Stratum device and the P4Runtime internal device ID associated to the ASIC (usually ``1``), in the format of ``grpc://[device_addr]?device_id=[P4Runtime device ID]`` The IP address can be replaced by a domain name. - ``driver``: ``stratum-bmv2`` or ``stratum-tofino``, depending on which switch this is. - ``pipeconf``: the P4 program to deploy on this switch. A list of available pipeconfs can be dumped by running ``pipeconfs`` in the ONOS CLI. When running with Tofino-based devices, we provide pre-installed pipeconfs with ID ``org.stratumproject.._``: - The available *profiles* are: - ``fabric``: for basic L2/L3 capabilities - ``fabric-upf``: with 4G/5G mobile user plane (UPF) support - ``fabric-int``: with INT support - ``fabric-upf-int``: with UPF and INT support - The supported *device-types*: Use the command ``weutil`` in the BMC mode to get the product name. - ``montara``: for dual-pipe Tofino ASIC SKUs (Product name is 100BF-32X) - ``mavericks``: for quad-pipe Tofino ASIC SKUs (Product name is 100BF-32QS) - The Intel/Barefoot SDE version used in Stratum, e.g., ``sde_9_5_0`` .. caution:: You should avoid using reserved MPLS labels for ``ipv4NodeSid`` and ``ipv6NodeSid``. Please check here for the reserved values: http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml .. note:: Most of the SD-Fabric configurations support dynamic updates. Unfortunately, we currently **do not support dynamic device configuration updates**. You will have to restart (reboot) the switch if the corresponding device configuration changes. Having said that, when introducing a completely new device in the network, the device configurations pushed before ONOS connects to the switch for the first time should be applied correctly. Port IDs for Tofino-based devices --------------------------------- Before describing the ONOS netcfg, it is worth nothing how we refer to ports for Tofino-based devices. netcfg uses the format ``device:/``. ```` is a unique, arbitrary value that should be consistent with the ``id`` field defined in Stratum chassis config. See :ref:`singleton_port` for more details. Bridging and Unicast Routing ---------------------------- In the following we illustrate how to enable basic bridging and routing on a per-port basis. .. attention:: - VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports) - VLAN **4090** is reserved for pseudowire transport flow rules on the spines Access Ports ^^^^^^^^^^^^ The necessary but minimum configuration for an access port is simply a VLAN. .. code-block:: json { "ports" : { "device:leaf1/12" : { "interfaces" : [{ "name" : "serverA-intf", "vlan-untagged": 10 }] }, "device:leaf1/16" : { "interfaces" : [{ "name" : "serverB-intf", "vlan-untagged": 10 }] } } } The example above shows two ports (12 and 16) on switch ``leaf1`` that have been assigned to VLAN 10 using the ``vlan-untagged`` keyword. It simply means that packets come in and leave out of these switches untagged, but internally they are assigned VLAN 10 and they belong to the bridging domain defined for VLAN 10. ``name`` is used to associate the interface with a globally unique, user friendly name. It can be omitted. With the configuration shown above, the packets will always be bridged, but they cannot be routed out of the VLAN (e.g. to other subnets). To add the capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar to `interface-vlans or SVIs in traditional networks `_). .. code-block:: json { "ports" : { "device:leaf1/12" : { "interfaces" : [{ "name" : "serverA-intf", "ips" : [ "10.0.1.254/24"], "vlan-untagged": 10 }] }, "device:leaf1/16" : { "interfaces" : [{ "name" : "serverB-intf", "ips" : [ "10.0.1.254/24"], "vlan-untagged": 10 }] } } } In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the gateway IP for hosts in this subnet is ``10.0.1.254/32``. When the desire is to route out of a VLAN, this assignment is currently necessary on all ports configured in the same VLAN. .. note:: Typically we only expect a single subnet for a VLAN. Similar to traditional networks, for us, a subnet == VLAN. Different VLANs should be configured in different subnets. In certain use-cases, it may be necessary to configure multiple subnets in the same VLAN. This is possible by adding more subnet/gateway IPs in the ``ips`` array. .. attention:: The same subnet cannot be configured on multiple leaf switches. We usually configure one subnet for all the ports on the same leaf switch. Tagged Ports ^^^^^^^^^^^^ Tagged port configuration is similar. .. code-block:: json { "ports" : { "device:leaf1/24" : { "interfaces" : [{ "name" : "serverA-intf", "ips" : [ "10.0.2.254/24", "10.0.4.254/24" ], "vlan-tagged" : [ 20, 40 ] }] } } } The configuration above for port 24 on switch ``leaf1`` shows two VLANs 20 and 40 configured on that port, with corresponding subnets and gateway IPs. Note that there is no specific ordering required in the ``ips`` or ``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets. In a future release, we will correlate VLAN and subnets configuration in a more readable way. Native VLAN on Tagged Ports ^^^^^^^^^^^^^^^^^^^^^^^^^^^ An additional configuration ``vlan-native`` possible on tagged ports includes the ability to specify a VLAN (and thus a bridging domain) for incoming untagged packets. Typically, such configuration in trunk ports in traditional networks is referred to a native VLAN. .. code-block:: json { "ports" : { "device:leaf1/24" : { "interfaces" : [ { "name" : "serverA-intf", "ips" : [ "10.0.2.254/24", "10.0.4.254/24", "10.0.1.254/24" ], "vlan-tagged" : [ 20, 40 ], "vlan-native" : 10 }] } } } Note that it is also necessary to configure the subnet/gateway IP corresponding to the native VLAN if you wish to route out of that VLAN. Configuring interface for IPv6 [#f1]_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is similar to configure IPv6 routing. Simply replace the addresses in ``ips`` with IPv6 addresses. For example: .. code-block:: json { "ports" : { "device:leaf1/24" : { "interfaces" : [ { "name" : "serverA-intf", "ips" : [ "10.0.2.254/24", "2000::1ff/120" ], "vlan-tagged" : [ 20, 40 ] }] } } } .. note:: There is a known issue that breaks dynamic VLAN configuration. Until the issue get resolved, you need to restart the switch agent to reinstall the flows. IPv6 Router Advertisement [#f1]_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Router Advertisement overview """"""""""""""""""""""""""""" Router advertisement application is for enabling **Router Advertisement** and **Router Solicitation** functionalities supported by IPv6 routers. More details are available in `RFC 4861 `_. Application identifies which IPv6 interfaces are currently configured in the system and it will try to send out **unsolicited Router Advertisement** (RA) messages from these interfaces. Each such RA message will have two mandatory options named **Source link-layer address** and **MTU**. Additional RA option **prefix** can be enabled using component configuration **raGlobalPrefixConfStatus**. Application also processes **Router Solicitations** (RS) sent from hosts. Upon receiving RS on a particular interface application stops RA transmission in that interface and immediately sends RA targeted to the solicited host. After that application continues unsolicited RA transmission on that interface. Activate and configure RA """"""""""""""""""""""""" RA application can be activated from CLI by running .. code-block:: console onos> app activate routeradvertisement Behavior of RA application is controlled by ONOS component configuration subsystem and following are possible configuration options. - ``raThreadDelay``: Delay between consecutive RA transmissions - ``raPoolSize``: Capacity of thread pool to be used for RA transmissions - ``raFlagMbitStatus``: RA flag “Managed address configuration” enabled/disabled - ``raFlagObitStatus``: RA flag “Other configuration” enabled/disabled - ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router prefixes will be available in RA only if this flag is “true” - ``raGlobalPrefixConfStatus``: Enable switch level global prefix configuration. Once ``raGlobalPrefixConfStatus`` is enabled, RA prefix option is generated from port configuration of device, see for more details. To set the options, following the command (example for ``raOptionPrefixStatus``) .. code-block:: console onos> cfg set org.onosproject.ra.RouterAdvertisementManager raOptionPrefixStatus true Prefix details are picked up from network interface configuration. RA app will filter out link-local IPs while preparing prefixes. For example, in following configuration, Prefix will include only **2001:0558:FF10:04C9::2:1ff/120**. .. code-block:: json { "ports": { "device:leaf2/16": { "interfaces": [{ "ips": [ "192.168.114.1/24", "2001:0558:FF10:04C9::2:1ff/120", "FE80::4EA8:2AFF:FE24:8E5F/120" ], "vlan-untagged": "11", "name": "18-15" }] } } } Global prefix configuration """"""""""""""""""""""""""" In some cases, users may want to have a set of global prefix **advertised on all edge interfaces**. Such prefixes can be configured in **devices** section of network configuration in the following way. .. code-block:: json { "devices": { "device:leaf2": { "routeradvertisement" : { "prefixes": [ "2001:0558:FF10:04C9::3:1ff/120"] } } } } .. note:: When global prefix is configured, RA app will ignore any prefixes configured on switch interfaces. Notes about interface config ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ There is no need to configure ports on switches that are meant to connect to other switches. The VLAN (untagged or tagged) configuration is only meant for ports that are connected to hosts (edge ports). .. image:: ../images/config-vlan.png Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g. VLAN 20 in the figure above. However this does not mean that the ports are in the same bridging domain, because in the fabric, the communication between ToRs is through a routed network. In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can communicate with another host on VLAN 20 (untagged or tagged) connected to a different ToR, but the MAC addresses will change as the traffic goes through a routed network. Please do not use this feature to connect switches in unsupported topologies as shown in the example below. The fabric is not designed to be one big Ethernet fabric. The bridging domain is restricted to one ToR. If the bridging domain is extended across two ToRs directly linked to each other, there is a chance of loops. In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should not be used as such. .. image:: ../images/config-vlan-invalid.png .. rubric:: Footnotes .. [#f1] IPv6 support on the data plane (P4 program) is still work-in-progress.