- Getting Started
- Building Blocks
- Application Onboarding
- Reference Architectures
- Edge Applications
- Cloud Adapters
- Development Kits
- Release history
SPDX-License-Identifier: Apache-2.0 Copyright (c) 2019 Intel Corporation
OpenNESS Experience Kits
- OpenNESS setup playbooks
- Customizing kernel, grub parameters, and tuned profile & variables per host
- IP address range allocation for various CNIs and interfaces
- Default values
- Use newer realtime kernel (3.10.0-1062)
- Use newer non-rt kernel (3.10.0-1062)
- Use tuned 2.9
- Default kernel and configure tuned
- Change amount of HugePages
- Change size of HugePages
- Change amount and size of HugePages
- Remove input output memory management unit (IOMMU) from grub params
- Add custom GRUB parameter
- Configure OVS-DPDK in kube-ovn
- Adding new CNI plugins for Kubernetes (Network Edge)
The OpenNESS Experience Kit (OEK) repository contains a set of Ansible* playbooks for the easy setup of OpenNESS in Network Edge mode.
OpenNESS setup playbooks
Customizing kernel, grub parameters, and tuned profile & variables per host
OEKs allow a user to customize kernel, grub parameters, and tuned profiles by leveraging Ansible’s feature of
groups_vars/[edgenode|controller|edgenode_vca]_groupdirectories contain variables applicable for the respective groups and they can be used in
host_varsto change on per node basis while
group_vars/allcontains cluster wide variables.
OEKs contain a
host_vars/ directory that can be used to place a YAML file (
node01.yml). The file would contain variables that would override roles’ default values.
NOTE: Despite the ability to customize parameters (kernel), it is required to have a clean CentOS* 7.8.2003 operating system installed on hosts (from a minimal ISO image) that will be later deployed from Ansible scripts. This OS shall not have any user customizations.
To override the default value, place the variable’s name and new value in the host’s vars file. For example, the contents of
host_vars/node01.yml that would result in skipping kernel customization on that node:
The following are several common customization scenarios.
IP address range allocation for various CNIs and interfaces
The OpenNESS Experience kits deployment uses/allocates/reserves a set of IP address ranges for different CNIs and interfaces. The server or host IP address should not conflict with the default address allocation. In case if there is a critical need for the server IP address used by the OpenNESS default deployment, it would require to modify the default addresses used by the OpenNESS.
Following files specify the CIDR for CNIs and interfaces. These are the IP address ranges allocated and used by default just for reference.
flavors/media-analytics-vca/all.yml:19:vca_cidr: "22.214.171.124/12" group_vars/all/10-default.yml:90:calico_cidr: "10.243.0.0/16" group_vars/all/10-default.yml:93:flannel_cidr: "10.244.0.0/16" group_vars/all/10-default.yml:96:weavenet_cidr: "10.32.0.0/12" group_vars/all/10-default.yml:99:kubeovn_cidr: "10.16.0.0/16,100.64.0.0/16,10.96.0.0/12" roles/kubernetes/cni/kubeovn/controlplane/templates/crd_local.yml.j2:13: cidrBlock: "192.168..0/24"
The 192.168.x.y is used for SRIOV and interface service IP address allocation in Kube-ovn CNI. So it is not allowed for the server IP address, which conflicting with this range. Completely avoid the range of address defined as per the netmask as it may conflict in routing rules.
Eg. If the server/host IP address is required to use 192.168.x.y while this range by default used for SRIOV interfaces in OpenNESS. The IP address range for cidrBlock in roles/kubernetes/cni/kubeovn/controlplane/templates/crd_local.yml.j2 file can be changed to 192.167..0/24 to use some other IP segment for SRIOV interfaces.
Here are several default values:
# --- machine_setup/custom_kernel kernel_skip: false # use this variable to disable custom kernel installation for host kernel_repo_url: http://linuxsoft.cern.ch/cern/centos/7.8.2003/rt/CentOS-RT.repo kernel_repo_key: http://linuxsoft.cern.ch/cern/centos/7.8.2003/os/x86_64/RPM-GPG-KEY-cern kernel_package: kernel-rt-kvm kernel_devel_package: kernel-rt-devel kernel_version: 3.10.0-1127.19.1.rt56.1116.el7.x86_64 kernel_dependencies_urls:  kernel_dependencies_packages:  # --- machine_setup/grub hugepage_size: "2M" # Or 1G hugepage_amount: "5000" default_grub_params: "hugepagesz= hugepages= intel_iommu=on iommu=pt" additional_grub_params: "" # --- machine_setup/configure_tuned tuned_skip: false # use this variable to skip tuned profile configuration for host tuned_packages: - tuned-2.11.0-8.el7 - http://linuxsoft.cern.ch/scientific/7.8/x86_64/os/Packages/tuned-profiles-realtime-2.11.0-8.el7.noarch.rpm tuned_profile: realtime tuned_vars: | isolated_cores=2-3 nohz=on nohz_full=2-3
Use different realtime kernel (3.10.0-1062)
kernel-rt-kvm-3.10.0-1127.19.1.rt56.1116.el7.x86_64 from buil-in repository is installed.
To use another version (e.g.,
kernel-rt-kvm-3.10.0-1062.9.1.rt56.1033.el7.x86_64), create a
host_var file for the host with content:
Use different non-rt kernel (3.10.0-1062)
The OEK installs a real-time kernel by default. However, the non-rt kernel is present in the official CentOS repository. Therefore, to use a different non-rt kernel, the following overrides must be applied:
kernel_repo_url: "" # package is in default repository, no need to add new repository kernel_package: kernel # instead of kernel-rt-kvm kernel_devel_package: kernel-devel # instead of kernel-rt-devel kernel_version: 3.10.0-1062.el7.x86_64 dpdk_kernel_devel: "" # kernel-devel is in the repository, no need for url with RPM # Since, we're not using rt kernel, we don't need a tuned-profiles-realtime but want to keep the tuned 2.11 tuned_packages: - http://linuxsoft.cern.ch/scientific/7x/x86_64/os/Packages/tuned-2.11.0-8.el7.noarch.rpm tuned_profile: balanced tuned_vars: ""
Use tuned 2.9
tuned_packages: - tuned-2.9.0-1.el7fdp - tuned-profiles-realtime-2.9.0-1.el7fdp
Default kernel and configure tuned
kernel_skip: true # skip kernel customization altogether # update tuned to 2.11 but don't install tuned-profiles-realtime since we're not using rt kernel tuned_packages: - http://linuxsoft.cern.ch/scientific/7x/x86_64/os/Packages/tuned-2.11.0-8.el7.noarch.rpm tuned_profile: balanced tuned_vars: ""
Change amount of HugePages
hugepage_amount: "1000" # default is 5000
Change size of HugePages
hugepage_size: "1G" # default is 2M
Change amount and size of HugePages
hugepage_amount: "10" # default is 5000 hugepage_size: "1G" # default is 2M
Remove input output memory management unit (IOMMU) from grub params
default_grub_params: "hugepagesz= hugepages="
Add custom GRUB parameter
Configure OVS-DPDK in kube-ovn
By default, OVS-DPDK is enabled. To disable it, set a flag:
NOTE: This flag should be set in
roles/kubernetes/cni/kubeovn/common/defaults/main.ymor added to
Additionally, HugePages in the OVS pod can be adjusted once default HugePage settings are changed.
kubeovn_dpdk_socket_mem: "1024,0" # Amount of hugepages reserved for OVS per NUMA node (node 0, node 1, ...) in MB kubeovn_dpdk_hugepage_size: "2Mi" # Default size of hugepages, can be 2Mi or 1Gi kubeovn_dpdk_hugepages: "1Gi" # Total amount of hugepages that can be used by OVS-OVN pod
NOTE: If the machine has multiple NUMA nodes, remember that HugePages must be allocated for each NUMA node. For example, if a machine has two NUMA nodes,
kubeovn_dpdk_socket_mem: "1024,1024"or similar should be specified.
kubeovn_dpdk_socket_memis changed, set the value of
kubeovn_dpdk_hugepagesto be equal to or greater than the sum of
kubeovn_dpdk_socket_memvalues. For example, for
kubeovn_dpdk_socket_mem: "1024,1024", set
kubeovn_dpdk_hugepagesto at least
2Gi(equal to 2048 MB).
kubeovn_dpdk_lcore_maskcan be set on per node basis but the HugePage amount allocated with
kubeovn_dpdk_socket_memcannot be greater than
kubeovn_dpdk_hugepages, which is the same for the whole cluster.
OVS pods limits are configured by:
kubeovn_dpdk_resources_requests: "1Gi" # OVS-OVN pod RAM memory (requested) kubeovn_dpdk_resources_limits: "1Gi" # OVS-OVN pod RAM memory (limit)
CPU settings can be configured using:
kubeovn_dpdk_pmd_cpu_mask: "0x4" # DPDK PMD CPU mask kubeovn_dpdk_lcore_mask: "0x2" # DPDK lcore mask
Adding new CNI plugins for Kubernetes (Network Edge)
- The role that handles CNI deployment must be placed in the
- Subroles for control plane and node (if needed) should be placed in the
- If there is a part of common command for both control plane and node, additional sub-roles can be created:
NOTE: The automatic inclusion of the
commonrole should be handled by Ansible mechanisms (e.g., usage of meta’s
- Name of the main role must be added to the
- If additional requirements must checked before running the playbook (to not have errors during execution), they can be placed in the
roles/kubernetes/cni/tasks/precheck.ymlfile, which is included as a pre_task in plays for both Edge Controller and Edge Node.
The following are basic prechecks that are currently executed:
- Check if any CNI is requested (i.e.,
kubernetes_cniis not empty).
- Check if
sriovis not requested as primary (first on the list) or standalone (only on the list).
- Check if
kubeovnis requested as a primary (first on the list).
- Check if the requested CNI is available (check if some CNI is requested that isn’t present in the
- Check if any CNI is requested (i.e.,
- CNI roles should be as self-contained as possible (unless necessary, CNI-specific tasks should not be present in
- If the CNI needs a custom OpenNESS service (e.g., Interface Service in case of
kube-ovn), it can be added to the
Preferably, such tasks would be contained in a separate task file (e.g.,
roles/openness/controlplane/tasks/kube-ovn.yml) and executed only if the CNI is requested. For example: ```yaml
- name: deploy interface service for kube-ovn include_tasks: kube-ovn.yml when: “‘kubeovn’ in kubernetes_cnis” ```
- If the CNI is used as an additional CNI (with Multus*), the network attachment definition must be supplied (refer to Multus docs for more info).