Manual NUMA Optimizations

VMware ESX Server 2.0

Features | Documentation | Knowledge Base | Discussion Forums

If you have applications that use a lot of memory or have a small number of virtual machines, then you may want to optimize performance by setting your NUMA optimizations manually. However, for most users, ESX Server's automatic NUMA optimizations, as described in the previous section, should provide you with good performance.

There are two NUMA options you may set manually:

CPU affinity — See the following section.
Memory affinity — See Associating Future Virtual Machine Memory Allocations with a NUMA Node.

Typically, to bind a virtual machine to a NUMA node, you should set the virtual machine's CPU affinity to use only the CPUs on the desired node, and set the NUMA memory affinity to the same node.

Note: If you set these optimizations manually, then ESX Server does not automatically "rebalance" the nodes if one node becomes overloaded. You must balance the NUMA nodes to avoid overloading any single NUMA node.

Associating Virtual Machines to a Single NUMA Node

You can improve the performance of the applications on a virtual machine by associating it to the CPU numbers on a single NUMA node (manual CPU affinity). (See NUMA Configuration Information for information on obtaining these CPU numbers.)

VMware Management Interface — Associate a virtual machine to a single NUMA node. Click Edit in the Scheduling Affinity section of the CPU page for the virtual machine. Then click the appropriate choices next to Run on Processor(s) and Do not Run on Processor(s). Click OK.
See Managing CPU Resources from the Management Interface for additional information.
Virtual machine configuration file — Add the following:
sched.cpu.affinity = <set>
where <set> comprises CPU numbers on a single NUMA node. This entry binds all virtual CPUs in this virtual machine to the NUMA node.
For example, typing sched.cpu.affinity = 4,5,6,7 binds this virtual machine to the NUMA node that has physical CPUs 4 through 7.
See Editing the Virtual Machine Configuration File for additional information on this entry.
procfs interface on the service console
/proc/vmware/vm/<id>/cpu/affinity
Write a comma-separated list of the CPU numbers on a single NUMA node. See Using procfs for additional information on this entry.

Note: If you manually set CPU affinity by one of the preceding options, then ESX Server automatically sets the virtual machine's memory to be allocated on the same NUMA node. If you want to disable this feature, you need to change the NUMAAutoMemAffinity configuration option to 0 (zero). For more information on changing this advanced option, see Changing Advanced Settings.

Associating Future Virtual Machine Memory Allocations with a NUMA Node

You can also improve performance by specifying that all future memory allocations on a virtual machine use pages associated with a single NUMA node (manual memory affinity). When the virtual machine uses "local" memory, the performance improves on this virtual machine. (See Obtaining NUMA Statistics to determine the NUMA node number.)

Note: You should specify nodes to be used for future memory allocations only if you have also specified CPU affinity. If you make manual changes only to the memory affinity settings, automatic NUMA rebalancing will not work properly.

Do one of the following:

VMware Management Interface — Associate a virtual machine to a single NUMA node. Click Edit in the Memory Affinity section of the Memory page for the virtual machine. Then click the appropriate choices next to the NUMA nodes. Click OK.
See Managing Memory Resources from the Management Interface for additional information.
Virtual machine configuration file — Add the following:
sched.mem.affinity = <NUMA_node>
where <NUMA_node> is the number of a single NUMA node.
procfs interface on the service console:
/proc/vmware/vm/<id>/mem/affinity
Write the number of the NUMA node.

Example of Binding a Virtual Machine to a Single NUMA Node on an 8-way Server

The following example illustrates manually binding four CPUs to a single NUMA node for a virtual machine. In the example, we want this virtual machine to run only on node 1.

An example output of cat /proc/vmware/NUMA/hardware is:


`System type : IBM x440-compatible`
`# NUMA Nodes : 2`
`Total memory : 14336 MB`
`Node`	`ID`	`MachineMem`	`ManagedMem`	`CPUs`
`0`	`00`	`4096 MB`	`1210 MB`	`0 1 2 3`
`1`	`01`	`10240 MB`	`6143 MB`	`4 5 6 7`

The CPUs — for example, 4, 5, 6 and 7 — are the physical CPU numbers.

Complete one of the following to bind a two-way virtual machine to use only the last four physical CPUs of an eight-processor machine:
- Add the following in the virtual machine's configuration file.
  sched.cpu.affinity = 4,5,6,7
- In the VMware Management Interface, associate a virtual machine to a single NUMA node by checking the appropriate boxes next to Run on Processor(s) in the CPU tab of the virtual machine details page.
Set the virtual machine's memory affinity to specify that all of the virtual machine's memory should be allocated on node 1.
- Add the following in the virtual machine's configuration file.
  sched.mem.affinity = 1

Completing these two steps ensure that the virtual machine runs only on NUMA node 1 and, when possible, allocates memory from the same node.

Prev Contents Last Next