VMware ESX Server 2.0Features | Documentation | Knowledge Base | Discussion ForumsESX Server 2.0 includes additional support for machines that are based on NUMA (Non-Uniform Memory Access) architecture. NUMA machines are made up of multiple nodes (also called CECs on IBM eServer machines). Each node comprises one to four processors and main memory. In a node, each CPU has the same distance from its "local memory." Each processor can access memory on any node, but accessing memory on a different node (referred to as "remote memory") is substantially slower than accessing "local memory" that lies on the same node as the processor. That is, the memory access speed for CPUs on a node vary, depending on the "distance" of the memory from the node. For additional information on NUMA and supported NUMA platforms, refer to the VMware ESX Server2 NUMA Support White Paper, available at www.vmware.com/pdf/esx2_NUMA.pdf. For more information on NUMA management by VMware ESX Server, see the numa(8) man page. This section describes how to obtain statistics about your NUMA system. This command checks for the presence of a NUMA system. If it finds a NUMA system, it also lists the number of nodes, the amount of memory, and the physical CPUs on the NUMA node. Type the following: cat /proc/vmware/NUMA/hardware An example output is:
The absence of the /proc/vmware/NUMA directory indicates that this system is not a NUMA system. The system type indicates the hardware for your NUMA system (in this case, an IBM x445 server). There are two NUMA nodes. The fields in the table are defined as follows:
Total memory tells you how much memory is physically installed on each NUMA node. However not all of this memory may be managed by the VMkernel, as some of the memory is used by the service console. Type the following: cat /proc/vmware/mem/ An example output is:
In this preceding example, the total memory managed by the VMkernel for the NUMA nodes is listed in the Totals row. (This amount may be smaller than the total amount of physical memory on the server machine. Type the following: cat /proc/vmware/vm/<id>/mem/numa An example output is:
The preceding output indicates that the virtual machine, with the specified ID, occupies 51MB of memory on node 0, and no memory on node 1. Note: In this preceding example, the memory affinity is set so that only pages associated with node 0 are allocated for this virtual machine (sched.mem.affinity = 0). If memory affinity had not been set, then typically the output would have shown a more even distribution of memory between nodes 0 and 1. For more information, see Associating Future Virtual Machine Memory Allocations with a NUMA Node. By default, ESX Server balances virtual machines and their related data between the available NUMA nodes. ESX Server attempts to maximize use of "local memory," that lies on the same NUMA node as the virtual machine that is running. ESX Server automatically assigns each virtual machine to a temporary "home" NUMA node. The virtual machine only runs on CPUs in the home node, with access to its "local memory." Periodically, ESX Server compares the utilization levels of all NUMA nodes and attempts to "rebalance" the nodes if one node has a higher utilization level than the other nodes. ESX Server rebalances the nodes by changing a virtual machine's "home" NUMA node from the overutilized node to an underutilized node. When the NUMA nodes are balanced, ESX Server again attempts to maximize use of "local memory." For additional information on this process refer to the numa man page. You may also set affinity manually as described in the next section. If you do so, then ESX Server won't automatically rebalance the nodes, and you must balance the NUMA nodes to avoid overloading any single node. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
