Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
•   Memory.MemoryPowerManagement=Disable
•   Memory.PatrolScrub=Disable
•   OperatingModes.ChooseOperatingMode=Maximum Performance
•   Power.PlatformControlledType=Maximum Performance
•   Processors.C1EnhancedMode=Disable
•   Processors.PackageACPIC-StateLimit=ACPI C2
•   Processors.ProcessorPerformanceStates=Disable

Storage Notes

• The IBM Flex System V7000 storage node provides VM disk images for the system under test as well as the hypervisor image.
• There are 4 storage pools, each pool comprising 1 RAID5 array, and each array having 6 x 400GB SSDs. Storage pool 1 contains 10 LUNs for VM images and 1 LUN for the hypervisor image. Storage pools 2, 3, and 4 each contain 10 LUNs for VM images.
• Each of the VM LUNs has 6 partitions, 1 for each of the 6 VM types in a tile.

Virtualization Software Notes

• The following boot options were used:
•   intel_iommu=on iommu=pt selinux=0 elevator=deadline cgroup_disable=memory,blkio
• The following services are configured to be disabled during runlevel initialization:
•   abrtd, abrt-oops, atd, avahi-daemon, chronyd, crond, dbus, firewalld, iprdump, iprinit, iprupdate, irqbalance, ksmtuned, libstoragemgmt, lvm2-lvmetad, NetworkManager, polkit, postfix, rhsmcertd, rngd, smartd
• /etc/rc.local was modified to include:
•   echo NO_WAKEUP_PREEMPT >/sys/kernel/debug/sched_features
• The following items were added in /etc/sysctl.conf:
•   kernel.sched_min_granularity_ns = 4000000
•   kernel.sched_wakeup_granularity_ns = 4000000
•   kernel.sched_latency_ns = 32000000
• Network configuration is as follows:
•   The latest Mellanox network drivers, mlnx-en-2.3-1.0.0, were downloaded from the official Mellanox support web site, built with the default options, and installed.
•   Configuration file /etc/modprobe.d/mlx4_core.conf for 40Gb Ethernet includes:
•    options mlx4_core enable_64b_cqe_eqe=1 num_vfs=36
•   210 virtual functions were used for all the VMs.
•   Ethernet devices eth0, eth2, eth4, eth6, eth159, eth161, eth163,and eth165 are the 8 physical functions that are used by the host to provide physical transport for the virtual functions.
•   The PCI devices for the eight physical functions are each routed to one of the eight sockets (NUMA nodes).
•   The IRQs for each PCI device are pinned to the CPUs on the socket to which the PCI device is routed.
•   Each VM is given a virtual function that is routed to the NUMA node to which the VM's VCPUs are pinned.
• VM configurations are as follows:
•   All VMs are managed with libvirt and created using the virsh utility.
•   All VMs are created with the following XML template (exceptions noted below template):
•    <domain type='kvm'>
•     <name>$vm_name</name>
•     <memory>$vm_memory</memory>
•     <currentMemory>$vm_memory</currentMemory>
•     <vcpu cpuset='$host-cpu-list'>$vm_vcpus</vcpu>
•     <os>
•      <type arch='x86_64' machine='pc'>hvm</type>
•      <boot dev='hd'/>
•     </os>
•     <features>
•      <acpi/>
•      <apic/>
•      <pae/>
•     </features>
•     <cpu match='exact'>
•      <model>SandyBridge</model>
•      <feature policy='force' name='x2apic'/>
•     </cpu>
•     <clock offset='utc'>
•      <timer name='pit' tickpolicy='delay'/>
•     </clock>
•     <on_poweroff>destroy</on_poweroff>
•     <on_reboot>restart</on_reboot>
•     <on_crash>restart</on_crash>
•     <devices>
•      <emulator>/usr/libexec/qemu-kvm</emulator>
•      <disk type='block' device='disk'>
•       <driver name='qemu' type='raw' cache='none' io='native'/>
•       <source dev='$vm_disk'/>
•       <target dev='vda' bus='virtio'/>
•      </disk>
•      <hostdev mode='subsystem' type='pci' managed='no'>
•       <source>
•        <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
•       </source>
•      </hostdev>
•      <serial type='pty'>
•       <target port='0'/>
•      </serial>
•      <console type='pty'>
•       <target type='serial' port='0'/>
•      </console>
•      <input type='mouse' bus='ps2'/>
•      <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
•      <memballoon model='none'/>
•     </devices>
•    </domain>
•    VMs are bound to host CPUs via their XML configuration in the following manner:
•     All VMs of tiles 1, 2, 3, and 4 are bound to NUMA node 0.
•     All VMs of tiles 5, 6, 7, and 8 are bound to NUMA node 1.
•     All VMs of tiles 9, 10, 11, and 12 are bound to NUMA node 2.
•     All VMs of tiles 13, 14, 15, and 16 are bound to NUMA node 3.
•     All VMs of tiles 17, 18, 19, and 20 are bound to NUMA node 4.
•     All VMs of tiles 21, 22, 23, and 24 are bound to NUMA node 5.
•     All VMs of tiles 25, 26, 27, and 28 are bound to NUMA node 6.
•     All VMs of tiles 29, 30, 31, and 32 are bound to NUMA node 7.
•     Tile 33 App and Db VMs and tile 34 App VM are bound to the last 28 CPUs on NUMA node 0.
•     Tile 33 Web and Infra VMs, tile 35 App VM, and tile 36 App VM are bound to the last 29 CPUs on NUMA node 1.
•     Tile 33 Batch VM, tile 34 Web and Infra VMs, and tile 35 Web, Infra, and Batch VM are bound to the last 29 CPUs on NUMA node 2.
•     Tile 33 Mail VM, tile 34 Mail and Batch VMs, tile 35 Mail VM, and tile 36 Web, Infra, Mail, and Batch VMs are bound to the last 29 CPUs on NUMA node 3.
•     Tile 37 App and Db VMs and tile 38 App VM are bound to the last 29 CPUs on NUMA node 4.
•     Tile 37 Web and Infra VMs, tile 39 App VM, and tile 40 App VM are bound to the last 29 CPUs on NUMA node 5.
•     Tile 37 Batch VM, tile 38 Web and Infra VMs, and tile 39 Web, Infra, and Batch VM are bound to the last 29 CPUs on NUMA node 6.
•     Tile 37 Mail VM, tile 38 Mail and Batch VMs, tile 39 Mail VM, and tile 40 Web, Infra, Mail, and Batch VMs are bound to the last 29 CPUs on NUMA node 7.
•   App and Db VMs are reniced -10.
•   Web and Infra VMs are reniced +10.
•   Mail VMs are reniced +16.

Web Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• httpd.conf changes:
•   KeepAlive on
•   KeepAliveTimeout 60
•   MaxKeepAliveRequest 0
•   ServerLimit 10240 (for prefork)
•   MaxClients 10240 (for prefork)
• Web VM used NFS to access files from infra VM.
• No options were specified in the NFS mount.
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• After the Infra VM is booted and its NFS service is up, the Web VM mounts the NFS file-system via test automation software.
• root file-system options: barrier=0
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 81920
•   net.ipv4.tcp_max_tw_buckets = 1500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.tcp_tw_reuse=1
•   net.ipv4.tcp_keepalive_time = 120
•   net.ipv4.tcp_keepalive_intvl = 30
•   net.ipv4.tcp_timestamps = 0
•   vm.swappiness=0
• The datastore is mounted on /var/log/httpd

Application Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.
• Heap size for AppSrv01 has been changed to 2048 MB.
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• root file-system options: barrier=0
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   vm.swappiness=0
• The following changes were made to the defaults in run.properties:
•   runLO=4
•   warmUpStyle = 1
•   linearWarmUpPercentage = .50

Mail Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• Dovecot was compiled with default options and the following dovecot.conf was used:
•   base_dir = /var/run/dovecot/
•   ssl = no
•   disable_plaintext_auth = no
•   listen = *
•   log_path = /var/log/dovecot.log
•   mail_debug = no
•   mail_location = dbox:/mailstore/dovecot/maildir/%u
•   mdbox_rotate_size = 2048
•   maildir_very_dirty_syncs = yes
•   mbox_very_dirty_syncs = yes
•   passdb {
•    args = /etc/passwd.dovecot
•    driver = passwd-file
•   }
•   protocols = imap
•   service auth {
•    user = root
•   }
•   service imap-login {
•    inet_listener imap {
•     port = 143
•    }
•    process_min_avail = 10
•   }
•   userdb {
•    driver = passwd
•   }
•   dict {
•   }
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• root file-system options: barrier=0
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1

Database Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   atd, auditd, autofs, cgconfig, cgred, crond, htcacheclean, httpd, ip6tables, iptables, netconsole, nfs, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• Database tuning options:
•   db2 update db cfg for specdb using DATABASE_MEMORY 6100000
•   db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
•   alter bufferpool IBMDEFAULTBP size 5500000
•   alter tablespace roll_space NO FILE SYSTEM CACHING
•   alter tablespace C_cust_space NO FILE SYSTEM CACHING
•   alter tablespace C_supp_space NO FILE SYSTEM CACHING
•   alter tablespace C_site_space NO FILE SYSTEM CACHING
•   alter tablespace C_parts_space NO FILE SYSTEM CACHING
•   alter tablespace C_custinv_space NO FILE SYSTEM CACHING
•   alter tablespace M_parts_space NO FILE SYSTEM CACHING
•   alter tablespace M_bom_space NO FILE SYSTEM CACHING
•   alter tablespace M_wo_space NO FILE SYSTEM CACHING
•   alter tablespace M_lo_space NO FILE SYSTEM CACHING
•   alter tablespace M_inv_space NO FILE SYSTEM CACHING
•   alter tablespace O_cust_space NO FILE SYSTEM CACHING
•   alter tablespace O_ords_space NO FILE SYSTEM CACHING
•   alter tablespace O_ordl_space NO FILE SYSTEM CACHING
•   alter tablespace O_item_space NO FILE SYSTEM CACHING
•   alter tablespace S_comp_space NO FILE SYSTEM CACHING
•   alter tablespace S_sc_space NO FILE SYSTEM CACHING
•   alter tablespace S_po_space NO FILE SYSTEM CACHING
•   alter tablespace S_site_space NO FILE SYSTEM CACHING
•   alter tablespace S_supp_space NO FILE SYSTEM CACHING
•   alter tablespace U_space NO FILE SYSTEM CACHING
•   update database config for specdb using LOGFILSIZ 65536
•   update db cfg for specdb using logprimary 64 logsecond 4
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• 12288 2MB huge-pages were allocated prior to starting the database instance.
• root file-system options: barrier=0
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 32384
•   net.ipv4.tcp_max_tw_buckets = 1000000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   vm.swappiness=0
•   kernel.sem = 500 64000 400 512

Infraserver VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrtd, autofs, certmonger, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, ntpd, ntpdate, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, tomcat6, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• httpd.conf changes:
•   Listen port to 81
•   ServerLimit 10240 (for prefork)
•   MaxClients 10240 (for prefork)
• /etc/httpd/conf/fcgi.conf added:
•   LoadModule fastcgi_module modules/mod_fastcgi.so
•   ScriptAlias /fcgi-bin/ /var/www/fcgi-bin/
•   FastCgiIpcDir /etc/httpd/fastcgi
•   <Directory /var/www/fcgi-bin>
•    AllowOverride None
•    Options +ExecCGI -Includes
•    SetHandler fastcgi-script
•    Order allow,deny
•    Allow from all
•   </Directory>
•   AddHandler fastcgi-script fcgi
•   FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server is enabled to serve files to Web VM.
• /etc/exports file contains:
•   /support-download *(rw,no_subtree_check,crossmnt,no_root_squash)
• fcgi package was built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4.
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• root file-system options: noatime,barrier=0,data=writeback
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 81920
•   net.ipv4.tcp_max_tw_buckets = 1500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.tcp_tw_reuse=1
•   net.ipv4.tcp_keepalive_time = 120
•   net.ipv4.tcp_keepalive_intvl = 30
•   net.ipv4.tcp_timestamps = 0
•   net.ipv4.tcp_syn_retries = 4
•   net.ipv4.tcp_retries2 = 7
•   vm.swappiness=0

Batch Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• /etc/rc.local was modified to include:
•   echo tsc >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with virtual-guest profile.
• root file-system options: barrier=0
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1

Client Driver Notes

• The first two client systems (IBM Flex System x240) each host five prime client VMs using Red Hat Enterprise Linux 6.5. Ten of the remaining client systems each host three prime client VMs. One client system (IBM System x3550 M2) is the prime controller.
• Each prime client VM has 4 VCPUs, 8192 MB memory, and a 32 GB disk. 1 virtio_net device and 1 virtio_blk device are used in all client VMs.

Other Notes

• 2 x Flex System Fabric CN4093 10Gb Converged Scalable Switches are installed in switch bays 1 and 2 of the Flex System Enterprise Chassis.
• The system under test (SUT) is a blade system that plugs into the Flex System Enterprise Chassis. The NICs in the SUT connect to the switches via the backplanes within the chassis.
• The two IBM Flex System x240 clients are also blade systems that are plugged into the chassis and connect to the switches via the backplanes within the chassis.
• The external client systems are connected to the switches using 10Gb direct connect cables.
• 1 x Flex System FC3171 8Gb SAN Switch is installed in switch bay 3 of the chassis and is used to connect the IBM Flex System V7000 to the SUT.
• The IBM Flex System V7000 storage node is also located in the Flex System Enterprise Chassis and has connectivity to the 8Gb fibre channel SAN switch.

Performance Details: