Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
•   IMM.ThermalModePolicy=Performance
•   Memory.MemoryMode=Independent
•   Memory.PatrolScrub=Disable
•   OperatingModes.ChooseOperatingMode=Maximum Performance
•   Power.PlatformControlledType=Maximum Performance
•   Processors.C1EnhancedMode=Disable
•   Processors.PackageACPIC-StateLimit=ACPI C2
•   Processors.ProcessorPerformanceStates=Disable

Storage Notes

• The System Under Test has 16 400GB internal SSDs configured as two RAID5 arrays with eight disks each.
• The first array contains 17 logical volumes for VM images and 1 logical volume for the hypervisor image. The second array contains 16 logical volumes for VM images.
• Each of the VM logical volumes has 6 partitions, 1 for each of the 6 VM types in a tile.

Virtualization Software Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb rdshell intel_iommu=on iommu=pt pci=realloc nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce nmi_watchdog=0
• The following services are configured to be disabled during runlevel initialization:
•   abrtd, atd, cgconfig, cgred, crond, dnsmasq, ebtables, ip6tables, ipmi, ipmievd, iptables, irqbalance, ksm, ksmtuned, libvirt-guests, mdmonitor, netconsole, nfs, ntpd, ntpdate, numad, oddjobd, postfix, psacct, quota_nld, radvd, rdisc, restorecond, rngd, rpcsvcgssd, saslauthd, smartd, sssd, winbind, ypbind
• Configuration file /etc/modprobe.d/ixgbe.conf includes:
•   options ixgbe max_vfs=24
• The following line was added to /etc/modprobe.d/blacklist.conf:
•   blacklist ixgbevf
• tuned was activated with the virtual-host profile.
• The following items were added in /etc/sysctl.conf:
•   kernel.sched_min_granularity_ns = 10000000
•   kernel.sched_wakeup_granularity_ns = 10000000
•   kernel.sched_latency_ns = 32000000
•   kernel.sched_migration_cost = 250000
•   vm.dirty_background_ratio = 5
•   vm.dirty_ratio = 10
•   vm.swappiness = 0
• Network configuration is as follows:
•   174 virtual functions were used for all the VMs.
•   Ethernet devices eth2, eth3, eth4, eth5, eth6, eth7, eth8, and eth9 are the physical functions that are used by the host to provide physical transport for the virtual functions.
•   The IRQs for each virtual function PCI device are pinned to the CPUs on the socket to which the PCI device is routed.
•   Each VM is given a virtual function that is routed to the NUMA node to which the VM's VCPUs are pinned.
• VM configurations are as follows:
•   All VMs are managed with libvirt and created using the virsh utility.
•   All VMs are created with the following XML template (exceptions noted below template):
•    <domain type='kvm'>
•     <name>$vm_name</name>
•     <memory>$vm_memory</memory>
•     <currentMemory>$vm_memory</currentMemory>
•     <vcpu cpuset='$host-cpu-list'>$vm_vcpus</vcpu>
•     <os>
•      <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
•      <boot dev='hd'/>
•     </os>
•     <features>
•      <acpi/>
•      <apic eoi='on'/>
•      <pae/>
•     </features>
•     <cpu mode='host-passthrough'>
•      <topology sockets='1' cores='$vm_vcpus' threads='1'/>
•     </cpu>
•     <clock offset='utc'>
•      <timer name='pit' tickpolicy='delay'/>
•     </clock>
•     <on_poweroff>destroy</on_poweroff>
•     <on_reboot>restart</on_reboot>
•     <on_crash>restart</on_crash>
•     <devices>
•      <emulator>/usr/libexec/qemu-kvm</emulator>
•      <disk type='block' device='disk'>
•       <driver name='qemu' type='raw' cache='none' io='native'/>
•       <source dev='$vm_disk'/>
•       <target dev='vda' bus='virtio'/>
•      </disk>
•      <hostdev mode='subsystem' type='pci' managed='no'>
•       <source>
•        <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
•       </source>
•      </hostdev>
•      <serial type='pty'>
•       <target port='0'/>
•      </serial>
•      <console type='pty'>
•       <target type='serial' port='0'/>
•      </console>
•      <input type='mouse' bus='ps2'/>
•      <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
•      <memballoon model='none'/>
•     </devices>
•    </domain>
•    VMs are bound to host CPUs via their XML configuration in the following manner:
•     All VMs of tiles 4, 7, 10, 13, 20, 23, 26, and 29 are bound to NUMA node 0.
•     All VMs of tiles 1, 8, 11, 14, 17, 24, 27, and 30 are bound to NUMA node 1.
•     All VMs of tiles 2, 5, 12, 15, 18, 21, 28, and 31 are bound to NUMA node 2.
•     All VMs of tiles 3, 6, 9, 16, 19, 22, 25, and 32 are bound to NUMA node 3.
•     Mail and Batch VMs of tile 33 are bound to NUMA node 0.
•     Db VM of tile 33 is bound to NUMA node 1.
•     App VM of tile 33 is bound to NUMA node 2.
•     Web and Infra VMs of tile 58 are bound to NUMA node 3.
•   App VMs are reniced -16.
•   Db VMs are reniced -8.
•   Web VMs are reniced -4.
•   Infra, Mail, and Batch VMs are reniced 19.

Web Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
•   ixgbevf module option used: InterruptThrottleRate=2000
• root file-system options: barrier=0,noatime
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 81920
•   net.ipv4.tcp_max_tw_buckets = 1500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.tcp_tw_reuse=1
•   net.ipv4.tcp_tw_recycle = 1
•   net.ipv4.tcp_keepalive_time = 120
•   net.ipv4.tcp_keepalive_intvl = 30
•   net.ipv4.tcp_timestamps = 0
•   vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited
• httpd.conf changes:
•   KeepAlive On
•   MaxKeepAliveRequests 0
•   KeepAliveTimeout 150
•   MinSpareServers 50
•   MaxSpareServers 100
•   ServerLimit 10240
•   MaxClients 10240
•   MaxRequestsPerChild 32000
•   StartServers 8
•   The following modules were disabled:
•    auth_basic auth_digest authn_file authn_alias authn_anon authn_dbm authn_default authz_user authz_owner authz_groupfile authz_dbm authz_default ldap authnz_ldap include env ext_filter expires deflate usertrack dav status info dav_fs negotiation actions speling userdir substitute rewrite proxy proxy_balancer proxy_ftp proxy_http proxy_ajp proxy_connect cache suexec disk_cache cgi version
•   LogLevel error
•   The following httpd language modules were disabled:
•    ca cs da de el en eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW
•   #LanguagePriority
•   #ForceLanguagePriority
•   #AddHandler type-map var
•   #AddType text/html .shtml
•   #AddOutputFilter INCLUDES .shtml
•   #BrowserMatch "Mozilla/2" nokeepalive
•   #BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
•   #BrowserMatch "RealPlayer 4\.0" force-response-1.0
•   #BrowserMatch "Java/1\.0" force-response-1.0
•   #BrowserMatch "JDK/1\.0" force-response-1.0
•   #BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
•   #BrowserMatch "MS FrontPage" redirect-carefully
•   #BrowserMatch "^WebDrive" redirect-carefully
•   #BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
•   #BrowserMatch "^gnome-vfs/1.0" redirect-carefully
•   #BrowserMatch "^XML Spy" redirect-carefully
•   #BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully
• ssl.conf changes:
•   #CustomLog
• Web VM used NFS to access files from infra VM.
• After the Infra VM is booted and its NFS service is up, the Web VM mounts the NFS file-system via test automation software.
• The datastore is mounted on /var/log/httpd
• NFS mount options used: noatime,actimeo=14400,nocto

Application Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, mlnx-en.d, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
•   ixgbevf module option used: InterruptThrottleRate=2500
• root file-system options: barrier=0
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.
• Heap size for AppSrv01 has been changed to 3072 MB.
• The following changes were made to the defaults in run.properties:
•   runLO=4
•   warmUpStyle = 1
•   linearWarmUpPercentage = .60

Mail Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
•   ixgbevf module option used: InterruptThrottleRate=1000
• root file-system options: barrier=0,noatime,data=writeback
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited
• Dovecot was compiled with default options, and the following dovecot.conf was used:
•   base_dir = /var/run/dovecot/
•   ssl = no
•   ssl_cert =
•   ssl_key =
•   disable_plaintext_auth = no
•   listen = *
•   log_path = /var/log/dovecot.log
•   mail_debug = no
•   mail_location = dbox:/mailstore/dovecot/maildir/%u
•   mdbox_rotate_size = 8192
•   maildir_very_dirty_syncs = yes
•   mbox_very_dirty_syncs = yes
•   mbox_write_locks = fcntl
•   mmap_disable = yes
•   mail_fsync = never
•   passdb {
•     args = scheme=plain /etc/passwd.dovecot
•     driver = passwd-file
•   }
•   protocols = imap
•   service auth {
•     user = root
•   }
•   service imap-login {
•     inet_listener imap {
•       port = 143
•     }
•     process_min_avail = 10
•     service_count = 0
•   }
•   service imap {
•     process_limit = 1024
•   }
•   userdb {
•     driver = passwd
•   }
•   dict {
•   }

Database Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, mlnx-en.d, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
•   ixgbevf module option used: InterruptThrottleRate=1500
• root file-system options: noatime,barrier=0,data=writeback
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   kernel.randomize_va_space=0
•   kernel.sem = 500 64000 400 512
•   net.core.somaxconn = 32384
•   net.ipv4.tcp_max_tw_buckets = 1000000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited
• 14336 2MB huge-pages were allocated prior to starting the database instance.
• Database tuning options:
•   db2 update db cfg for specdb using DATABASE_MEMORY 7077888
•   db2set DB2_LARGE_PAGE_MEM=DB
•   db2set DB2_PINNED_BP=ON
•   alter bufferpool IBMDEFAULTBP size 6370099
•   alter tablespace roll_space NO FILE SYSTEM CACHING
•   alter tablespace C_cust_space NO FILE SYSTEM CACHING
•   alter tablespace C_supp_space NO FILE SYSTEM CACHING
•   alter tablespace C_site_space NO FILE SYSTEM CACHING
•   alter tablespace C_parts_space NO FILE SYSTEM CACHING
•   alter tablespace C_custinv_space NO FILE SYSTEM CACHING
•   alter tablespace M_parts_space NO FILE SYSTEM CACHING
•   alter tablespace M_bom_space NO FILE SYSTEM CACHING
•   alter tablespace M_wo_space NO FILE SYSTEM CACHING
•   alter tablespace M_lo_space NO FILE SYSTEM CACHING
•   alter tablespace M_inv_space NO FILE SYSTEM CACHING
•   alter tablespace O_cust_space NO FILE SYSTEM CACHING
•   alter tablespace O_ords_space NO FILE SYSTEM CACHING
•   alter tablespace O_ordl_space NO FILE SYSTEM CACHING
•   alter tablespace O_item_space NO FILE SYSTEM CACHING
•   alter tablespace S_comp_space NO FILE SYSTEM CACHING
•   alter tablespace S_sc_space NO FILE SYSTEM CACHING
•   alter tablespace S_po_space NO FILE SYSTEM CACHING
•   alter tablespace S_site_space NO FILE SYSTEM CACHING
•   alter tablespace S_supp_space NO FILE SYSTEM CACHING
•   alter tablespace U_space NO FILE SYSTEM CACHING
•   update database config for specdb using LOGFILSIZ 65536
•   update db cfg for specdb using logprimary 64 logsecond 4

Infraserver VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrtd, autofs, certmonger, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, ntpd, ntpdate, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, tomcat6, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
•   ixgbevf module option used: InterruptThrottleRate=1000
• root file-system options: noatime,barrier=0,data=writeback
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 81920
•   net.ipv4.tcp_max_tw_buckets = 1500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
•   net.ipv4.tcp_tw_reuse=1
•   net.ipv4.tcp_keepalive_time = 120
•   net.ipv4.tcp_keepalive_intvl = 30
•   net.ipv4.tcp_timestamps = 0
•   net.ipv4.tcp_syn_retries = 4
•   net.ipv4.tcp_retries2 = 7
•   vm.swappiness=0
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited
• httpd.conf changes:
•   Listen port to 81
•   ServerLimit 10240
•   MaxClients 10240
• fcgi package was built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4.
• /etc/httpd/conf/fcgi.conf added:
•   LoadModule fastcgi_module modules/mod_fastcgi.so
•   ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
•   FastCgiIpcDir /etc/httpd/fastcgi
•   <Directory "/var/www/fcgi-bin">
•    AllowOverride None
•    Options +ExecCGI -Includes
•    SetHandler fastcgi-script
•    Order allow,deny
•    Allow from all
•   </Directory>
•   AddHandler fastcgi-script fcgi
•   FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server is enabled to serve files to Web VM.
• /etc/exports file contains:
•   /support-download *(rw,async,no_subtree_check,crossmnt,no_root_squash)

Batch Server VM Notes

• The following boot options were used:
•   selinux=0 audit=0 elevator=deadline
• The following services are disabled:
•   abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• root file-system options: barrier=0
• /etc/rc.local was modified to include:
•   echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
•   fs.file-max = 500000
•   net.core.somaxconn = 8192
•   net.ipv4.tcp_max_tw_buckets = 500000
•   net.ipv4.conf.default.arp_filter = 1
•   net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
•   * soft memlock unlimited
•   * hard memlock unlimited

Client Driver Notes

• The first three client systems (System x3550 M5) each host eleven prime client VMs using Red Hat Enterprise Linux 6.6. One client system (IBM System x3550 M2) running Red Hat Enterprise Linux 6.5 is the prime controller.
• Each prime client VM has 4 VCPUs, 8196 MB memory, and a 16 GB disk. 1 ixgbevf device and 1 virtio_blk device are used in all client VMs.

Other Notes

• An external 10Gb Lenovo RackSwitch G8264 switch was used to connect the client systems to the system under test (SUT).
• The SUT used the four dual-port 10Gb PCIe NICs for communication with the VMs.
• The SUT NICs and the external client systems are connected to the switch using 10Gb direct connect cables.
• An external 1Gb Lenovo RackSwitch G8052 switch was used to manage all the systems.
• All systems are connected to the 1Gb switch via a separate cable plugged into a 1Gb port on the system.

Performance Details: