Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• HP Power Profile: Maximum Performance
• Thermal Configuration: Maximum Cooling
• SR-IOV: Enabled
• Processor Power and Utilization Monitoring: Disabled
• Memory Refresh Rate: 1x Refresh

Storage Notes

• 2 x 600GB internal SAS drives were connected to the P841i controller, built as RAID1 and used for OS
• Each D3700 disk enclosure was connected to one port of each HP Smart Array P431 controller
• Each D3700 contained 24 x 400 GB MLC SSD drives
• Each enclosure was partitioned and used as follows:
• Create 2 x RAID10 array and each has 12 x SSD drives
• Create 8 x logic disks for each RAID array
• Each logic disk was partitioned for the VMs of 1 tile

Virtualization Software Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline clocksource=tsc intel_iommu=on pci=realloc cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce
• The following services were disabled:
• abrt-oops abrtd atd avahi-daemon chronyd crond dbus iprdump iprinit iprupdate libstoragemgmt lvm2-lvmetad polkit postfix rhsmcertd rngd smartd firewalld ksmtuned irqbalance NetworkManager
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 500000
• kernel.numa_balancing = 0
• Network configuration is as follows:
• One 331FLR NIC port(eno4) was used for host access
• The 4 x 560SFP+ ports were assigned to bridges (br0,br1,br2,br3)
• The 560SFP+ were configured with
• max_vfs=63,63,63,63
• All the VMs assigned with a VF
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vmname</name>
• <memory>$vmem</memory>
• <currentMemory>$vmem</currentMemory>
• <vcpu placement='static' cpuset='$host_cpu_list'>$vcpus</vcpu>
• <os>
• <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='custom' match='exact'>
• <model>SandyBridge</model>
• <vendor>Intel/>vendor>
• <topology sockets='1' cores='$vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </disk>
• <interface type='hostdev' managed='yes'/>
• <mac address='$vmac'/>
• <source>
• <address type='pci' domain='0x0000' bus='0x$bus' slot='0x$slot' function='0x$func'/>
• </source>
• </interface>
• <memballoon model='none'>
• </devices>
• </domain>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -16
• dbserver = -10
• webserver = -10
• infraserver = 19
• mailserver = 19
• All VMs of tiles 1, 6, 11, 16, 17, 22, 27, and 32 were bound to NUMA node 0
• All VMs of tiles 2, 5, 10, 14, 18, 21, 26, and 30 were bound to NUMA node 1
• All VMs of tiles 3, 7, 9, 15, 19, 23, 25, and 31 were bound to NUMA node 2
• All VMs of tiles 4, 8, 12, 13, 20, 24, 28, and 29 were bound to NUMA node 3

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• ixgbevf module option used: InterruptThrottleRate=2000
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• httpd.conf changes
• Timeout 20
• KeepAlive on
• MaxKeepAliveRequest 0
• KeepAliveTimeout 150
• StartServers 4400
• MinSpareServers 50
• MaxSpareServers 100
• ServerLimit 8000
• MaxClients 8000
• MaxRequestsPerChild 32000
• The following httpd modules were not used:
• auth_basic auth_digest authn_file authn_alias authn_anon authn_dbm authn_default authz_user authz_owner authz_groupfile authz_dbm authz_default ldap authnz_ldap include ext_filter expires deflate usertrack dav status info dav_fs negotiation actions speling userdir substitute rewrite proxy proxy_balancer proxy_ftp proxy_http proxy_ajp proxy_connect cache suexec disk_cache cgi version

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2048 MB
• The following change was made to the defaults in run.properties
• runLO=2
• ixgbevf module option used: InterruptThrottleRate=1500

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• ixgbevf module option used: InterruptThrottleRate=1000
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = [email protected]
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• ixgbevf module option used: InterruptThrottleRate=1500
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6100000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 54 logsecond 4

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes
• Timeout 20
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• CustomLog logs/access_log common
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• KeepAlive Off
• Listen 81
• ServerName besim:80
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• ixgbevf module option used: InterruptThrottleRate=3000

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• ixgbevf module option used: InterruptThrottleRate=4000

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.5 KVM
• The prime controller ran on the first vClient
• Two vClients were used for each tile. One vClient per tile ran the
• webserver workload and the other vClient per tile ran the other workloads
• Each vClient has 4 vCPUs, 6GB memory, 16GB disk, 1 virtio_blk device and 1 virtio_net device
• The first vClient also acted as prime controller therefore had one more virtio_net device for management
• All physical clients have 2 x 10 Gbps ports, and were configured as software bridges for the vClients
• The client VM images were provided by the system local disk
• 9 2P physical clients hosted 36 vClients, 4 vClients ran in each physical client. One 4P physical client
• hosted xx vClients

Other Notes

• 1 x HP H3C S5820-24XG-SFP+ switch was used to connect the clients and the SUT

Performance Details: