Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Thermal Configuration set to Maximum Cooling
• Turbo Mode:Enable

Storage Notes

• 2 x 300GB internal SAS drivers were connected to the LSI3008 controller,built as RAID1 and used for OS
• The SUT was connected to 2 x HUAWEI Dorado5100 via 2 x Dual port QLogic Corp ISP2532-based 8Gbps Fiber Channel HBA,each HUAWEI Dorado5100 has 4 enclosures,each encloseure has 24 x 200GB SSD drives.
• 2 x HUAWEI Dorado5100 holds 8 x 24 x 200 GB SSD drives. There are 8 RAID10 arrays, each consisting of 24 drives . Each array has 4 LUNs,each LUN was partitioned for the VMs of 1 tile.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• vm.swappiness = 0
• vm.dirty_ratio = 10
• vm.dirty_background_ratio = 5
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 500000
• The following boot options were used:
• hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off elevator=deadline selinux=0 audit=0 intel_iommu=on iommu=pt cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce nmi_watchdog=0 pci=realloc
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, dnsmasq, ebtables, ip6tables, iptables, irqbalance, iscsi, iscsid, ksm, ksmtuned, mdmonitor, netconsole, ntpdate, numad, oddjobd, postfix, psacct, quota_nld, radvd, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcsvcgssd, saslauthd, smartd, sssd, vncserver, winbind, ypbind
• ixgbe module option used: max_vfs=63,63,63,63,63,63,63,63,63,63,63,63 InterruptThrottleRate=4000
• 2 Intel X540 for internal communication between web server & infra server, 4 Intel 82599 configured for SUT-client and app-db communication. A maximum of 768 virtual functions were configured and 190 virtual functions were used for the web, app, db, mail infra and batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 1,6,11,16,17,22,27,32 and dbserver33 were bound to NUMA node 0
• All VMs of tiles 2,5,10,14,18,21,26,30 and appserver33 were bound to NUMA node 1
• All VMs of tiles 3,7,9,15,19,23,25,31,webserver33 and infraserver33 were bound to NUMA node 2
• All VMs of tiles 4,8,12,13,20,24,28,29,batchserver33 and mailserver33 were bound to NUMA node 3
• web, mail, and infra VMs have a cputune shares value of 1024
• db and app VMs have a cputune shares value of 16384
• batch VMs have a cputune shares value of 258
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template. (Each VM's xml template is at System_Under_Test/vmconfig directory):
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <cpu model='host-passthrough'>
• <topology sockets=$sockets cores=$cores threads=$threads/>
• </cpu>
• <cputune> <shares>$shares</shares> </cputune>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <apic eoi='on'/>
• <pae/>
• <acpi/>
• </features>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• web and batch VMs do not use qemu for disk emulator, and therefore have the following declaration instead of 'disk type' block above:
• <controller type='scsi' model='vhost-scsi' index='0'>
• <source wwpn='naa.$vm_scsi' event_idx='on'/>
• <driver queue='$queue_num'/>
• </controller>
• The following section is repeated for the second PCI device used in the app,db,web and infra VMs:
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• The following section is repeated for the second disk used in the db,mail and infra VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vdb' bus='virtio'/>
• </disk>

Web Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• KeepAlive on
• MaxKeepAliveRequests 0
• KeepAliveTimeout 60
• StartServers 10
• MinSpareServers 25
• MaxSpareServers 75
• ServerLimit 4400
• MaxClients 4400
• MaxRequestsPerChild 1024
• CustomLog logs/access_log common
• StartServers 2
• MaxClients 150
• <Directory />
• Options FollowSymLinks
• AllowOverride None
• Order allow,deny
• Allow from all
• </Directory>
• httpd.conf adds:
• ServerName webserver
• Listen 81
• <Directory "/var/www/html/support">
• Options FollowSymLinks
• AllowOverride None
• Order deny,allow
• Allow from all
• </Directory>
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• The following httpd modules were disabled:
• env dav status info dav_fs actions proxy proxy_balancer proxy_ftp proxy_http proxy_ajp proxy_connect cgi
• The following httpd modules were loaded:
• file_cache mem_cache
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Application Server VM Notes

• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• * - nofile 1048576
• Update the JVM from 1.6.0 which is installed with the Websphere 8.5.5 to 1.7.0 which is download from IBM WebSite

Mail Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• mysql hard memlock unlimited
• mysql soft memlock unlimited

Database Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• root file-system defaults options:barrier=0
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 610000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• db2 update database config for specdb using LOGFILSIZ 65536
• db2 update db cfg for specdb using logprimary 54 logsecond 4
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• vm.nr_hugepages=12288
• vm.hugetlb_shm_group=501
• vm.nr_overcommit_hugepages=512
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• * hard nofile 102400
• * soft nofile 102400
• * hard nproc 102400
• * soft nproc 102400

Infraserver VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• Timeout 20
• KeepAlive on
• MaxKeepAliveRequests 0
• KeepAliveTimeout 60
• StartServers 20
• MinSpareServers 20
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• StartServers 20
• MaxClients 4096
• MaxRequestsPerChild 640000
• Listen 81
• ServerName besim:80
• httpd.conf adds:
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgilpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• Addhandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 500
• The following httpd modules were loaded:
• file_cache mem_cache fastcgi
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• NFS export options:
• rw,no_subtree_check,crossmnt,no_root_squash

Batch Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 10 VCPUs, 20480 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: