Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• EIST(P-states): Enable
• Turbo Mode: Enable
• C State limit: C0/C1 state
• Memory Lockstep mode: Disable

Storage Notes

• All storage accessed with 4 x 8 Gb Fibre Channel links. And the SUT was accessed via four fibre channel controllers
• 1 x Huawei Dorado5100 was used with 4 disk enclousures (96 drives used) attached to the Dorado5100 via 4 x 8 Gb drive ports. A RAID10 array containing 24 x 200 GB SSD drives is configured on each Dorado5100 disk enclosure. There are 4 RAID10 arrays on Dorado5100. Each RAID10 array has only 4 LUNs,each LUN is used to store 1 tile of VM images, mailserver datastore and infraserver datastore. There are 16 tiles on this Dorado5100 (16 LUNs used).
• 3 x HUASY Dorado2100 were used via 3 x 8 Gb drive port each. Each HUASY Dorado2100 contains 24 x 100 GB SSD drives. There are 3 RAID10 arrays, each consisting of 24 drives (72 drives used). One RAID10 array has 6 LUNs for 6 dbserver's datastore,the second RAID10 array contains 4 LUNs for 4 tiles of VM images, mailserver datastore and infraserver datastore, the last RAID10 array contains 3 LUNs for 3 tiles of VM images, mailserver datastore and infraserver datastore.
• The Server RH5885H V3 of SUT contains 4 x 300 GB SAS drives. A RAID5 array consists of 4 hard disks. This array is used as local disk for installing Virtualization Platform. And the boot device is on the local disk /dev/sda.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• vm.swappiness = 10
• vm.dirty_ratio = 10
• vm.dirty_background_ratio = 5
• The following boot options were used:
• showopts hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off elevator=deadline selinux=0 audit=0 intel_iommu=on iommu=pt cgroup_disable=memory,blkio nmi_watchdog=0
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, dnsmasq, ebtables, ip6tables, iptables, irqbalance, iscsi, iscsid, ksm, ksmtuned, mdmonitor, netconsole, ntpdate, numad, oddjobd, postfix, psacct, quota_nld, radvd, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcsvcgssd, saslauthd, smartd, sssd, vncserver, winbind, ypbind
• After boot up, the following options were used:
• echo 160000000 > /proc/sys/kernel/sched_latency_ns
• echo 80000000 > /proc/sys/kernel/sched_min_granularity_ns
• echo 150000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
• ixgbe module option used: InterruptThrottleRate=4000
• 2 Intel X540 for internal communication between WEBserver & INFRAserver, 4 Intel 82599 configured for SUT-client and APP-DB communication. A maximum of 756 virtual functions were configured and 196 virtual functions were used for the Web, App, DB, Mail Infra and Batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 1,2,3,4 are bound to NUMA node 0
• All VMs of tiles 5,6,7,8 are bound to NUMA node 1
• All VMs of tiles 9,10,11,12 are bound to NUMA node 2
• All VMs of tiles 13,14,15,16 are bound to NUMA node 3
• Tile 17 App,DB,Infra,Web and Mail VMs, Tile 21 App and Batch VMs, Tile 22 Batch VM, Tile 23 App,Infra, Web and Batch VMs are bound to NUMA node 0
• Tile 18 App,Infra,Web and Mail VMs, Tile 22 App,Infra and Web VMs are bound to NUMA node 1
• All VMs of tile 19 (except tile 19 Batch), Tile 17 Db VM, Tile 21 Mail VM, Tile 22 Mail VM and Tile 23 Mail VM are bound to NUMA node 2
• All VMs of tile 20, Tile 21 Infra, Web and Db VMs, Tile 17 Batch VM, Tile 18 Batch VM and Tile 19 Batch VM are bound to NUMA node 3
• Web, Mail, and Infra VMs have a cputune shares value of 1024
• DB and APP VMs have a cputune shares value of 16384
• Batch VMs have a cputune shares value of 128
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template. (Each VM's xml template is at System_Under_Test/vmconfig directory):
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <cpu>
• <model>SandyBridge</model>
• <feature policy='force' name='x2apic'/>
• </cpu>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <pae/>
• <apic eoi='on'/>
• <acpi/>
• </features>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• App Web Batch VMs do not use qemu for disk emulator, and therefore have the following declaration instead of 'disk type' block above:
• <controller type='scsi' model='vhost-scsi' index='0'>
• <source wwpn='naa.$vm_scsi' event_idx='on'/>
• <driver queue='$queue_num'/>
• </controller>

Web Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• MaxKeepAliveRequests 0
• Timeout 60
• KeepAliveTimeout 60
• StartServers 10
• MinSpareServers 25
• MaxSpareServers 75
• ServerLimit 4400
• MaxClients 4400
• MaxRequestsPerChild 1024
• CustomLog logs/access_log common
• Include conf.d/*.conf
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Application Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Glassfish domain.xml adds:
• To jdbc-connection-pool line parameter 'max-pool-size = 512' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size= 512' was set
• To mdb-container line, parameter 'max-pool-size = 512' was set
• jvm-options '-Xmn1600m' was set
• jvm-options '-XX:+DisableExplicitGC' was set
• jvm-options '-XX:+UseParallelOldGC' was set
• jvm-options '-XX:+UseLargePages' was set
• jvm-options '-XX:-UseBiasedLocking' was set
• jvm-options '-XX:CompileThreshold=1000' was set
• jvm-options '-XX:ParallelGCThreads=4' was set
• jvm-options '-XX:+AggressiveOpts' was set
• jvm-options '-XX:+UseCompressedOops' was set
• jvm-options '-XX:+UseFastAccessorMethods' was set
• jvm-options '-XX:+UseStringCache' was set
• jvm-options '-XX:+OptimizeStringConcat' was set
• jvm-options '-XX:LargePageSizeInBytes=2m' was set
• domain.xml changes:
• jvm-options '-Xmx4800m' was set
• jvm-options '-Xms4800m' was set
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• nvm.swappiness=0
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft memlock set to 3072000
• hard and soft nofile set to 130172
• hard and soft nproc set to 130172

Mail Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Database Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Database tuning options:
• large-pages
• lower_case_table_name = 1
• autocommit = 0
• datadir = /dbstore/mysql
• max_connections = 1500
• max_connect_errors = 6000
• innodb_buffer_pool_size = 8192M
• innodb_buffer_pool_instances = 1
• innodb_flush_method = O_DIRECT
• innodb_log_buffer_size= 1024M
• innodb_log_file_size= 1024M
• innodb_log_files_in_group = 3
• innodb_lock_wait_timeout = -1
• innodb_additional_mem_pool_size = 16M
• table_open_cache = 30000
• table_open_cache_instances = 16
• sort_buffer_size = 8M
• read_buffer_size = 8M
• read_rnd_buffer_size = 512M
• key_buffer_size = 32M
• back_log = 1560
• thread_cache_size = 128
• tmp_table_size = 512M
• max_allowed_packet = 128M
• skip-name-resolve = 1
• innodb_checksums = 0
• innodb_doublewrite = 0
• innodb_support_xa = 0
• innodb_thread_concurrency = 0
• innodb_flush_log_at_trx_commit = 0
• innodb_max_dirty_pages_pct= 50
• innodb_use_native_aio=1
• innodb_stats_persistent = 1
• innodb_adaptive_flushing = 1
• innodb_flush_neighbors = 0
• innodb_read_io_threads = 8
• innodb_write_io_threads = 8
• innodb_io_capacity = 40000
• innodb_purge_threads = 1
• innodb_adaptive_hash_index = 1
• innodb_monitor_enable = '%'
• performance_schema = OFF
• Following items were added to /etc/sysctl.conf:
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400
• net.ipv4.tcp_tw_reuse = 1
• net.ipv6.conf.all.disable_ipv6=1
• vm.nr_hugepages = 4600
• vm.hugetlb_shm_group = 498
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Infraserver VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• MaxKeepAliveRequests 0
• Timeout 20
• KeepAliveTimeout 60
• StartServers 20
• MinSpareServers 20
• MaxSpareServers 20
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Batch Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 10 VCPUs, 10240 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: