Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Thermal Configuration set to Maximum Cooling
• Turbo Mode:Enable

Storage Notes

• 2 x 300GB internal SAS drives were connected to the P402i controller, built as RAID1 and used for OS
• 6 x 300GB internal SAS drives were connected to the P402i controller, built as RAID5 and used for VM image
• 2 x D3700 disk enclosures were connected to the 2 port HP Smart Array P431 controllers
• One D3700 contained 21 x 240 GB MLC SSD drives
• Enclosure was partitioned and used as follows:
• Create 1 x RAID5 array with 12 x SSD drives,and used for dbserver's datastore
• Create 1 x RAID5 array with 9 x SSD drives,and used for mailserver's datastore
• The other D3700 contained 16 x 240 GB MLC SSD drives
• Enclosure was partitioned and used as follows:
• Create 2 x RAID5 arrays,and each has 8 x SSD drives.
• One is used for Tile1-4 infraserver data,and another for Tile5-7 infraserver data.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• kernel.core_uses_pid = 1
• vm.dirty_background_ratio = 5
• vm.swappiness = 10
• vm.dirty_ratio = 10
• kernel.numa_balancing = 0
• The following boot options were used:
• quiet nohz=on nohz_full=0-31 selinux=0 audit=0 intel_iommu=on iommu=pt cgroup_disable=memory,blkio
• The following services are disabled:
• acpid ipievd lldpad ntp cron rsync irqbalance nbd-client
• The following lines were added to /etc/rc.local:
• echo 80000000 >/proc/sys/kernel/sched_min_granularity_ns
• echo 150000000 >/proc/sys/kernel/sched_wakeup_granularity_ns
• echo 160000000 >/proc/sys/kernel/sched_latency_ns
• echo 1000000 >/proc/sys/kernel/sched_shares_ratelimit
• echo NO_WAKEUP_PREEMPT >/sys/kernel/debug/sched_features
• touch /var/run/h3c_cas_cvk
• /usr/bin/set-printk-console 2
• 1 hp 560-SFP for internal communication between WEBserver & INFRAserver, a maximum of 60 virtual functions were configured and 51 virtual functions were used for the Web, App, DB, Mail Infra and Batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• The 1st dbserver and 2nd dbserver were bound to NUMA node 0 and prefer to allocate memory from node 0
• All VMs of tiles 1,2,3,4 (except the 1st dbserver which are bound to NUMA node 0) are bound to NUMA node 1 prefer to allocate memory from node 1
• All VMs of tiles 5,6,7 (except all of the mailservers are bound to NUMA node 1) are bound to NUMA node 0 prefer to allocate memory from node 0
• Web VMs have a cputune shares value of 1024
• Infra VMs have a cputune shares value of 64
• Mail VMs have a cputune shares value of 64
• DB and APP VMs have a cputune shares value of 16384
• Batch VMs have a cputune shares value of 256
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template.
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <cputune>
• <shares>$vm_cpushare</shares>
• </cputune>
• <numatune>
• <memory mode='strict' nodeset='$numa_node'/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
• <system>linux</system>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='custom' match='exact'>
• <model fallback='allow'>Westmere</model>
• <feature policy='force' name='x2apic'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'/>
• </devices>
• </domain>

Web Server VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• /etc/fstab changes:
• root file-system options: barrier=0
• nfs file-system defaults options: noacl,noatime,actimeo=14400,nocto
• MTU=9000 for eth21 (internal network).
• The MTU is set after all VMs are started.
• nginx.conf changes:
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• log_format common '$remote_addr - $remote_user [$time_local], "$request"
• $status $body_bytes_sent "
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_log_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• conf.d/default.conf changes:
• listen 80 backlog=8192;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• conf.d/example_ssl.conf changes:
• listen 443 ssl;
• root /var/www/html;
• index index.html index.htm index.php;
• autoindex on;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• ulimit -u unlimited
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Application Server VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• /etc/fstab changes:
• root file-system options: noatime,barrier=0,data=writeback
• Glassfish domain.xml adds:
• To jdbc-connection-pool line parameter 'max-pool-size = 512' was set
• To ejb-container line, parameter 'max-pool-size= 512' was set
• To mdb-container line, parameter 'max-pool-size = 512' was set
• jvm-options '-Xmn1600m' was set
• jvm-options '-XX:+DisableExplicitGC' was set
• jvm-options '-XX:+UseParallelOldGC' was set
• jvm-options '-XX:+UseLargePages' was set
• jvm-options '-XX:-UseBiasedLocking' was set
• jvm-options '-XX:CompileThreshold=1000' was set
• jvm-options '-XX:ParallelGCThreads=4' was set
• jvm-options '-XX:+AggressiveOpts' was set
• jvm-options '-XX:+UseCompressedOops' was set
• jvm-options '-XX:+UseFastAccessorMethods' was set
• jvm-options '-XX:+UseStringCache' was set
• jvm-options '-XX:+OptimizeStringConcat' was set
• jvm-options '-XX:LargePageSizeInBytes=256m' was set
• domain.xml changes:
• jvm-options '-Xmx4800m' was set
• jvm-options '-Xms4800m' was set
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft memlock set to 3072000
• hard and soft nofile set to 130172
• hard and soft nproc set to 130172

Mail Server VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• /etc/fstab changes:
• root file-system options: noatime,barrier=0,data=writeback
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:~/
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mdbox_rotate_size = 8192
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = [email protected]
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• net.ipv4.ip_local_port_range = 4096 65535
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• ulimit -u unlimited
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Database Server VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• /etc/fstab changes:
• root file-system options: noatime,barrier=0,data=writeback
• datadisk file-system options:noatime,nodiratime,nobarrier
• Database tuning options:
• malloc-lib=/usr/lib64/libjemalloc.so.1
• skip-grant-tables
• datadir=/dbstore/mysql
• socket=/var/lib/mysql/mysql.sock
• sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
• large-pages
• lower_case_table_name = 1
• autocommit = 0
• max_connections = 1500
• max_connect_errors = 6000
• innodb_buffer_pool_size = 8192M
• innodb_buffer_pool_instances = 1
• innodb_flush_method = O_DIRECT
• innodb_log_buffer_size= 1024M
• innodb_log_file_size= 1024M
• innodb_log_files_in_group = 3
• innodb_lock_wait_timeout = -1
• innodb_additional_mem_pool_size = 16M
• innodb_checksum_algorithm=NONE
• table_open_cache = 30000
• table_open_cache_instances = 16
• sort_buffer_size = 8M
• read_buffer_size = 8M
• read_rnd_buffer_size = 512M
• key_buffer_size = 32M
• back_log = 1560
• thread_cache_size = 256
• tmp_table_size = 512M
• max_allowed_packet = 128M
• skip-name-resolve = 1
• innodb_thread_concurrency = 0
• innodb_checksums = 0
• innodb_doublewrite = 0
• innodb_support_xa = 0
• innodb_flush_log_at_trx_commit = 0
• innodb_max_dirty_pages_pct= 50
• innodb_use_native_aio=1
• innodb_stats_persistent = 1
• innodb_adaptive_flushing = 1
• innodb_flush_neighbors = 0
• innodb_read_io_threads = 8
• innodb_write_io_threads = 8
• innodb_io_capacity = 40000
• innodb_purge_threads = 1
• innodb_adaptive_hash_index = 1
• innodb_monitor_enable = '%'
• performance_schema = OFF
• Following items were added to /etc/sysctl.conf:
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400
• net.ipv4.tcp_tw_reuse = 1
• net.ipv6.conf.all.disable_ipv6=1
• vm.nr_hugepages = 4600
• vm.hugetlb_shm_group = 27
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• ulimit -u unlimited
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 331072
• hard and soft nproc set to 578528
• hard and soft memlock set to 33554432

Infraserver VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• /etc/fstab changes:
• root file-system options: rw,noatime,barrier=0,data=writeback
• datadisk file-system options:noatime,nodiratime,nobarrier
• MTU=9000 for eth28 (internal network).
• The MTU is set after all VMs are started.
• httpd.conf changes:
• MaxKeepAliveRequests 100
• Timeout 120
• KeepAliveTimeout 15
• StartServers 20
• MinSpareServers 20
• MaxSpareServers 20
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• ulimit -u unlimited
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Batch Server VM Notes

• The following boot options were used:
• selinux=0 elevator=deadline audit=0 cgroup_disable=memory hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• auditd atd mcelogd abrtd haldaemon cpuspeed kdump abrt-ccpp acpid rsyslog messagebus cupsd crond automount irqbalance iptables ip6tables
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.5.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 10240 MB memory, and a 10 GB disk,1 ixgbevf device and 1 virtio_blk device are used.
• Prime client for tile0 have 20VCPUs, and the other prime client VMs have 10 VCPUs.
• Each webserver client VM has 10240 MB memory, and a 10 GB disk,1 ixgbevf device and 1 virtio_blk device are used.
• Prime web client for tile0 have 10 VCPUs, and the other web client VMs have 5 VCPUs.

Other Notes

• A H3C S5830 24-port switch was used to connect client systems to SUT. The 10GbE
• ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: