Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Turbo Mode:Enable

Storage Notes

• All storage accessed with 2 x 8 Gb Fibre Channel links. And the SUT was accessed via for fibre channel controllers
• 1 x HUASY Dorado2100 units were used via 1 x 8 Gb drive port.HUASY Dorado2100 holds 24 x 100 GB SSD drives. There is 1 RAID5 array, consisting of 24 drives. Array has 7 LUNs each for 1 tile.
• The Server RH2288H V2-12L of SUT holds 2 x 300 GB SAS drives. There is 1 RAID1 array, consisting of 2 drives. This array was used as local disk for installing Virtualization Platform. And the boot device is on the local disk /dev/sda.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• kernel.sched_min_granularity_ns = 80000000
• kernel.sched_wakeup_granularity_ns = 150000000
• kernel.sched_latency_ns = 160000000
• kernel.sched_migration_cost_ns = 205000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• net.ipv6.conf.all.disable_ipv6 = 1
• net.core.netdev_max_backlog = 500000
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• fs.aio-max-nr = 70656
• The following boot options were used:
• oops=panic net.ifnames=0 biosdevname=0 noexec=on 8250.nr_uarts=8 efi=old_map nmi_watchdog=0 intel_iommu=on iommu=pt selinux=0 pci=realloc console=tty0 console=ttyS0,115200 nohz=on highres=on hpet=disable reserve_kbox_mem=16M crashkernel=334M@48M panic=3 crash_kexec_post_notifiers audit=0 coredump_filter=0x33f LANG=en_US.UTF-8 clocksource=tsc tsc=reliable cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb rdshell nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce idle=poll
• The following services are disabled:
• dm-event iscsid iscsiuio kdump libvirt-guests getosstat irqbalance kbox guest_kbox_ram sysmonitor cron openvswitch system-ovs euleros-security haveged mcelog sysalarm sysstat UVPHostd vBMC_agentd systemd-readahead-collect systemd-readahead-drop systemd-readahead-replay systemd_cgconfig ipmi sigma-serial rsyslog
• After boot up, the following options were used:
• ixgbe module option used: InterruptThrottleRate=4000
• 2 Intel 82599 configured for SUT-client and infra-webserver NFS communication. A maximum of 128 virtual functions were configured and 51 virtual functions were used for the Web, App, DB, Mail Infra and Batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• The 1st dbserver and 2nd dbserver were bound to NUMA node 1 and prefer to allocate memory from node 1
• All VMs of tiles 1,2,3,4 (except the 1st dbserver which are bound to NUMA node 1) are bound to NUMA node 0 prefer to allocate memory from node 0
• All VMs of tiles 5,6,7 (except all of the batchservers and mailserver7 are bound to NUMA node 0) are bound to NUMA node 1 prefer to allocate memory from node 1
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template. (Each VM's xml template is at System_Under_Test/vmconfig directory):
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu<>
• <cpu mode='host-passthrough'>
• <topology sockets='$sockets' cores='$cores' threads='$threads'/>
• </cpu>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic eoi='on'/>
• <pae/>
• </features>
• <clock offset='utc'>
• <timer name='rtc' tickpolicy='catchup' track='guest'/>
• <timer name='hpet' present='no'/>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk'/>
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='yes'>
• <driver name='vfio'/>
• <source>
• <address bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <input type='tablet' bus='usb'/>
• <input type='mouse' bus='ps2'/>
• <graphics type='vnc' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <video>
• <model type='cirrus' heads='1'/>
• </video>
• <memballoon model='none'/>
• </devices>
• </domain>
• The following section is repeated for the second disk used in the db,mail and infra VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vdb' bus='virtio'/>
• </disk>

Web Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout = 60
• net.ipv4.tcp_tw_recycle = 1
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/nginx/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_protocols SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Application Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• Glassfish domain.xml adds:
• To jdbc-connection-pool line parameter 'max-pool-size = 512' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size= 512' was set
• To mdb-container line, parameter 'max-pool-size = 512' was set
• jvm-options '-Xmn1600m' was set
• jvm-options '-XX:+DisableExplicitGC' was set
• jvm-options '-XX:+UseParallelOldGC' was set
• jvm-options '-XX:+UseLargePages' was set
• jvm-options '-XX:-UseBiasedLocking' was set
• jvm-options '-XX:CompileThreshold=1000' was set
• jvm-options '-XX:ParallelGCThreads=4' was set
• jvm-options '-XX:+AggressiveOpts' was set
• jvm-options '-XX:+UseCompressedOops' was set
• jvm-options '-XX:+UseFastAccessorMethods' was set
• jvm-options '-XX:+UseStringCache' was set
• jvm-options '-XX:+OptimizeStringConcat' was set
• jvm-options '-XX:LargePageSizeInBytes=2m' was set
• domain.xml changes:
• jvm-options '-Xmx4800m' was set
• jvm-options '-Xms4800m' was set
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2500
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Mail Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=1000
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Database Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Database tuning options:
• large-pages
• lower_case_table_name = 1
• autocommit = 0
• datadir = /dbstore/mysql
• max_connections = 1500
• max_connect_errors = 6000
• innodb_buffer_pool_size = 8192M
• innodb_buffer_pool_instances = 1
• innodb_flush_method = O_DIRECT
• innodb_log_buffer_size= 1024M
• innodb_log_file_size= 1024M
• innodb_log_files_in_group = 3
• innodb_lock_wait_timeout = 50
• innodb_additional_mem_pool_size = 20M
• table_open_cache = 30000
• table_open_cache_instances = 16
• sort_buffer_size = 8M
• read_buffer_size = 8M
• read_rnd_buffer_size = 512M
• key_buffer_size = 32M
• back_log = 1560
• thread_cache_size = 256
• tmp_table_size = 512M
• max_allowed_packet = 128M
• skip-name-resolve = 1
• innodb_checksums = 0
• innodb_doublewrite = 0
• innodb_support_xa = 0
• innodb_thread_concurrency = 0
• innodb_flush_log_at_trx_commit = 1
• innodb_max_dirty_pages_pct= 75
• innodb_use_native_aio=1
• innodb_stats_persistent = 1
• innodb_adaptive_flushing = 1
• innodb_flush_neighbors = 0
• innodb_read_io_threads = 8
• innodb_write_io_threads = 8
• innodb_io_capacity = 40000
• innodb_purge_threads = 1
• innodb_adaptive_hash_index = 1
• innodb_monitor_enable = '%'
• performance_schema = OFF
• innodb_checksum_algorithm=NONE
• Following items were added to /etc/sysctl.conf:
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400
• net.ipv4.tcp_tw_reuse = 1
• net.ipv6.conf.all.disable_ipv6=1
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=1500
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Infraserver VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• httpd.conf changes:
• Timeout 20
• KeepAlive Off
• MinSpareServers 20
• StartServers 20
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Batch Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=off
• The following services are disabled:
• acpid abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 1048576 > /dev/null 2>&1
• ulimit -u 1000000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• /etc/security/limits.conf changes:
• hard and soft memlock set to unlimited

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 4 VCPUs, 6144 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: