Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• HP Power Profile: Maximum Performance
• Thermal Configuration: Maximum Cooling
• SR-IOV: Enabled
• Processor Power and Utilization Monitoring: Disabled
• Patrol Scrubbing: Disabled

Storage Notes

• 2 x 800GB internal SAS drives were connected to the P841i controller, built as RAID1 and used for OS
• 3 x D3700 disk enclosures were used and contained 24 x 800 GB MLC SSD SAS drives each
• 2 x D3700 enclosures were configured as follows:
• Create 2 x RAID 10 arrays each containing 12 x SSD drives
• Create 8 x logical disks for each RAID array
• Each logical disk was partitioned for the VMs of 1 tile
• Each D3700 was connected to a single port of one HPE Smart Array P431 controllers
• 1 x D3700 enclosure was configured as follows:
• Create 2 x RAID 10 arrays each containing 12 x SSD drives
• Create 10 x logical disks for each RAID array
• Each logical disk was partitioned for the VMs of 1 tile
• D3700 was connected to a single port of one HPE Smart Array P431 controllers

Virtualization Software Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline clocksource=tsc intel_iommu=on pci=realloc cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce
• The following services were disabled:
• abrt-oops abrtd atd avahi-daemon chronyd crond dbus iprdump iprinit iprupdate libstoragemgmt lvm2-lvmetad polkit postfix rhsmcertd rngd smartd firewalld ksmtuned irqbalance NetworkManager
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 500000
• kernel.numa_balancing = 0
• /etc/sysctl.conf was updated with the following lines:
• net.ipv4.ip_forward = 0
• Network configuration is as follows:
• One 331FLR NIC port(eno4) was used for host access
• The 8 x 560SFP+ ports were assigned to bridges (br0,br1,br2,br3,br4,br5,br6,br7)
• The 560SFP+ were configured with
• max_vfs=63
• All the webservers, infraservers, mailservers, appservers and dbservers were assigned with a VF
• Batch VMs did not use VFs, instead virtio-net via the bridge was used
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vmname</name>
• <memory>$vmem</memory>
• <currentMemory>$vmem</currentMemory>
• <vcpu placement='static' cpuset='$host_cpu_list'>$vcpus</vcpu>
• <os>
• <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='custom' match='exact'>
• <model>SandyBridge</model>
• <vendor>Intel/>vendor>
• <topology sockets='1' cores='$vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </disk>
• <interface type='hostdev' managed='yes'/>
• <mac address='$vmac'/>
• <source>
• <address type='pci' domain='0x0000' bus='0x$bus' slot='0x$slot' function='0x$func'/>
• </source>
• </interface>
• <memballoon model='none'>
• </devices>
• </domain>
• Batch VMs had the following declaration instead of 'hostdev' block above:
• <interface type='bridge'/>
• <source bridge='$bridge'/>
• <mac address='$vmac'/>
• <model type='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </interface>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -19
• dbserver = -10
• webserver = -16
• infraserver = 10
• mailserver = 19
• All VMs of tiles 4,7,10,13,20,23,26,29,36,39 and 42 were bound to NUMA node 0
• All VMs of tiles 1,8,11,14,17,24,27,30,33,40 and 43 were bound to NUMA node 1
• All VMs of tiles 2,5,12,15,18,21,28,31,34,37 and 44 were bound to NUMA node 2
• All VMs of tiles 3,6,9,16,19,22,25,32,35,38 and 45 were bound to NUMA node 3
• Appserver45 DBserver45 Mailserver45 Batchserver45 and Batchserver46 were bound to NUMA node 0
• Appserver47 Mailserver47 Batchserver47 Webserver45 and Infraserver45 were bound to NUMA node 2
• Appserver46 and Mailserver46 were bound to NUMA node 3
• Infraserver46 and webserver46 were bound to NUMA node 1

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• ixgbevf module option used: InterruptThrottleRate=3000
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name webserver;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were changed in /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 102400
• apache hard nproc 102400
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 4096 MB
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• ixgbevf module option used: InterruptThrottleRate=2500
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 130172
• * hard nofile 130172
• * soft nproc 130172
• * hard nproc 130172
• spec soft nproc 130172
• spec hard nproc 130172
• * soft memlock 8388608
• * hard memlock 8388608
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• ixgbevf module option used: InterruptThrottleRate=1000
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = [email protected]
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• dovecot soft nproc 102400
• dovecot hard nproc 102400
• dovenull soft nproc 102400
• dovenull hard nproc 102400
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• ixgbevf module option used: InterruptThrottleRate=1500
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6100000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 54 logsecond 4
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 578528
• * hard nproc 578528
• * soft nofile 331072
• * hard nofile 331072
• db2inst1 soft nproc 578528
• db2inst1 hard nproc 578528
• postgres soft nproc 578528
• postgres hard nproc 578528
• * soft memlock 33554432
• * hard memlock 33554432
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes
• Timeout 20
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• CustomLog logs/access_log common
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• ixgbevf module option used: InterruptThrottleRate=3000
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 102400
• apache hard nproc 102400
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cpuspeed, cups, ip6tables, iptables, irqbalance, kdump, mcelogd, netfs, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd, rpcidmapd, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• Batch VMs had the following declaration instead of 'hostdev' block above:
• <interface type='bridge'/>
• <source bridge='$bridge'/>
• <mac address='$vmac'/>
• <model type='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </interface>
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 102400
• * hard nproc 102400
• * soft nofile 102400
• * hard nofile 102400
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 7.1 KVM
• The prime controller ran on the first vClient
• One vClients were used for each tile.
• One vClient per tile ran all the workloads
• Each vClient has 4 vCPUs, 6GB memory, 16GB disk, 1 virtio_blk device and 1 ixgbevf device
• The first vClient acted as prime controller therefore had one more virtio_net device for management
• All physical clients have 2 x 10 Gbps ports, which were configured to have 16 VFs each. Each vClient used one VF.
• The client VM images were provided by the system local disk
• 3 2P physical clients hosted 47 vClients, 16 vClients ran in each physical client.

Other Notes

• 1 x HP H3C S5820-24XG-SFP+ switch was used to connect the clients and the SUT

Performance Details: