Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Workload Profile: Custom
• Processor Power and Utilization Monitoring: Disabled
• SNC: Disabled
• XPT Prefetcher: Disabled

Storage Notes

• 16 x 800GB internal SAS drives were connected to the P408i-a and P408i-p SR Controllers
• Create 1 x RAID 10 array containing 2 x SSD drives, used for OS, on P408i-a
• Create 1 x RAID 50 array containing 6 x SSD drives on P408i-a
• Create 1 x RAID 50 array containing 8 x SSD drives on P408i-p

Virtualization Software Notes

• Boot parameters set: intel_iommu=on selinux=0 audit=0 cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb elevator=deadline clocksource=tsc pci=realloc nmi_watchdog=0 nosoftlockup ipv6.disable=1 intel_idle.max_cstate=0 mce=ignore_ce
• The following services were disabled:
• abrt-oops abrtd cpupower crond firewalld iprdump iprinit iprupdate irqbalance kdump ksmtuned NetworkManager postfix rhnsd
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• vm.dirty_background_ratio = 2
• vm.dirty_ratio = 5
• vm.swappiness = 0
• kernel.numa_balancing = 0
• kernel.sched_min_granularity_ns = 20000000
• kernel.sched_wakeup_granularity_ns = 25000000
• kernel.sched_migration_cost_ns = 50000
• kernel.sched_latency_ns = 100000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.core.netdev_max_backlog = 500000
• vm.min_free_kbytes = 4096
• /etc/sysctl.conf was updated with the following lines:
• net.ipv4.ip_forward = 0
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.conf.all.arp_ignore = 1
• kernel.msgmnb = 65536
• kernel.msgmax = 65536
• kernel.shmmax = 68719476736
• kernel.shmall = 4294967296
• vm.min_free_kbytes = 4096
• vm.dirty_background_ratio = 2
• vm.dirty_ratio = 5
• Network configuration is as follows:
• One 331i Adapter NIC port(eno1) was used for host access
• The 8 x 560FLR-SFP ports were assigned to bridges (ens1-port1, ens1-port2,ens2-port1, ens2-port2, ens5-port1, ens5-port2, ens6-port1, ens6-port2)
• 48 virtual functions were configured for each port
• Ethernet devices ens1f0,ens1f1,ens2f0,ens2f1,ens5f0,ens5f1,ens6f0 and ens6f1 are the physical functions that are used by the host to provide physical transport for the virtual functions.
• All the guest VMs use Virtual Functions.
• Each VM is given a virtual function that is routed to the NUMA node to which the VM's VCPUs are pinned.
• VM configurations were as follows:
• All VMs were managed by vdsm
• All VMs were created with the following XML template:
• <domain type='kvm' id=$vmid>
• <name>$vm_name;/name>
• <uuid>$vm_uuid</uuid>
• <metadata xmlns:ovirt="http://ovirt.org/vm/tune/1.0">
• <ovirt:qos/>
• </metadata>
• <memory>$vm_memory</memory>
• <currentMemory unit='KiB'>$vm_memory</currentMemory>
• <vcpu placement='static' current=$host_cpu_list</vcpu>
• <cputune>
• <vcpupin vcpu='$vcpu' cpuset='$cpunodeset'/>
• </cputune>
• <numatune>
• <memnode cellid='$' mode='preferred' nodeset=$vm_node/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
• <smbios mode='sysinfo'/>
• </os>
• <features>
• <acpi/>
• </features>
• <cpu mode='custom' match='exact'>
• <model fallback='allow'>Skylake-Client</model>
• <topology sockets=$ cores=$ threads='1'/>
• <numa>
• <cell id=$ cpus=$ memory=$vm_memory unit='KiB'/>
• </numa>
• </cpu>
• <clock offset='variable' adjustment='0' basis='utc'>
• <timer name='rtc' tickpolicy='catchup'/>
• <timer name='pit' tickpolicy='delay'/>
• <timer name='hpet' present='no'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='file' device='cdrom'>
• <driver name='qemu' type='raw'/>
• <source startupPolicy='optional'/>
• <backingStore/>
• <target dev='hdc' bus='ide'/>
• <readonly/>
• <alias name='ide0-1-0'/>
• <address type='drive' controller='0' bus='1' target='0' unit='0'/>
• </disk>
• <disk type='file' device='disk' snapshot='no'>
• <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
• <source file=$/>
• <backingStore/>
• <target dev='vda' bus='virtio'/>
• <boot order='1'/>
• <alias name='virtio-disk0'/>
• </disk>
• <interface type='hostdev'>
• <mac address=$vm_mac_addr/>
• <driver name='vfio'/>
• <source>
• <address type='pci' domain='0x0000' bus=$bus slot=$slot function=$function/>
• </source>
• </interface>
• <channel type='unix'>
• <source mode='bind' path=$path_vdsm/>
• <target type='virtio' name='com.redhat.rhevm.vdsm' state='connected'/>
• <alias name='channel0'/>
• <address type='virtio-serial' controller='0' bus='0' port='1'/>
• </channel>
• <channel type='unix'>
• <source mode='bind' path=$path_guest_agent/>
• <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
• <alias name='channel1'/>
• <address type='virtio-serial' controller='0' bus='0' port='2'/>
• </channel>
• <channel type='spicevmc'>
• <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
• <alias name='channel2'/>
• <address type='virtio-serial' controller='0' bus='0' port='3'/>
• </channel>
• <input type='mouse' bus='ps2'>
• <alias name='input0'/>
• </input>
• <input type='keyboard' bus='ps2'>
• <alias name='input1'/>
• </input>
• <graphics type='spice' tlsPort='$port' autoport='yes' listen=$host_ip defaultMode='secure' passwdValidTo=$>
• <listen type='network' address=$host_ip network='vdsm-ovirtmgmt'/>
• <channel name='main' mode='secure'/>
• <channel name='display' mode='secure'/>
• <channel name='inputs' mode='secure'/>
• <channel name='cursor' mode='secure'/>
• <channel name='playback' mode='secure'/>
• <channel name='record' mode='secure'/>
• <channel name='smartcard' mode='secure'/>
• <channel name='usbredir' mode='secure'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -19
• dbserver = -19
• webserver = -19
• infraserver = 8
• mailserver = 10
• batchserver = 19
• All VMs of tiles 1-8, 17-24, 34 and 36 were bound to NUMA node 0
• All VMs of tiles 9-16, 25-32, 33 and 35 were bound to NUMA node 1
• RHVM configuration:
• Host is configured using Red Hat Virtualization Manager (RHVM).
• The RHVM is a physical server running RHEL 7.3 OS with RHV 4.1 rhevm package installed. SUT is Red Hat
• Enterprise Linux 7.3 host managed by the RHVM. Using RHVM GUI, a new data center, gen10-sc2013, was created.
• Within the data center, one cluster was created, dl380gen10-sc2013, and the SUT was added to cluster.
• Two local storage pools were created, local01 and local02, corresponding to the RAID50 arrays on SUT.
• A virtual network was created for each 10GbE NIC port and a corresponding vNIC profile was configured on each
• virtual network to enable passthrough.

Web Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• net.ipv4.tcp_keepalive_time = 120
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• keepalive_requests 200;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were changed in /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.backlog = 81920
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• pm.max_children = 500
• pm.start_servers = 200
• pm.min_spare_servers = 10
• pm.max_spare_servers = 200
• request_terminate_timeout = 300
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2048 MB
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 130172
• * hard nofile 130172
• * soft nproc 130172
• * hard nproc 130172
• spec soft nproc 1000000
• spec hard nproc 1000000
• * soft memlock 8388608
• * hard memlock 8388608

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = [email protected]
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• dovecot soft nproc 1000000
• dovecot hard nproc 1000000
• dovenull soft nproc 1000000
• dovenull hard nproc 1000000
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6100000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 54 logsecond 4
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 578528
• * hard nproc 578528
• * soft nofile 331072
• * hard nofile 331072
• db2inst1 soft nproc 1000000
• db2inst1 hard nproc 1000000
• * soft memlock 33554432
• * hard memlock 33554432

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes
• Timeout 20
• ## KeepAliveTimeout 15
• ##Include conf.d/*.conf
• AddIcon /icons/bomb.gif core
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• CustomLog logs/access_log common
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 1000000
• apache hard nproc 1000000
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrtd, atd, auditd, autofs, cups, ip6tables, iptables, irqbalance, kdump, nfslock, portreserve, postfix, rhnsd, rhsmcertd, rpcgssd
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 102400
• * hard nproc 102400
• * soft nofile 102400
• * hard nofile 102400
• The source code for the Intel kernel module ixgbevf version 2.14.2 was download from the official web site, built with the default configuration options, and installed.

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 7.1 KVM
• The prime controller ran on the first vClient
• One vClients were used for each tile.
• One vClient per tile ran all the workloads
• Each vClient has 6 vCPUs, 12GB memory, 16GB disk, 1 virtio_blk device and 1 ixgbevf device
• The first vClient acted as prime controller therefore had one more virtio_net device for management
• All physical clients have 2 x 10 Gbps ports, which were configured to have 16 VFs each. Each vClient used one VF.
• The client VM images were provided by the system local disk
• Three physical clients each hosted 10 vClients, and one physical client hosted 6 vClients for a total of 36 vClients.

Other Notes

• 1 x HP H3C S5820-24XG-SFP+ switch was used to connect the clients and the SUT
• The RHVM client system is a DL360 Gen9 server with 2 x 2.0GHz 14-core Intel Xeon E5-2660 v4 processors, 4 x 16GB DDR4-2400MHz RDIMMs
• 2 x 600GB 15k SAS drives (RAID 10, used for OS), and 6 x 600GB 15k SAS drives (RAID 50, used for storage)

Performance Details: