Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to High Performance
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts; No: guests.
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented:
• Yes: hosts; No: guests.
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented:
• Yes: hosts; No: guests.
• The test sponsor attests, as of date of publication, that CVE-2018-3620 and CVE-2018-3646 (Foreshadow) are mitigated in the system as tested and documented:
• Yes: hosts; No: guests.

Storage Notes

• The System Under Test has 2 480GB SATA SSDs configured as one RAID1 array in BIOS by SAS3408iMR raid controller for the hypervisor OS that was booted.
• The System Under Test has 4 3.2TB NVMe SSDs configured as 4 RAID10 arrays for vm images and datastores.
• An NVMe SSD disk is divided into four partitions, which are combined with four partitions from another NVMe SSD to create two RAID 10 arrays.

Virtualization Software Notes

• The following boot options were used:
• iommu.passthrough=1 default_hugepagesz=1G hugepagesz=1G hugepages=1400
• The following services were disabled:
• iptables, ip6tables, irpbalance, firewalld
• Selinux was set to disabled.
• Swap partition was disabled.
• Tuned service configuration:
• tuned-adm profile throughput-performance
• Network configuration is as follows:
• The Ethernet devices enp133s0, enp137s0, enp139s0 and enp143s0 served as physical functions used by the host to provide physical transport for the virtual functions.
• 35 virtual functions were configured for each device.
• Web, Infra, App, Mail, Batch and DB VMs were allocated 1 VF each.
• The interrupts for the virtual functions allocated by the network device enp133s0 were bound to CPU 0-1.
• The interrupts for the virtual functions allocated by the network device enp137s0 were bound to CPU 32-33.
• The interrupts for the virtual functions allocated by the network device enp139s0 were bound to CPU 64-65.
• The interrupts for the virtual functions allocated by the network device enp143s0 were bound to CPU 96-97.
• Each of the four NUMA nodes is allocated 350 huge pages, with each page sized at 1 GB.
• VM configurations are as follows:
• All VMs are managed with libvirt and virsh utility
• All VMs are created with the following XML template:
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host_cpu_list'>$vm_cpus</vcpu>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='aarch64' machine='virt-6.2'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='host-passthrough' check='none'>
• <topology sockets='1' cores='$vm_vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-system-aarch64</emulator>
• <disk type='file' device='disk'>
• <driver name='qemu' type='qcow2' cache='none' io='native'/>
• <source dev='$vm_disk'/>
• <target dev='vda' bus='virtio'/>
• </disk>
• <interface type='hostdev' managed='yes'>
• <mac address='$mac_address'/>
• <driver name='vfio'/>
• <label>'ethx'</label>
• <source>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </interface>
• </devices>
• </domain>
• The following section is added for all VMs:
• <memoryBacking>
• <hugepages>
• <page size='1048576' unit='KiB'/>
• </hugepages>
• </memoryBacking>
• The following section is repeated for the second disk used in the infra, mail and db VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk'/>
• <target dev='vdb' bus='virtio'/>
• </disk>
• VMs' nice values were set after startup via the renice command to the following settings:
• app = -20
• db = -12
• web = -8
• infra = 19
• mail = 19
• batch = 0
• VMs were bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 01, 02, 03, 04 were bound to CPU 1-23.
• All VMs of tiles 05, 06, 07, 08 were bound to CPU 33-55.
• All VMs of tiles 09, 10, 11, 12 were bound to CPU 65-87.
• All VMs of tiles 13, 14, 15, 16 were bound to CPU 97-119.
• Db VM of tile 01 was bound to CPU 24-31.
• Db VM of tile 05 was bound to CPU 56-63.
• Db VM of tile 09 was bound to CPU 88-95.
• Db VM of tile 13 was bound to CPU 120-127.

Web Server VM Notes

• The following boot options were used:
• audit=0 ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• * soft memlock unlimited
• * hard memlock unlimited
• * hard nofile 1000000
• * soft nofile 1000000
• * hard nproc 100000
• * soft nproc 100000
• /etc/sysctl.conf changes:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• Contents of httpd configuration file /etc/httpd/conf/httpd.conf:
• ServerRoot "/etc/httpd
• Listen 80
• Include conf.modules.d/*.conf
• User apache
• Group apache
• ServerAdmin root@localhost
• <Directory />
•   AllowOverride none
•   Require all denied
• </Directory>
• DocumentRoot "/var/www/html
• <Directory "/var/www">
•   AllowOverride None
•   Require all granted
• </Directory>
• <Directory "/var/www/html">
•   Options Indexes FollowSymLinks
•   AllowOverride None
•   Require all granted
• </Directory>
• <IfModule dir_module>
•   DirectoryIndex index.html
• </IfModule>
• <Files ".ht*">
•   Require all denied
• </Files>
• ErrorLog "logs/error_log
• LogLevel warn
• <IfModule log_config_module>
•   LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
•   LogFormat "%h %l %u %t "%r" %>s %b" common
•   <IfModule logio_module>
•     LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %I %O" combinedio
•   </IfModule>
•   CustomLog "logs/access_log" combined
• </IfModule>
• <IfModule alias_module>
•   ScriptAlias /cgi-bin/ "/var/www/cgi-bin/
• </IfModule>
• <Directory "/var/www/cgi-bin">
•   AllowOverride None
•   Options None
•   Require all granted
• </Directory>
• <IfModule mime_module>
•   TypesConfig /etc/mime.types
•   AddType application/x-compress .Z
•   AddType application/x-gzip .gz .tgz
•   AddType text/html .shtml
•   AddOutputFilter INCLUDES .shtml
• </IfModule>
• AddDefaultCharset UTF-8
• <IfModule mime_magic_module>
•   MIMEMagicFile conf/magic
• </IfModule>
• EnableSendfile on
• IncludeOptional conf.d/*.conf
• Timeout 120
• KeepAlive On
• MaxKeepAliveRequests 0
• KeepAliveTimeout 15
• ServerName webserver
• <IfModule prefork.c>
• StartServers 5
• MinSpareServers 5
• MaxSpareServers 20
• ServerLimit 10240
• MaxClients 10240
• MaxRequestsPerChild 10000
• </IfModule>
• <IfModule worker.c>
• StartServers 4
• MaxClients 65535
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 10000
• </IfModule>
• <IfModule event.c>
• ServerLimit 64
• StartServers 8
• MaxRequestWorkers 10240
• MinSpareThreads 75
• MaxSpareThreads 400
• ThreadLimit 200
• ThreadsPerChild 160
• MaxConnectionsPerChild 10000
• </IfModule>
• LoadModule php5_module modules/libphp5.so
• AddHandler php5-script .php
• AddType text/html .php
• DirectoryIndex index.php
• Mutex pthread
• Contents of configuration file /etc/httpd/conf.d/ssl.conf:
• Listen 443 https
• SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
• SSLSessionCache shmcb:/run/httpd/sslcache(512000)
• SSLSessionCacheTimeout 300
• SSLRandomSeed startup file:/dev/urandom 256
• SSLRandomSeed connect builtin
• SSLCryptoDevice builtin
• <VirtualHost _default_:443>
• ErrorLog logs/ssl_error_log
• TransferLog logs/ssl_access_log
• LogLevel warn
• SSLEngine on
• SSLProtocol all
• SSLCipherSuite ALL:RC4+RSA:+SSLv3
• SSLCertificateFile /etc/pki/tls/certs/localhost.crt
• SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
• <Files ~ ".(cgi|shtml|phtml|php3?)$">
•   SSLOptions +StdEnvVars
• </Files>
• <Directory "/var/www/cgi-bin">
•   SSLOptions +StdEnvVars
• </Directory>
• BrowserMatch "MSIE [2-5]" /
•   nokeepalive ssl-unclean-shutdown /
•   downgrade-1.0 force-response-1.0
• CustomLog logs/ssl_request_log /
•   "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b
• </VirtualHost>
• Web VM used NFS to access files from infra VM.
• /etc/fstab file contains:
• infraserver${index}-int:/home/webfiles/downloads /var/www/html/support/downloads nfs defaults 0 0

Application Server VM Notes

• The following boot options were used:
• audit=0 default_hugepagesz=2m hugepagesz=2m ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• root hard memlock 3072000
• root soft memlock 3072000
• root hard nofile 130172
• root soft nofile 130172
• root hard nproc 130172
• root soft nproc 130172
• /etc/sysctl.conf changes:
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.conf.all.arp_filter=1
• net.ipv4.ip_local_port_range = 10000 65000
• fs.file-max = 500000
• vm.min_free_kbytes = 65536
• net.ipv4.tcp_tw_reuse=1
• vm.nr_hugepages = 3500
• vm.swappiness = 0
• net.ipv4.ip_forward = 0
• net.ipv4.conf.default.rp_filter = 1
• net.ipv4.conf.default.accept_source_route = 0
• kernel.sysrq = 0
• kernel.core_uses_pid = 1
• net.ipv4.tcp_syncookies = 1
• net.bridge.bridge-nf-call-ip6tables = 0
• net.bridge.bridge-nf-call-iptables = 0
• net.bridge.bridge-nf-call-arptables = 0
• kernel.msgmnb = 65536
• kernel.msgmax = 65536
• kernel.shmmax = 68719476736
• kernel.shmall = 4294967296
• The following value were set to glassfish domain.xml:
• Set the max-pool-size attribute value to 512 in the jdbc-connection-pool element
• Set the steady-pool-size attribute value to 16 in the jdbc-connection-pool element
• Set the max-thread-pool-size attribute to 50 for the thread-pool element with the name http-thread-pool
• Set the max-thread-pool-size attribute to 100 for the thread-pool element with the name admin-thread-pool

Mail Server VM Notes

• The following boot options were used:
• ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• * soft memlock unlimited
• * hard memlock unlimited
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• listen = *
• dict {
• }
• !include conf.d/*.conf
• !include_try local.conf
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• passdb {
•  args = scheme=plain /etc/dovecot/passwd.dovecot
•  driver = passwd-file
• }
• service auth {
•  user = root
• }
• service imap-login {
•  inet_listener imap {
•   port = 143
•  }
•  process_min_avail = 10
•  service_count = 0
• }
• service imap {
•  process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
•  driver = passwd
• }

Database Server VM Notes

• The following boot options were used:
• audit=0 ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• * soft nproc 578528
• * hard nproc 578528
• * soft memlock 33554432
• * hard memlock 33554432
• postgres soft nproc 578528
• postgres hard nproc 578528
• /etc/sysctl.conf changes:
• kernel.sem = 500 64000 400 512
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.conf.all.arp_filter=1
• net.ipv4.ip_local_port_range = 10000 65000
• fs.file-max = 500000
• vm.min_free_kbytes = 65536
• net.ipv4.tcp_tw_reuse=1
• /var/lib/pgsql/data/postgresql.conf changes:
• logging_collector = off
• listen_addresses = '*'
• port = 5432
• max_connections = 2200
• ssl = off
• shared_buffers=7000MB
• temp_buffers = 2000
• max_prepared_transactions = 1000
• work_mem=200MB
• maintenance_work_mem = 1025MB
• vacuum_cost_delay = 50
• bgwriter_delay = 200ms
• bgwriter_lru_maxpages=100
• fsync = on
• wal_sync_method = fdatasync
• full_page_writes = off
• wal_buffers = 2500
• commit_delay = 10
• checkpoint_segments = 256
• checkpoint_timeout = 300
• checkpoint_warning = 99
• enable_seqscan = off
• random_page_cost = 1.5
• cpu_tuple_cost = 0.001
• cpu_index_tuple_cost = 0.0005
• cpu_operator_cost = 0.00025
• effective_cache_size = 40GB
• default_statistics_target = 100
• log_min_duration_statement = 1500
• log_statement = 'none'
• default_transaction_isolation = 'read committed'
• lc_messages = 'C'
• lc_monetary = 'C'
• lc_numeric = 'C'
• lc_time = 'C'
• deadlock_timeout = 1000
• Using ramdisk on VM to store database data and logs:
• Added in /etc/fstab: tmpfs /mnt/ramdisk tmpfs rw,size=10G 0 0

Infraserver VM Notes

• The following boot options were used:
• ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• * soft memlock unlimited
• * hard memlock unlimited
• /etc/sysctl.conf changes:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• /etc/httpd/conf/httpd.conf changes:
• Listen 81
• <IfModule prefork.c>
• StartServers 5
• MinSpareServers 5
• MaxSpareServers 20
• ServerLimit 10240
• MaxClients 10240
• MaxRequestsPerChild 10000
• </IfModule>
• <IfModule worker.c>
• StartServers 4
• MaxClients 65535
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 10000
• </IfModule>
• LoadModule fcgid_module modules/mod_fcgid.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/
• FcgidIPCDir /etc/httpd/fcgid
• <Directory "/var/www/fcgi-bin">
•     AllowOverride None
•     Options +ExecCGI -Includes
•     SetHandler fcgid-script
•     Order allow,deny
•     Allow from all
• </Directory>
• AddHandler fcgid-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server was enabled to serve files to the Web VM.
• /etc/exports file contains:
• /home/webfiles/downloads webserver${index}-int(rw,no_root_squash)

Batch Server VM Notes

• The following boot options were used:
• ipv6.disable=1
• The following services were disabled:
• irqbalance abrtd smartd firewalld multipathd
• Selinux was set to disabled
• Swap partition was disabled
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• ulimit -u unlimited
• ulimit -s unlimited
• /etc/security/limits.conf changes:
• * soft memlock unlimited
• * hard memlock unlimited

Client Driver Notes

• All prime client VMs run Red Hat Enterprise Linux Server 7.4.
• One virtual client was used for each tile.
• The first virtual client is the prime controller.
• Each prime client VM has 8 VCPUs, 16 GB memory, and a 40 GB disk. 1 vfio_pci and 1 virtio_blk device were used in all client VMs.

Other Notes

• An external HUAWEI CloudEngine 6865E-48S8CQ switch was used to connect the client systems to the system under test (SUT).
• The SUT used the three quad-port 25Gb PCIe NICs for communication with the VMs.
• The SUT NICs and the external client systems were connected to the switch using Fiber optic network cable.

Performance Details: