Introduction
When working with Oracle RAC, it’s strongly advised to use Jumbo Frames for the network that provides the interconnect between nodes.
As the nodes tend to send a lot of blocks back and forth across this private network, the larger the size of the packets, the fewer of them there are to send.
The usual block size for an Oracle database is 8192 bytes.
The standard MTU (Maximum Transmission Unit) for IP frame size is 1500 bytes.
Sending an 8k Oracle block requires assembling six chunks of data to create a frame or packet.
However, if Jumbo Frames are used (9000 bytes), the entire block fits neatly into a single frame or packet.
—–
Note: Viewing the mechanical effects of MTU in action requires a fair amount of effort to setup a SPAN or Port Mirror. Use that port to capture the traffic from the wire. This is not being done for this test.
Why this explanation? Because, as shown below, the packet size will be ~8k, even though the MTU is set to 1500. Because we cannot see the effects of MTU directly on the client or server, these effects are inferred from other data.
“Frame” and “packet” are terms that seem to be used interchangeably. However, they are context-sensitive. That is, they occupy different layers of the OSI model.
—–
On with the story…
Recently, I was working with a two Node Oracle RAC system that runs in a VMWare ESXi 6.5 environment.
It was thought that due to the optimizations being performed by VMWare in the virtual network stack, that Jumbo Frames were unnecessary.
However, that does not seem to be the case.
After some testing of throughput using both the standard 1500 byte MTU and 9000 byte Jumbo Frame MTU, the larger MTU size resulted in a 22% increase in throughput speed.
Why did that happen? Well, keep reading to find out.
The Test Lab
Though the VMWare testing was done on Oracle Linux 7.8, the following experiments are being performed on Ubuntu 20.04 LTS.
As there was no need to run Oracle, Ubuntu works just fine for these tests.
Following are the two servers created:
ubuntu-test-mule-01: tcp test client - 192.168.1.226 ubuntu-test-mule-02: tcp test server - 192.168.1.227
Some version information:
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '^(NAME=|VERSION=)' /etc/os-release NAME="Ubuntu" VERSION="20.04 LTS (Focal Fossa)"
Network Configuration
Other than having different IP addresses, ubuntu-test-mule-01 and ubuntu-test-mule-02 are set up exactly the same way.
Because this version of Ubuntu uses netplan to configure the interfaces, we modified the /etc/netplan/00-installer-config.yaml file to configure the two test interfaces.
The interfaces used for the testing are enp0s8 and enp0s9.
Then, netplan apply was used to enable the changes.
root@ubuntu-mule-01:~/perl-sockets/packet-test# cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp0s3:
dhcp4: true
enp0s8:
dhcp4: false
addresses: [192.168.154.4/24]
mtu: 9000
enp0s9:
dhcp4: false
addresses: [192.168.199.35/24]
version: 2
The results:
# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 08:00:27:b8:5c:dc brd ff:ff:ff:ff:ff:ff inet 192.168.1.227/24 brd 192.168.1.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:feb8:5cdc/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000 link/ether 08:00:27:ab:d5:44 brd ff:ff:ff:ff:ff:ff inet 192.168.154.5/24 brd 192.168.154.255 scope global enp0s8 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:feab:d544/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 08:00:27:22:fc:a2 brd ff:ff:ff:ff:ff:ff inet 192.168.199.36/24 brd 192.168.199.255 scope global enp0s9 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fe22:fca2/64 scope link valid_lft forever preferred_lft forever
Configure the TCP Test
Some time ago, I put together some Perl scripts for network throughput testing.
There were several reasons for this, including:
- You can copy and paste the code if necessary.
- It is easy to modify for different tests.
The following was run on each of the test mule servers:
# git clone https://github.com/jkstill/perl-sockets.git
On the ubuntu-mule-02 Server:
The following changes were made to server.pl:
First, disable the dot reporting. By default a “.” is printed every 256 packets that are received. You can disable this using the following line:
my $displayReport = 0; # set to 0 to disable reporting
Now, set the listening addresses.
Default code:
my $sock = IO::Socket::INET->new( #LocalAddr => '192.168.1.255', # uncomment and edit adddress if needed LocalPort => $port, Proto => $proto, Listen => 1, Reuse => 1 ) or die "Cannot create socket: $@";
We changed this to reflect the network interfaces that would be used for the testing on the server-side.
my $sock = IO::Socket::INET->new( LocalAddr => '192.168.154.5', # MTU 9000 #LocalAddr => '192.168.199.36', # MTU 1500 LocalPort => $port, Proto => $proto, Listen => 1, Reuse => 1 ) or die "Cannot create socket: $@";
The appropriate interface was used for each test.
On the ubuntu-mule-01 Client:
Some test data was created. The use of /dev/urandom and gzip makes it unlikely that you can perform any compression. This is something I learned to do for quick throughput tests using ssh, as ssh compresses data. It’s probably not necessary in this case, but then again, it doesn’t hurt to ensure the test data is non-compressible.
root # cd perl-sockets/packet-test root # dd if=/dev/urandom bs=1048576 count=101 | gzip - | dd iflag=fullblock bs=1048576 count=100 of=testdata-100M.dat root # dd if=/dev/urandom bs=1048576 count=1025 | gzip - | dd iflag=fullblock bs=1048576 count=1024 of=testdata-1G.dat root@ubuntu-mule-01:~/perl-sockets/packet-test# ls -l testdata-1* -rw-r--r-- 1 root root 104857600 May 11 16:05 testdata-100M.dat -rw-r--r-- 1 root root 1073741824 May 11 16:04 testdata-1G.dat
The Driver Script
We used the packet-driver.sh script to run each of the tests from the client-side.
This script simply runs a throughput test 23 times in succession, using the specified MTU size.
#!/usr/bin/env bash
: ${1:?Call with 'packet-driver.sh <SIZE> '!}
: ${mtu:=$1}
if ( echo $mtu | grep -vE '1500|9000' ); then
echo Please use 1500 or 9000
exit 1
fi
declare -A localHosts
declare -A remoteHosts
localHosts[9000]=192.168.154.4
localHosts[1500]=192.168.199.35
remoteHosts[9000]=192.168.154.5
remoteHosts[1500]=192.168.199.36
blocksize=8192
testfile=testdata-1G.dat
cmd="./client.pl --remote-host ${remoteHosts[$mtu]} --local-host ${localHosts[$mtu]} --file $testfile --buffer-size $blocksize"
for i in {0..22}
do
echo "executing: $cmd"
$cmd
done
Perform the Tests
For each test, we enabled the correct interface in server.pl, and then the started the server.
For the client-side, we called the packet-driver.sh script with the required MTU size.
The MTU size passed on the command line determines which interface is used on the client-side.
1500 MTU
On the server-side, make sure the address in server.pl is set for the 1500 byte MTU interface. Then, start the server:
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '\s+LocalAddr' server.pl LocalAddr => '192.168.199.36', # MTU 1500 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./server.pl | tee mtu-1500.log Initial Receive Buffer is 425984 bytes Server is now listening ... Initial Buffer size set to: 2048
On the client-side, run packet-driver.sh:
root@ubuntu-mule-01:~/perl-sockets/packet-test# ./packet-driver.sh 1500 executing: ./client.pl --remote-host 192.168.199.36 --local-host 192.168.199.35 --file testdata-1G.dat --buffer-size 8192 remote host: 192.168.199.36 port: 4242 bufsz: 8192 simulated latency: 0 bufsz: 8192 Send Buffer is 425984 bytes Connected to 192.168.199.36 on port 4242 Sending data...
9000 MTU
On the server-side, make sure the address in server.pl is set for the 9000 byte MTU interface. Then, start the server:
root@ubuntu-mule-02:~/perl-sockets/packet-test# grep -E '\s+LocalAddr' server.pl LocalAddr => '192.168.154.5', # MTU 9000 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./server.pl | tee mtu-9000.log Initial Receive Buffer is 425984 bytes Server is now listening ... Initial Buffer size set to: 2048
Now, run packet-driver.sh on the client:
root@ubuntu-mule-01:~/perl-sockets/packet-test# ./packet-driver.sh 9000 executing: ./client.pl --remote-host 192.168.154.5 --local-host 192.168.154.4 --file testdata-1G.dat --buffer-size 8192 remote host: 192.168.154.5 port: 4242 bufsz: 8192 simulated latency: 0 bufsz: 8192 Send Buffer is 425984 bytes Connected to 192.168.154.5 on port 4242 Sending data...
Reporting
When all tests are complete, use packet-averages.pl to calculate the averages across all tests per MTU size.
root@ubuntu-mule-02:~/perl-sockets/packet-test# ./packet-averages.pl < mtu-1500.log key/avg: Bytes Received 1073733637.000000 key/avg: Avg Packet Size 7898.147391 key/avg: Packets Received 135948.304348 key/avg: Average milliseconds 0.043824 key/avg: Avg Megabytes/Second 172.000000 key/avg: Avg milliseconds/MiB 5.818500 key/avg: Total Elapsed Seconds 6.850447 key/avg: Network Elapsed Seconds 5.958098 root@ubuntu-mule-02:~/perl-sockets/packet-test# ./packet-averages.pl < mtu-9000.log key/avg: Bytes Received 1073733637.000000 key/avg: Avg Packet Size 7519.793478 key/avg: Packets Received 142790.217391 key/avg: Average milliseconds 0.033652 key/avg: Avg Megabytes/Second 213.165217 key/avg: Avg milliseconds/MiB 4.692753 key/avg: Total Elapsed Seconds 5.495095 key/avg: Network Elapsed Seconds 4.805343
The average Total Elapsed Seconds for the 9000 MTU tests is only 80% of the time required for the 1500 MTU tests.
From these results, it appears as if using Jumbo Frames is a pretty clear winner, even in a virtualized environment.
This result might seem somewhat surprising, as the tests are not sending any data over a physical wire.
In this case, the “network” is only composed of VirtualBox host network adapters.
So, why then are Jumbo Frames still so much faster than the standard 1500 MTU size?
Performance Profiling
This time, we’ll run a single test for each MTU size.
We’ll use the perf profiler to gather process profile information on the client-side.
First, the 1500 MTU size:
perf record --output perf-mtu-1500.data ./client.pl --remote-host 192.168.199.36 --local-host 192.168.199.35 --file testdata-1G.dat --buffer-size 8192
Now the 9000 MTU size:
perf record --output perf-mtu-9000.data ./client.pl --remote-host 192.168.154.5 --local-host 192.168.154.4 --file testdata-1G.dat --buffer-size 8192
Let’s see some reports from perf.
1500 MTU
root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-1500.data --stats | grep TOTAL TOTAL events: 28648 root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-1500.data | head -20 # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 28K of event 'cpu-clock:pppH' # Event count (approx.): 7137500000 # # Overhead Command Shared Object Symbol # ........ ....... .................. .................................... # 56.35% perl [kernel.kallsyms] [k] e1000_xmit_frame 21.00% perl [kernel.kallsyms] [k] e1000_alloc_rx_buffers 9.71% perl [kernel.kallsyms] [k] e1000_clean 2.22% perl [kernel.kallsyms] [k] __softirqentry_text_start 1.74% perl [kernel.kallsyms] [k] __lock_text_start 0.61% perl [kernel.kallsyms] [k] copy_user_generic_string 0.58% perl [kernel.kallsyms] [k] clear_page_rep 0.39% perl libpthread-2.31.so [.] __libc_read 0.30% perl [kernel.kallsyms] [k] do_syscall_64
9000 MTU
root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-9000.data --stats | grep TOTAL TOTAL events: 25259 root@ubuntu-mule-01:~/perl-sockets/packet-test# root@ubuntu-mule-01:~/perl-sockets/packet-test# perf report -i perf-mtu-9000.data | head -20 # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 25K of event 'cpu-clock:pppH' # Event count (approx.): 6290500000 # # Overhead Command Shared Object Symbol # ........ ....... .................. ........................................ # 64.15% perl [kernel.kallsyms] [k] e1000_xmit_frame 16.92% perl [kernel.kallsyms] [k] e1000_alloc_jumbo_rx_buffers 9.03% perl [kernel.kallsyms] [k] e1000_clean 1.79% perl [kernel.kallsyms] [k] __softirqentry_text_start 1.66% perl [kernel.kallsyms] [k] __lock_text_start 0.41% perl [kernel.kallsyms] [k] clear_page_rep 0.39% perl [kernel.kallsyms] [k] copy_user_generic_string 0.26% perl [kernel.kallsyms] [k] do_syscall_64 0.24% perl libpthread-2.31.so [.] __libc_read
A conclusion we might draw from these reports:
When using an MTU of 9000, the test program spent more time sending data (e1000_xmit_frame) and less time in overhead.
Note that 16.92% of the time was spent in allocating Jumbo-sized frames through e1000_alloc_jumbo_rx_buffers in the MBU 9000 test, versus 21% of the time required in the MTU 1500 test for e1000_alloc_rx_buffers.
The reason for the performance increase, in this case, seems to be this: The use of Jumbo Frames simply requires less work for the server.
Rather than assembling six 1500 byte frames into a packet to accommodate our 8192-byte packet, Jumbo Frames can get it all in one frame.
Though these tests were run using servers virtualized with VirtualBox, the results are quite similar to those seen in servers running in VMWare ESXi.
The fact that the servers are virtual does not reduce the need to ensure that RAC nodes get the fastest possible throughput on the private network used for the interconnect… And that means using Jumbo Frames.