[PATCH v2 2/3] nvme-tcp: support specifying the congestion-control
Mingbao Sun
sunmingbao at tom.com
Fri Mar 25 05:11:23 PDT 2022
> 1. Can you please provide your measurements that support your claims?
Yes. I would provide a series of the testing result.
In the bottom of this mail, I would provide the first one.
>
> 2. Can you please provide a real, existing use-case where this provides
> true, measureable value? And more specifically, please clarify how the
> use-case needs a local tuning for nvme-tcp that would not hold for
> other tcp streams that are running on the host (and vice-versa).
>
As for the use-case.
I think multiple NVMe/TCP hosts simultaneously write data to a single target
is a much common use-case.
And this patchset just addresses the performance issue of this use-case.
Now let's see the measurements.
I have got 4 servers with exactly the same HW/SW combination.
Each server has a 50Gb bonding port consisted of 2 25Gb Mellanox ConnectX-4
Lx ports.
Basic SW info:
provo-dirt:~ # cat /etc/os-release
NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
......
provo-dirt:~ # uname -r
5.14.21-SP4-9c51d4e2+
Test method:
the first 3 servers TX, and the last one RX.
A small program was written to generate the traffic.
It prints the relative and absolute counters per second.
Each round last about 1 minute.
Here is the info and the role of these servers.
No. NAME IP ROLE
1 provo-dirt 169.254.85.1 TX
2 sandy-dirt 169.254.85.2 TX
3 orem-dirt 169.254.85.3 TX
4 ogden-dirt 169.254.85.4 RX
The output looks like this on the TX nodes:
[time lasts]: 1
[conn-num 8][tx on][rx off][msg-len 65536][data-type seq-u64]
[Relative Stat]
tx_total 0x7bc6 (31,686)
tx_total_bytes 0x7bc60000 (2,076,573,696)
tx_succ 0x7bbe (31,678)
tx_succ_bytes 0x7bbe0000 (2,076,049,408)
[Absolute Stat]
tx_total 0x7bc6 (31,686)
tx_total_bytes 0x7bc60000 (2,076,573,696)
tx_succ 0x7bbe (31,678)
tx_succ_bytes 0x7bbe0000 (2,076,049,408)
[time lasts]: 2
[conn-num 8][tx on][rx off][msg-len 65536][data-type seq-u64]
[Relative Stat]
tx_total 0x7bfd (31,741)
tx_total_bytes 0x7bfd0000 (2,080,178,176)
tx_succ 0x7bfd (31,741)
tx_succ_bytes 0x7bfd0000 (2,080,178,176)
[Absolute Stat]
tx_total 0xf7c3 (63,427)
tx_total_bytes 0xf7c30000 (4,156,751,872)
tx_succ 0xf7bb (63,419)
tx_succ_bytes 0xf7bb0000 (4,156,227,584)
[time lasts]: 3
[conn-num 8][tx on][rx off][msg-len 65536][data-type seq-u64]
[Relative Stat]
tx_total 0x7b8b (31,627)
tx_total_bytes 0x7b8b0000 (2,072,707,072)
tx_succ 0x7b8b (31,627)
tx_succ_bytes 0x7b8b0000 (2,072,707,072)
[Absolute Stat]
tx_total 0x1734e (95,054)
tx_total_bytes 0x1734e0000 (6,229,458,944)
tx_succ 0x17346 (95,046)
tx_succ_bytes 0x173460000 (6,228,934,656)
And the output looks like this on the RX node:
[time lasts]: 3
[conn-num 24][tx off][rx on][msg-len 65536][data-type seq-u64]
[Relative Stat]
rx_total 0x170af (94,383)
rx_total_bytes 0x170af0000 (6,185,484,288)
rx_msg 0x170af (94,383)
rx_msg_bytes 0x170af0000 (6,185,484,288)
rx_msg_succ 0x170af (94,383)
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[Absolute Stat]
rx_total 0x44150 (278,864)
rx_total_bytes 0x441500000 (18,275,631,104)
rx_msg 0x44138 (278,840)
rx_msg_bytes 0x441380000 (18,274,058,240)
rx_msg_succ 0x44138 (278,840)
rx_msg_succ_bytes 0x441380000 (18,274,058,240)
[time lasts]: 4
[conn-num 24][tx off][rx on][msg-len 65536][data-type seq-u64]
[Relative Stat]
rx_total 0x170ab (94,379)
rx_total_bytes 0x170ab0000 (6,185,222,144)
rx_msg 0x170ab (94,379)
rx_msg_bytes 0x170ab0000 (6,185,222,144)
rx_msg_succ 0x170ab (94,379)
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[Absolute Stat]
rx_total 0x5b1fb (373,243)
rx_total_bytes 0x5b1fb0000 (24,460,853,248)
rx_msg 0x5b1e3 (373,219)
rx_msg_bytes 0x5b1e30000 (24,459,280,384)
rx_msg_succ 0x5b1e3 (373,219)
rx_msg_succ_bytes 0x5b1e30000 (24,459,280,384)
[time lasts]: 5
[conn-num 24][tx off][rx on][msg-len 65536][data-type seq-u64]
[Relative Stat]
rx_total 0x170ae (94,382)
rx_total_bytes 0x170ae0000 (6,185,418,752)
rx_msg 0x170ae (94,382)
rx_msg_bytes 0x170ae0000 (6,185,418,752)
rx_msg_succ 0x170ae (94,382)
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[Absolute Stat]
rx_total 0x722a9 (467,625)
rx_total_bytes 0x722a90000 (30,646,272,000)
rx_msg 0x72291 (467,601)
rx_msg_bytes 0x722910000 (30,644,699,136)
rx_msg_succ 0x72291 (467,601)
rx_msg_succ_bytes 0x722910000 (30,644,699,136)
But for your convenience to fucos on the fluctuation of the bandwidth,
and for reducing the length of the mail,
We get all other line muted, we only show tx_succ_bytes and rx_msg_succ_bytes
in each second.
[Combination 1] dctcp + switch ECN-marking on
This is the perfect scenario.
Bandwidth of each node is perfectly stable.
No packet dropping on switch at all.
/*
* before loading traffic, clear the counters on the 2 switches.
*/
hound-dirt# clear qos statistics type queuing interface ethernet 1/1/4
hound-dirt# show queuing statistics interface ethernet 1/1/4
Interface ethernet1/1/4
Queue Packets Bytes Dropped-Packets Dropped-Bytes
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
hound-dirt#
hound-dirt# clear qos statistics type queuing interface ethernet 1/1/4
hound-dirt# show queuing statistics interface ethernet 1/1/4
Interface ethernet1/1/4
Queue Packets Bytes Dropped-Packets Dropped-Bytes
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
hound-dirt#
/*
* logs of RX node.
*/
ogden-dirt:/home/admin/tyler # cat /proc/sys/net/ipv4/tcp_congestion_control
dctcp
ogden-dirt:/home/admin/tyler # date; ./tcp_perf.exe --server-ip 169.254.85.4 --server-port 10001 --msg-len 65536 --conn-num 24 -s -r --data-random --svr-no-wait-all-conn-OK --show-bandwidth-only
Fri Mar 25 07:58:48 EDT 2022
DBG:/mnt/hgfs/src/linux-dev-framework-master/libs/app_utils/src/socket.c(104)-socket_init_2:
bind socket fd 3 to 169.254.85.4:10001 succeed
DBG:perf_frmwk.c(1472)-create_tcp_conns_start_load:
start listen on fd 3
conn [0] local 169.254.85.4:10001 peer 169.254.85.3:59660 created.
rx thread of conn 0 started
conn [1] local 169.254.85.4:10001 peer 169.254.85.3:59662 created.
rx thread of conn 1 started
conn [2] local 169.254.85.4:10001 peer 169.254.85.3:59664 created.
rx thread of conn 2 started
conn [3] local 169.254.85.4:10001 peer 169.254.85.3:59666 created.
rx thread of conn 3 started
conn [4] local 169.254.85.4:10001 peer 169.254.85.3:59668 created.
conn [5] local 169.254.85.4:10001 peer 169.254.85.3:59670 created.
rx thread of conn 4 started
conn [6] local 169.254.85.4:10001 peer 169.254.85.3:59672 created.
rx thread of conn 5 started
conn [7] local 169.254.85.4:10001 peer 169.254.85.3:59674 created.
rx thread of conn 6 started
rx thread of conn 7 started
conn [8] local 169.254.85.4:10001 peer 169.254.85.2:37956 created.
rx thread of conn 8 started
conn [9] local 169.254.85.4:10001 peer 169.254.85.2:37958 created.
rx thread of conn 9 started
conn [10] local 169.254.85.4:10001 peer 169.254.85.2:37960 created.
rx thread of conn 10 started
conn [11] local 169.254.85.4:10001 peer 169.254.85.2:37962 created.
rx thread of conn 11 started
conn [12] local 169.254.85.4:10001 peer 169.254.85.2:37964 created.
rx thread of conn 12 started
conn [13] local 169.254.85.4:10001 peer 169.254.85.2:37966 created.
rx thread of conn 13 started
conn [14] local 169.254.85.4:10001 peer 169.254.85.2:37968 created.
rx thread of conn 14 started
conn [15] local 169.254.85.4:10001 peer 169.254.85.2:37970 created.
rx thread of conn 15 started
[time lasts]: 1
rx_msg_succ_bytes 0x161500000 (5,927,600,128)
conn [16] local 169.254.85.4:10001 peer 169.254.85.1:49468 created.
rx thread of conn 16 started
conn [17] local 169.254.85.4:10001 peer 169.254.85.1:49470 created.
rx thread of conn 17 started
conn [18] local 169.254.85.4:10001 peer 169.254.85.1:49472 created.
rx thread of conn 18 started
conn [19] local 169.254.85.4:10001 peer 169.254.85.1:49474 created.
rx thread of conn 19 started
conn [20] local 169.254.85.4:10001 peer 169.254.85.1:49476 created.
rx thread of conn 20 started
conn [21] local 169.254.85.4:10001 peer 169.254.85.1:49478 created.
rx thread of conn 21 started
conn [22] local 169.254.85.4:10001 peer 169.254.85.1:49480 created.
rx thread of conn 22 started
conn [23] local 169.254.85.4:10001 peer 169.254.85.1:49482 created.
24 connection(s) created in total
rx thread of conn 23 started
[time lasts]: 2
rx_msg_succ_bytes 0x1709a0000 (6,184,108,032)
[time lasts]: 3
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 4
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 5
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 6
rx_msg_succ_bytes 0x170b00000 (6,185,549,824)
[time lasts]: 7
rx_msg_succ_bytes 0x170590000 (6,179,848,192)
[time lasts]: 8
rx_msg_succ_bytes 0x170a90000 (6,185,091,072)
[time lasts]: 9
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[time lasts]: 10
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[time lasts]: 11
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 12
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 13
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 14
rx_msg_succ_bytes 0x170b20000 (6,185,680,896)
[time lasts]: 15
rx_msg_succ_bytes 0x170a70000 (6,184,960,000)
[time lasts]: 16
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 17
rx_msg_succ_bytes 0x170b20000 (6,185,680,896)
[time lasts]: 18
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 19
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 20
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 21
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 22
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 23
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 24
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 25
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 26
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 27
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 28
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 29
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 30
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 31
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 32
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 33
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 34
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 35
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 36
rx_msg_succ_bytes 0x170b10000 (6,185,615,360)
[time lasts]: 37
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 38
rx_msg_succ_bytes 0x170a90000 (6,185,091,072)
[time lasts]: 39
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 40
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 41
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 42
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 43
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 44
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[time lasts]: 45
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 46
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 47
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 48
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[time lasts]: 49
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 50
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 51
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 52
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
[time lasts]: 53
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 54
rx_msg_succ_bytes 0x170ac0000 (6,185,287,680)
[time lasts]: 55
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 56
rx_msg_succ_bytes 0x170ae0000 (6,185,418,752)
[time lasts]: 57
rx_msg_succ_bytes 0x170a90000 (6,185,091,072)
[time lasts]: 58
rx_msg_succ_bytes 0x170af0000 (6,185,484,288)
[time lasts]: 59
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 60
rx_msg_succ_bytes 0x170a50000 (6,184,828,928)
[time lasts]: 61
rx_msg_succ_bytes 0x170aa0000 (6,185,156,608)
[time lasts]: 62
rx_msg_succ_bytes 0x170ad0000 (6,185,353,216)
[time lasts]: 63
rx_msg_succ_bytes 0x170ab0000 (6,185,222,144)
^Ccaught signal 2
/*
* logs of TX node 1.
*/
provo-dirt:/home/admin/tyler # cat /proc/sys/net/ipv4/tcp_congestion_control
dctcp
provo-dirt:/home/admin/tyler # date; ./tcp_perf.exe --server-ip 169.254.85.4 --server-port 10001 --msg-len 65536 --conn-num 8 -c -t --data-random --show-bandwidth-only
Fri Mar 25 07:58:53 EDT 2022
conn [0] local 169.254.85.1:49468 peer 169.254.85.4:10001 created.
conn [1] local 169.254.85.1:49470 peer 169.254.85.4:10001 created.
conn [2] local 169.254.85.1:49472 peer 169.254.85.4:10001 created.
conn [3] local 169.254.85.1:49474 peer 169.254.85.4:10001 created.
conn [4] local 169.254.85.1:49476 peer 169.254.85.4:10001 created.
conn [5] local 169.254.85.1:49478 peer 169.254.85.4:10001 created.
conn [6] local 169.254.85.1:49480 peer 169.254.85.4:10001 created.
conn [7] local 169.254.85.1:49482 peer 169.254.85.4:10001 created.
8 connection(s) created in total
tx thread of conn 0 started
tx thread of conn 1 started
tx thread of conn 2 started
tx thread of conn 3 started
tx thread of conn 4 started
tx thread of conn 5 started
tx thread of conn 6 started
tx thread of conn 7 started
[time lasts]: 1
tx_succ_bytes 0x765c0000 (1,985,740,800)
[time lasts]: 2
tx_succ_bytes 0x78f30000 (2,029,191,168)
[time lasts]: 3
tx_succ_bytes 0x783d0000 (2,017,263,616)
[time lasts]: 4
tx_succ_bytes 0x78c90000 (2,026,438,656)
[time lasts]: 5
tx_succ_bytes 0x76bf0000 (1,992,228,864)
[time lasts]: 6
tx_succ_bytes 0x784f0000 (2,018,443,264)
[time lasts]: 7
tx_succ_bytes 0x765c0000 (1,985,740,800)
[time lasts]: 8
tx_succ_bytes 0x78140000 (2,014,576,640)
[time lasts]: 9
tx_succ_bytes 0x77bf0000 (2,009,006,080)
[time lasts]: 10
tx_succ_bytes 0x79210000 (2,032,205,824)
[time lasts]: 11
tx_succ_bytes 0x786d0000 (2,020,409,344)
[time lasts]: 12
tx_succ_bytes 0x79480000 (2,034,761,728)
[time lasts]: 13
tx_succ_bytes 0x76610000 (1,986,068,480)
[time lasts]: 14
tx_succ_bytes 0x76940000 (1,989,410,816)
[time lasts]: 15
tx_succ_bytes 0x76ef0000 (1,995,374,592)
[time lasts]: 16
tx_succ_bytes 0x76f20000 (1,995,571,200)
[time lasts]: 17
tx_succ_bytes 0x78190000 (2,014,904,320)
[time lasts]: 18
tx_succ_bytes 0x775c0000 (2,002,518,016)
[time lasts]: 19
tx_succ_bytes 0x77040000 (1,996,750,848)
[time lasts]: 20
tx_succ_bytes 0x77d90000 (2,010,710,016)
[time lasts]: 21
tx_succ_bytes 0x79470000 (2,034,696,192)
[time lasts]: 22
tx_succ_bytes 0x79730000 (2,037,579,776)
[time lasts]: 23
tx_succ_bytes 0x760c0000 (1,980,497,920)
[time lasts]: 24
tx_succ_bytes 0x779e0000 (2,006,843,392)
[time lasts]: 25
tx_succ_bytes 0x78100000 (2,014,314,496)
[time lasts]: 26
tx_succ_bytes 0x78bb0000 (2,025,521,152)
[time lasts]: 27
tx_succ_bytes 0x77df0000 (2,011,103,232)
[time lasts]: 28
tx_succ_bytes 0x75840000 (1,971,585,024)
[time lasts]: 29
tx_succ_bytes 0x772d0000 (1,999,437,824)
[time lasts]: 30
tx_succ_bytes 0x785d0000 (2,019,360,768)
[time lasts]: 31
tx_succ_bytes 0x766f0000 (1,986,985,984)
[time lasts]: 32
tx_succ_bytes 0x77510000 (2,001,797,120)
[time lasts]: 33
tx_succ_bytes 0x78720000 (2,020,737,024)
[time lasts]: 34
tx_succ_bytes 0x790e0000 (2,030,960,640)
[time lasts]: 35
tx_succ_bytes 0x797b0000 (2,038,104,064)
[time lasts]: 36
tx_succ_bytes 0x78710000 (2,020,671,488)
[time lasts]: 37
tx_succ_bytes 0x79bb0000 (2,042,298,368)
[time lasts]: 38
tx_succ_bytes 0x76f80000 (1,995,964,416)
[time lasts]: 39
tx_succ_bytes 0x79780000 (2,037,907,456)
[time lasts]: 40
tx_succ_bytes 0x79270000 (2,032,599,040)
[time lasts]: 41
tx_succ_bytes 0x78660000 (2,019,950,592)
[time lasts]: 42
tx_succ_bytes 0x77970000 (2,006,384,640)
[time lasts]: 43
tx_succ_bytes 0x78270000 (2,015,821,824)
[time lasts]: 44
tx_succ_bytes 0x774d0000 (2,001,534,976)
[time lasts]: 45
tx_succ_bytes 0x77a80000 (2,007,498,752)
[time lasts]: 46
tx_succ_bytes 0x78790000 (2,021,195,776)
[time lasts]: 47
tx_succ_bytes 0x7a730000 (2,054,356,992)
[time lasts]: 48
tx_succ_bytes 0x78c80000 (2,026,373,120)
[time lasts]: 49
tx_succ_bytes 0x78eb0000 (2,028,666,880)
[time lasts]: 50
tx_succ_bytes 0x76fd0000 (1,996,292,096)
[time lasts]: 51
tx_succ_bytes 0x77e30000 (2,011,365,376)
[time lasts]: 52
tx_succ_bytes 0x78de0000 (2,027,814,912)
[time lasts]: 53
tx_succ_bytes 0x779f0000 (2,006,908,928)
[time lasts]: 54
tx_succ_bytes 0x78310000 (2,016,477,184)
[time lasts]: 55
tx_succ_bytes 0x77b20000 (2,008,154,112)
[time lasts]: 56
tx_succ_bytes 0x76760000 (1,987,444,736)
[time lasts]: 57
tx_succ_bytes 0x78060000 (2,013,659,136)
[time lasts]: 58
tx_succ_bytes 0x78a30000 (2,023,948,288)
[time lasts]: 59
tx_succ_bytes 0x79c20000 (2,042,757,120)
[time lasts]: 60
tx_succ_bytes 0x76ff0000 (1,996,423,168)
[time lasts]: 61
tx_succ_bytes 0x78d30000 (2,027,094,016)
[time lasts]: 62
tx_succ_bytes 0x77cd0000 (2,009,923,584)
/*
* logs of TX node 2.
*/
sandy-dirt:/home/admin/tyler # cat /proc/sys/net/ipv4/tcp_congestion_control
dctcp
sandy-dirt:/home/admin/tyler # date; ./tcp_perf.exe --server-ip 169.254.85.4 --server-port 10001 --msg-len 65536 --conn-num 8 -c -t --data-random --show-bandwidth-only
Fri Mar 25 07:58:52 EDT 2022
conn [0] local 169.254.85.2:37956 peer 169.254.85.4:10001 created.
conn [1] local 169.254.85.2:37958 peer 169.254.85.4:10001 created.
conn [2] local 169.254.85.2:37960 peer 169.254.85.4:10001 created.
conn [3] local 169.254.85.2:37962 peer 169.254.85.4:10001 created.
conn [4] local 169.254.85.2:37964 peer 169.254.85.4:10001 created.
conn [5] local 169.254.85.2:37966 peer 169.254.85.4:10001 created.
conn [6] local 169.254.85.2:37968 peer 169.254.85.4:10001 created.
conn [7] local 169.254.85.2:37970 peer 169.254.85.4:10001 created.
8 connection(s) created in total
tx thread of conn 0 started
tx thread of conn 1 started
tx thread of conn 2 started
tx thread of conn 3 started
tx thread of conn 4 started
tx thread of conn 5 started
tx thread of conn 6 started
tx thread of conn 7 started
[time lasts]: 1
tx_succ_bytes 0xb7980000 (3,080,192,000)
[time lasts]: 2
tx_succ_bytes 0x83350000 (2,201,288,704)
[time lasts]: 3
tx_succ_bytes 0x80fd0000 (2,164,064,256)
[time lasts]: 4
tx_succ_bytes 0x80790000 (2,155,413,504)
[time lasts]: 5
tx_succ_bytes 0x82a00000 (2,191,523,840)
[time lasts]: 6
tx_succ_bytes 0x83350000 (2,201,288,704)
[time lasts]: 7
tx_succ_bytes 0x82350000 (2,184,511,488)
[time lasts]: 8
tx_succ_bytes 0x84260000 (2,217,082,880)
[time lasts]: 9
tx_succ_bytes 0x835c0000 (2,203,844,608)
[time lasts]: 10
tx_succ_bytes 0x81880000 (2,173,173,760)
[time lasts]: 11
tx_succ_bytes 0x80a70000 (2,158,428,160)
[time lasts]: 12
tx_succ_bytes 0x80db0000 (2,161,836,032)
[time lasts]: 13
tx_succ_bytes 0x7faa0000 (2,141,847,552)
[time lasts]: 14
tx_succ_bytes 0x82b10000 (2,192,637,952)
[time lasts]: 15
tx_succ_bytes 0x837e0000 (2,206,072,832)
[time lasts]: 16
tx_succ_bytes 0x82a60000 (2,191,917,056)
[time lasts]: 17
tx_succ_bytes 0x83f60000 (2,213,937,152)
[time lasts]: 18
tx_succ_bytes 0x82ac0000 (2,192,310,272)
[time lasts]: 19
tx_succ_bytes 0x83e90000 (2,213,085,184)
[time lasts]: 20
tx_succ_bytes 0x82420000 (2,185,363,456)
[time lasts]: 21
tx_succ_bytes 0x83ce0000 (2,211,315,712)
[time lasts]: 22
tx_succ_bytes 0x82480000 (2,185,756,672)
[time lasts]: 23
tx_succ_bytes 0x82470000 (2,185,691,136)
[time lasts]: 24
tx_succ_bytes 0x83450000 (2,202,337,280)
[time lasts]: 25
tx_succ_bytes 0x80820000 (2,156,003,328)
[time lasts]: 26
tx_succ_bytes 0x82d80000 (2,195,193,856)
[time lasts]: 27
tx_succ_bytes 0x81930000 (2,173,894,656)
[time lasts]: 28
tx_succ_bytes 0x816a0000 (2,171,207,680)
[time lasts]: 29
tx_succ_bytes 0x83cf0000 (2,211,381,248)
[time lasts]: 30
tx_succ_bytes 0x82330000 (2,184,380,416)
[time lasts]: 31
tx_succ_bytes 0x828d0000 (2,190,278,656)
[time lasts]: 32
tx_succ_bytes 0x82a50000 (2,191,851,520)
[time lasts]: 33
tx_succ_bytes 0x81600000 (2,170,552,320)
[time lasts]: 34
tx_succ_bytes 0x82060000 (2,181,431,296)
[time lasts]: 35
tx_succ_bytes 0x814b0000 (2,169,176,064)
[time lasts]: 36
tx_succ_bytes 0x81370000 (2,167,865,344)
[time lasts]: 37
tx_succ_bytes 0x811c0000 (2,166,095,872)
[time lasts]: 38
tx_succ_bytes 0x83010000 (2,197,880,832)
[time lasts]: 39
tx_succ_bytes 0x83c50000 (2,210,725,888)
[time lasts]: 40
tx_succ_bytes 0x82f70000 (2,197,225,472)
[time lasts]: 41
tx_succ_bytes 0x83730000 (2,205,351,936)
[time lasts]: 42
tx_succ_bytes 0x816a0000 (2,171,207,680)
[time lasts]: 43
tx_succ_bytes 0x83360000 (2,201,354,240)
[time lasts]: 44
tx_succ_bytes 0x81170000 (2,165,768,192)
[time lasts]: 45
tx_succ_bytes 0x83930000 (2,207,449,088)
[time lasts]: 46
tx_succ_bytes 0x84430000 (2,218,983,424)
[time lasts]: 47
tx_succ_bytes 0x81600000 (2,170,552,320)
[time lasts]: 48
tx_succ_bytes 0x812a0000 (2,167,013,376)
[time lasts]: 49
tx_succ_bytes 0x80860000 (2,156,265,472)
[time lasts]: 50
tx_succ_bytes 0x81b70000 (2,176,253,952)
[time lasts]: 51
tx_succ_bytes 0x83610000 (2,204,172,288)
[time lasts]: 52
tx_succ_bytes 0x82dc0000 (2,195,456,000)
[time lasts]: 53
tx_succ_bytes 0x83760000 (2,205,548,544)
[time lasts]: 54
tx_succ_bytes 0x83360000 (2,201,354,240)
[time lasts]: 55
tx_succ_bytes 0x82570000 (2,186,739,712)
[time lasts]: 56
tx_succ_bytes 0x82370000 (2,184,642,560)
[time lasts]: 57
tx_succ_bytes 0x83430000 (2,202,206,208)
[time lasts]: 58
tx_succ_bytes 0x81e40000 (2,179,203,072)
[time lasts]: 59
tx_succ_bytes 0x81d30000 (2,178,088,960)
[time lasts]: 60
tx_succ_bytes 0x80b50000 (2,159,345,664)
[time lasts]: 61
tx_succ_bytes 0x838c0000 (2,206,990,336)
[time lasts]: 62
tx_succ_bytes 0x82580000 (2,186,805,248)
[time lasts]: 63
tx_succ_bytes 0x81eb0000 (2,179,661,824)
/*
* logs of TX node 3.
*/
orem-dirt:/home/admin/tyler # cat /proc/sys/net/ipv4/tcp_congestion_control
dctcp
orem-dirt:/home/admin/tyler # date; ./tcp_perf.exe --server-ip 169.254.85.4 --server-port 10001 --msg-len 65536 --conn-num 8 -c -t --data-random --show-bandwidth-only
Fri Mar 25 07:58:51 EDT 2022
conn [0] local 169.254.85.3:59660 peer 169.254.85.4:10001 created.
conn [1] local 169.254.85.3:59662 peer 169.254.85.4:10001 created.
conn [2] local 169.254.85.3:59664 peer 169.254.85.4:10001 created.
conn [3] local 169.254.85.3:59666 peer 169.254.85.4:10001 created.
conn [4] local 169.254.85.3:59668 peer 169.254.85.4:10001 created.
conn [5] local 169.254.85.3:59670 peer 169.254.85.4:10001 created.
conn [6] local 169.254.85.3:59672 peer 169.254.85.4:10001 created.
conn [7] local 169.254.85.3:59674 peer 169.254.85.4:10001 created.
8 connection(s) created in total
tx thread of conn 0 started
tx thread of conn 1 started
tx thread of conn 2 started
tx thread of conn 3 started
tx thread of conn 4 started
tx thread of conn 5 started
tx thread of conn 6 started
tx thread of conn 7 started
[time lasts]: 1
tx_succ_bytes 0x125530000 (4,921,163,776)
[time lasts]: 2
tx_succ_bytes 0xa22d0000 (2,720,858,112)
[time lasts]: 3
tx_succ_bytes 0x77530000 (2,001,928,192)
[time lasts]: 4
tx_succ_bytes 0x76b90000 (1,991,835,648)
[time lasts]: 5
tx_succ_bytes 0x78070000 (2,013,724,672)
[time lasts]: 6
tx_succ_bytes 0x74cd0000 (1,959,591,936)
[time lasts]: 7
tx_succ_bytes 0x76a80000 (1,990,721,536)
[time lasts]: 8
tx_succ_bytes 0x75dc0000 (1,977,352,192)
[time lasts]: 9
tx_succ_bytes 0x756b0000 (1,969,946,624)
[time lasts]: 10
tx_succ_bytes 0x76620000 (1,986,134,016)
[time lasts]: 11
tx_succ_bytes 0x772f0000 (1,999,568,896)
[time lasts]: 12
tx_succ_bytes 0x77090000 (1,997,078,528)
[time lasts]: 13
tx_succ_bytes 0x77220000 (1,998,716,928)
[time lasts]: 14
tx_succ_bytes 0x77ce0000 (2,009,989,120)
[time lasts]: 15
tx_succ_bytes 0x781d0000 (2,015,166,464)
[time lasts]: 16
tx_succ_bytes 0x762b0000 (1,982,529,536)
[time lasts]: 17
tx_succ_bytes 0x765c0000 (1,985,740,800)
[time lasts]: 18
tx_succ_bytes 0x77010000 (1,996,554,240)
[time lasts]: 19
tx_succ_bytes 0x74ad0000 (1,957,494,784)
[time lasts]: 20
tx_succ_bytes 0x76820000 (1,988,231,168)
[time lasts]: 21
tx_succ_bytes 0x75b70000 (1,974,927,360)
[time lasts]: 22
tx_succ_bytes 0x754f0000 (1,968,111,616)
[time lasts]: 23
tx_succ_bytes 0x75e30000 (1,977,810,944)
[time lasts]: 24
tx_succ_bytes 0x75730000 (1,970,470,912)
[time lasts]: 25
tx_succ_bytes 0x77380000 (2,000,158,720)
[time lasts]: 26
tx_succ_bytes 0x77900000 (2,005,925,888)
[time lasts]: 27
tx_succ_bytes 0x76c00000 (1,992,294,400)
[time lasts]: 28
tx_succ_bytes 0x75e10000 (1,977,679,872)
[time lasts]: 29
tx_succ_bytes 0x784c0000 (2,018,246,656)
[time lasts]: 30
tx_succ_bytes 0x768c0000 (1,988,886,528)
[time lasts]: 31
tx_succ_bytes 0x76c70000 (1,992,753,152)
[time lasts]: 32
tx_succ_bytes 0x76c80000 (1,992,818,688)
[time lasts]: 33
tx_succ_bytes 0x77c10000 (2,009,137,152)
[time lasts]: 34
tx_succ_bytes 0x78010000 (2,013,331,456)
[time lasts]: 35
tx_succ_bytes 0x74ef0000 (1,961,820,160)
[time lasts]: 36
tx_succ_bytes 0x76fe0000 (1,996,357,632)
[time lasts]: 37
tx_succ_bytes 0x76600000 (1,986,002,944)
[time lasts]: 38
tx_succ_bytes 0x76360000 (1,983,250,432)
[time lasts]: 39
tx_succ_bytes 0x73e40000 (1,944,322,048)
[time lasts]: 40
tx_succ_bytes 0x76120000 (1,980,891,136)
[time lasts]: 41
tx_succ_bytes 0x735b0000 (1,935,343,616)
[time lasts]: 42
tx_succ_bytes 0x74e30000 (1,961,033,728)
[time lasts]: 43
tx_succ_bytes 0x772f0000 (1,999,568,896)
[time lasts]: 44
tx_succ_bytes 0x75d30000 (1,976,762,368)
[time lasts]: 45
tx_succ_bytes 0x76a00000 (1,990,197,248)
[time lasts]: 46
tx_succ_bytes 0x76400000 (1,983,905,792)
[time lasts]: 47
tx_succ_bytes 0x75820000 (1,971,453,952)
[time lasts]: 48
tx_succ_bytes 0x768f0000 (1,989,083,136)
[time lasts]: 49
tx_succ_bytes 0x752c0000 (1,965,817,856)
[time lasts]: 50
tx_succ_bytes 0x76050000 (1,980,039,168)
[time lasts]: 51
tx_succ_bytes 0x77280000 (1,999,110,144)
[time lasts]: 52
tx_succ_bytes 0x757b0000 (1,970,995,200)
[time lasts]: 53
tx_succ_bytes 0x76220000 (1,981,939,712)
[time lasts]: 54
tx_succ_bytes 0x74de0000 (1,960,706,048)
[time lasts]: 55
tx_succ_bytes 0x75040000 (1,963,196,416)
[time lasts]: 56
tx_succ_bytes 0x76c10000 (1,992,359,936)
[time lasts]: 57
tx_succ_bytes 0x779d0000 (2,006,777,856)
[time lasts]: 58
tx_succ_bytes 0x766d0000 (1,986,854,912)
[time lasts]: 59
tx_succ_bytes 0x76a40000 (1,990,459,392)
[time lasts]: 60
tx_succ_bytes 0x760e0000 (1,980,628,992)
[time lasts]: 61
tx_succ_bytes 0x76910000 (1,989,214,208)
[time lasts]: 62
tx_succ_bytes 0x75270000 (1,965,490,176)
[time lasts]: 63
tx_succ_bytes 0x769d0000 (1,990,000,640)
[time lasts]: 64
tx_succ_bytes 0x77470000 (2,001,141,760)
/*
* counters on the switch.
* no packet dropping at all.
*/
hound-dirt# show queuing statistics interface ethernet 1/1/4
Interface ethernet1/1/4
Queue Packets Bytes Dropped-Packets Dropped-Bytes
0 0 0 0 0
1 14 1372 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 22209070 200252338712 0 0
6 0 0 0 0
7 0 0 0 0
hound-dirt#
hound-dirt# show queuing statistics interface ethernet 1/1/4
Interface ethernet1/1/4
Queue Packets Bytes Dropped-Packets Dropped-Bytes
0 0 0 0 0
1 14 1372 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 22209070 200252338712 0 0
6 0 0 0 0
7 0 0 0 0
hound-dirt#
More information about the Linux-nvme
mailing list