[LEDE-DEV] [PATCH 1/3] examples (test): add example/test that shows tx-queue-ing issue

Alexandru Ardelean ardeleanalex at gmail.com
Wed Jun 7 04:09:29 PDT 2017


The context where we've seen this issue, is in
our internal tests (which are mostly shell scripts).

When we try to setup 200 interfaces with netifd,
there are about 200+ ubus entries (network.interface.* ).

When we use `ubus list` to iterate through those
entries, we've noticed the `ubus list` hangs indefinitely.
Naturally, the test times out.

After a few wrong assumptions, we've identified 4 issues:
* One addressed here 97ac89f
* One addressed here c09e4f0
* One to be addressed: tx_queue backlog getting filled up
  and messages getting silently discarded
* One to be addressed: the `retmsg` blob buf seems to be
  shared amongst clients, and there's a rare race where
  the `retmsg` seq & peer are inconsistent for a certain client.

The overall problem seems to be a case of
"fast producer, slow consumer".

If the `ubus list` output (from this script/test) is
piped to a file, the problem doesn't show up.
But if it's displayed on the serial console (which is
a slow consumer), the issue is easily reproduce-able
with this lua script.

In our tests (shell scripts) we don't display the `ubus list`
output to the serial console ; maybe the netifd data is
big enough that the tx_queue-ing issue occurs (also due
to socket buffer size).

Signed-off-by: Alexandru Ardelean <ardeleanalex at gmail.com>
---
 lua/test_many_ubus_entries.lua | 43 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100755 lua/test_many_ubus_entries.lua

diff --git a/lua/test_many_ubus_entries.lua b/lua/test_many_ubus_entries.lua
new file mode 100755
index 0000000..2324bf7
--- /dev/null
+++ b/lua/test_many_ubus_entries.lua
@@ -0,0 +1,43 @@
+#!/usr/bin/env lua
+
+require "ubus"
+require "uloop"
+
+uloop.init()
+
+local conn = ubus.connect()
+if not conn then
+	error("Failed to connect to ubus")
+end
+
+local my_method = {
+	test_0 = {
+		hello = {
+			function(req, msg)
+				conn:reply(req, {message="foo"});
+				print("Call to function 'hello'")
+				for k, v in pairs(msg) do
+					print("key=" .. k .. " value=" .. tostring(v))
+				end
+			end, {id = ubus.INT32, msg = ubus.STRING }
+		},
+	}
+}
+
+for i=1,2000 do
+	my_method["test_" .. tostring(i)] = my_method["test_0"]
+end
+
+conn:add(my_method)
+
+uloop.timer(
+	function()
+		-- running "ubus list" to make sure ubus contexts are isolated
+		os.execute("ubus list")
+		conn:close()
+		uloop.cancel()
+	end,
+	2000
+)
+
+uloop.run()
-- 
2.7.4




More information about the Lede-dev mailing list