[LEDE-DEV] Suggested improvement: Filtering uci export data

[Bob Taylor]

Hello All,

I'm trying to fix an issue with a slow-loading firewall on an
OpenWRT-based platform.

Because of the structure of the firewall I am supporting, there are a
large number of rules in the firewall config, resulting in a 100kb+
file.  While most of these are disabled by default, they are still
retained since the config is where they are defined.  The problem is
that certain scripts called by the firewall need to load the file and
iterate over a few other configurations, and each config_load is
taking on the order of seconds since the file is so large, exceeding
the acceptable time limit for a reload operation.

One of the ideas was to have the config being exported to the
environment shrunk down when we call config_load by one of two
methods, both of which involve modifying the uci binary:

    1) Filter out all "disabled" rules, which would mean filtering
sections based on an option value in the section (this could then
filter out all configs with "enabled=0")
    2) Filter out all section types that are not in a list of given
types (reducing the sections to be only those being operated on)

Both of these methods would work the same as they do now by default,
but with changes to uci and uci.sh, setting an environment variable
before calling config_load could change the behavior to only load
certain sections of data, drastically reducing the load time.  I have
also looked at the code that would need to be changed, and aside from
passing the list/item for comparison it looks pretty straightforward
to do option 2 and only slightly more complicated (and a bit slower)
to implement option 1.

We would like to check with the upstream developers (Hans Dedecker
told me that would be Jo-Philipp Which and Felix Fietkau) before
beginning the work to get opinions on if this would be an appropriate
approach (and if they think such a feature would be accepted upstream
if we provided the patch, as we'd like to not have company-specific
patches if at all possible).  Or perhaps someone can see a better
approach altogether (I am currently looking at reducing the number of
scripts, but it does look like some of them are here to stay).

Thanks,

Bob Taylor