[LEDE-DEV] jshn.sh: json_load replaces "-" with "_"

[Matthias Schiffer]

On 10/30/2017 09:58 PM, Paul Spooren wrote:
> On 10/30/17 3:28 AM, Matthias Schiffer wrote:
>> On 10/29/2017 09:21 PM, Paul Spooren wrote:
>>> Hi all,
>>>
>>> I'm currently trying to use jshn.sh but it changes the data and makes it
>>> unusable.
>>>
>>> Below an example:
>>>
>>> root at LEDE:~# json_string="{ \"foo-bar\": 10 }"
>>> root at LEDE:~# echo $json_string
>>> { "foo-bar": 10 }
>>> root at LEDE:~# json_init
>>> root at LEDE:~# json_load "$json_string"
>>> root at LEDE:~# json_dump
>>> { "foo_bar": 10 }
>>>
>>> Expected would be the same output as input. Is this a bug or a feature?
>>>
>>> Context:
>>>
>>> Running snapshot build. I'm trying to use rpcd-mod-packagelist [1] but
>>> as e.g. "kmod-r8169" becomes "kmod_r8169" it's useless.
>>>
>>> Best,
>>>
>>> Paul Spooren
>>>
>>> [1]:
>>> https://github.com/openwrt/packages/tree/master/utils/rpcd-mod-packagelist
>> This is definitely a bug in jshn.sh (or possibly a "known limitation" to
>> keep the code simple, as shell variable names can't contain dashes). It
>> should be possible to fix this in jshn with some clever escaping of keys -
>> at least for such "simple" special characters (without looking into it in
>> detail, I assume other unusual characters that aren't allowed in shell
>> variable names are also an issue; supporting the full range of unicode
>> might need more work, or entail a prohibitive performance penalty.)
>>
>> It might actually be much easier to fix the concrete issue by rewriting
>> rpcd-mod-packagelist in C (as I suggested in [2]).
>>
>> Matthias
>>
>>
>> [2] https://github.com/openwrt/packages/pull/5037
>>
>>
> As requested [1] here the detailed use case:
> 
> TL;DR: packages are parsed via rpcd-mod-packagelist, additional info is
> attached and send as a json post to an "image on demand" server.
> 
> I started to create an build server [2] that builds firmware images
> based on delivered parameters. A possible use case is an simple "online
> imagebuilder" [3] or as a simple way to upgrade you existing system to a
> new release without the need to reinstall any package.
> 
> ubus and rpcd is needed for the luci frontend
> luci-app-attendedsysupgrade [4]. Recently I created a CLI version with
> the same functionality [5]. To send a valid request I need a packagelist
> from rpcd-mod-packagelist and some information from $(ubus call system
> board). To combine both to a json request I tried the following:
> 
> json_init
> json_load "$(ubus call packagelist list)"
> json_add_string "target" $target
> json_add_string "subtarget" $subtarget
> 
> This results in a valid structure but all packagenames are renamed
> (rpcd-mod-packagelist becomes rpcd_mod_packagelist). The error happens
> as json_load runs jshn(.c) -r $input, jshn replaces all "-" with "_".
> This seemed strange to me as jshn.sh can handle json_add_string
> "foo-bar" "foo" (as used in rpcd-mod-packagelist).
> 
> A possible solution could be to add the code of rpcd-mod-packagelist to
> the attendedsysupgrade.sh to avoid json_load.
> 
> Paul
> 
> [1]: https://github.com/openwrt/packages/pull/5037#issuecomment-340416041
> [2]: https://github.com/aparcar/gsoc17-attended-sysupgrade
> [3]: https://ledeupdate.planetexpress.cc/imagebuilder
> [4]:
> https://github.com/openwrt/luci/tree/master/applications/luci-app-attendedsysupgrade
> [5]: https://github.com/aparcar/packages/tree/attendedsysupgrade
> 

Hmm, yes, avoiding rpcd altogether might be a good option here. Not so nice
from the modularity point of view, but it would
- avoid a dependency on rpcd for the CLI feature
- make the whole thing faster (as I mentioned, jshn_load can get really
slow with long JSON docs in busybox)

You could pull out the relevant opkg database parsing code into a separate
shell file used by both rpcd-mod-packagelist and the attended sysupgrade CLI.


Matthias

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/lede-dev/attachments/20171030/71f1b0ce/attachment.sig>