[LEDE-DEV] [PATCH] base-files: sysupgrade. Added the ability to use URL as source

Wed Aug 24 02:16:15 PDT 2016

On 24/08/2016 11:12, Сергеев Сергей wrote:
> Hello. Ok. Leave all as is. But...(comments below):
>>
>> On 13/08/2016 13:04, adron at yapic.net wrote:
>>> From: Sergey Sergeev <adron at yapic.net>
>>>
>>>    scp uls(like this
>>> scp://adron@192.168.88.6:lede/lede-ar71xx-mikrotik-NAND-512b-squashfs-sysupgrade.bin)
>>>
>>>      is supported too. And you also can specify custom ssh
>>> port(...8.88.6:22110:lede...)
>>>
>>> Signed-off-by: Sergey Sergeev <adron at yapic.net>
>>> ---
>>>   package/base-files/files/lib/upgrade/common.sh | 26
>>> +++++++++++++++++++++-----
>>>   package/base-files/files/sbin/sysupgrade       | 15 +++++++++++++++
>>>   2 files changed, 36 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/package/base-files/files/lib/upgrade/common.sh
>>> b/package/base-files/files/lib/upgrade/common.sh
>>> index ea03f77..080bea5 100644
>>> --- a/package/base-files/files/lib/upgrade/common.sh
>>> +++ b/package/base-files/files/lib/upgrade/common.sh
>>> @@ -180,20 +180,36 @@ get_image() { # <source> [ <command> ]
>>>       local from="$1"
>>>       local conc="$2"
>>>       local cmd
>>> +    local need_quotes=1
>>>         case "$from" in
>>> -        http://*|ftp://*) cmd="wget -O- -q";;
>>> -        *) cmd="cat";;
>>> +        scp://*)
>>> +            cmd="ssh"
>>> +            # extract custom ssh port(if specified)
>>> +            local custom_port=`echo $from | sed -ne
>>> 's!.\+:\([0-9]\+\):.\+!\1!p'`
>>> +            [ -n "$custom_port" ] && {
>>> +                cmd="$cmd -p $custom_port"
>>> +                # remove custom port from url
>>> +                from=`echo $from | sed -ne 's!:\([0-9]\+\):!:!p'`
>>> +            }
>>> +            need_quotes=0 # here we add quotes manually
>>> +            from=`echo $from | sed -ne 's!scp://!\"!p' | sed -ne
>>> 's!:\(.\+\)!\" \"cat \1\"!p'`
>>> +        ;;
>>> +        http://* | https://* | ftp://*) cmd="wget -O-" ;;
>>> +        *) cmd="cat" ;;
>>>       esac
>>> +
>>> +    # if needed => add quotes to $from var
>>> +    [ $need_quotes -eq 1 ] && from=\"$from\"
>>> +
>>>       if [ -z "$conc" ]; then
>>> -        local magic="$(eval $cmd \"$from\" 2>/dev/null | dd bs=2
>>> count=1 2>/dev/null | hexdump -n 2 -e '1/1 "%02x"')"
>>> +        local magic="$(eval $cmd $from | dd bs=2 count=1 2>/dev/null
>>> | hexdump -n 2 -e '1/1 "%02x"')"
>>>           case "$magic" in
>>>               1f8b) conc="zcat";;
>>>               425a) conc="bzcat";;
>>>           esac
>>>       fi
>>> -
>>> -    eval "$cmd \"$from\" 2>/dev/null ${conc:+| $conc}"
>>> +    eval "$cmd $from ${conc:+| $conc}"
>>>   }
>>>     get_magic_word() {
>>> diff --git a/package/base-files/files/sbin/sysupgrade
>>> b/package/base-files/files/sbin/sysupgrade
>>> index 2f441f8..fc1df22 100755
>>> --- a/package/base-files/files/sbin/sysupgrade
>>> +++ b/package/base-files/files/sbin/sysupgrade
>>> @@ -137,6 +137,21 @@ include /lib/upgrade
>>>     [ "$1" = "nand" ] && nand_upgrade_stage2 $@
>>>   +get_if_URL(){
>>> +    local url="$1"
>>> +    local url_repl_file="/tmp/sysupgrade-URL.bin"
>>> +
>>> +    case "$url" in
>>> +        http://* | https://* | ftp://* | scp://*)
>>> +        get_image "$url" "cat" > $url_repl_file
>>> +        ARGV=${ARGV/"$url"/"$url_repl_file"}
>>> +        ;;
>>> +    esac
>>> +}
>>> +
>>> +# get image file from URL if specified
>>> +get_if_URL "$ARGV"
>>> +
>> this get_if_url mus go away. the way it should work is
>> 1) derive local file basename from url
>> 2) if not there yet, download
>> 3) if already downloaded, cat it
>>
>> that way only get_image needs to be changed and the functionality is
>> self contained. get_image should transparently download the image and if
>> it already did that cat it
>>
>>     John
> Here's how it happens(without get_if_UR)  with TP-link devices(wget
> verbose mode is on):
> root at tl-wr841-v10:~# sysupgrade
> http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgra
> 
> de.bin
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Saving config files...
> killall: watchdog: no process killed
> Sending TERM to remaining processes ... dnsmasq ubusd askfirst logd rpcd
> netifd uhttpd sync_helper ntpd
> Sending KILL to remaining processes ... askfirst
> Switching to ramdisk...
> Performing system upgrade...
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> Downloading
> 'http://192.168.88.6/lede/tplink/lede-ar71xx-generic-tl-wr841-v10-squashfs-sysupgrade.bin'
> 
> Connecting to 192.168.88.6:80
> Writing to stdout
> Unlocking firmware ...
> 
> -                    100% |*******************************|  3392k
> 0:00:00 ETA
> Download completed (3473412 bytes)
> 
> Appending jffs2 data from /tmp/sysupgrade.tgz to firmwar
> Upgrade completed
> Rebooting system...
> 
> As you can see in this case, the same firmware file is downloaded 8
> times in a row!
> If we have a slow or unstable connection - this may be a problem.
> And in my opinion: it is logically incorrect.
> 

hence step 2/3. if get_image already has the file it should cat it
instead of downloading it again.

> P.S: In case of not TP-Link devices we even can'tupgrade the firmware
> using url(see target/linux/ar71xx/base-files/lib/upgrade/platform.sh,
> function platform_do_upgrade_combined that not use cat_image at all and
> try to do dd for raw url!)

well that would need fixing then wont it !

	John

> 
>>
>>
>>>   do_save_conffiles() {
>>>       local conf_tar="${1:-$CONF_TAR}"
>>>  
> 