OT help with wget

MacFH - C E Macfarlane - News news at macfh.co.uk
Thu Feb 17 16:01:23 PST 2022


I've sometimes used this bash script, which originally I wrote to 
download a section of a site concerning some hardware I owned when the 
site looked like it was about to close down, which in fact it did, so in 
due course I was glad that I'd had the foresight.

I usually find that it gets related images, etc, except where they 
aren't stored in the same sub tree as the root document.

(Note, beware unintended line wrap)

wGetList.sh
===========

#!/bin/sh
#	The name of the directory containing this script
DIRY="${0%/*}/"
# echo "Directory: ${DIRY}"

#	The filename of this script
SCRIPT="${0##*/}"
# echo "Filename: ${SCRIPT}"

#	User Agent string  -  Chrome
UAGENT="--user-agent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) 
Gecko/20100101 Firefox/56.0'"

#	Options for wget command
OPTIONS="-F -r -c -nc -t 3 -T 60 --retry-connrefused -w 20 --random-wait 
--waitretry=60 --no-check-certificate ${UAGENT}"

#	Input list of URLs to retrieve
LIST=""
LOG=""
if [ "${1:0:4}" != "http" ]
	then
		if [ -f "${1}" ]
			then
				LIST="-i ${1}"
				LOG="${1%.*}.log"
				shift
				OPTIONS="${OPTIONS} ${LIST} ${*}"
			else
				echo "WARNING - url list file '$1' not found!"
				exit 1
		fi
	else
		OPTIONS="${OPTIONS} ${*}"
		LOG="${1##*//}"
		LOG="${LOG%%/*}.log"
fi


#	WGET the files in the list of URLs
echo "echo ${OPTIONS} | xargs wget > \"${LOG}\" 2>&1"
echo ${OPTIONS} | xargs wget > "${LOG}" 2>&1 &


On 17/02/2022 17:03, Dave Widgery wrote:
> Hi
> Sorry I know this is very OT but i thought there might be a few people
> here that might be able to help and possibly email me directly.
> We have several blogs (using googles blogger) that my wife has created
> over the years, but I want to create local copy's on my PC. I used the
> following command.
> wget --mirror --convert-links --adjust-extension --page-requisites
> --no-parent http://xxxx.blogspot.com
> and it created a full structure of the blog on my PC but it still
> relies  on links to external websites for the images, can anybody
> suggest how to get it to also download copy's of the all the images as
> well?
> Again sorry for the OT post but I have been going round in circles for a while.
> Dave
> 
> _______________________________________________
> get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer




More information about the get_iplayer mailing list