Loss of XML feeds...

[Vangelis forthnet]

... Of course this is no news for those of you
frequenting recently the Support Forums

https://squarepenguin.co.uk/forums/thread-1193.html

and this mailing list

http://lists.infradead.org/pipermail/get_iplayer/2017-March/010310.html

A github issue has been created by the code maintainer:

https://github.com/get-iplayer/get_iplayer/issues/323

Some personal thoughts/remarks:

> This is ostensibly a temporary situation,
> but there is no way to to know if that is true.

Last time I refreshed the tv.cache was early Monday (Mar 13th) morning;
in the afternoon it was broken; even if the loss is indeed temporary,
what is trully ominous is the forewarning showed if one loads manually
in a browser one sample TV schedule URL, e.g.
http://www.bbc.co.uk/bbcone/programmes/schedules/london/this_week.xml =>
[quote]
In May 2017, regrettably, this system will permanently cease producing XML 
and rdf outputs.
A JSON service will be kept in place for a period following this
[/quote]
So even if the feeds are restored, say, tomorrow, they'll vanish for good
only in roughly six weeks' time - which is pretty soon... :-(
So, undoubtedly, this fact has to be adressed by the coder: move to
the eventual JSON feeds - but then what?

> This has broken programme indexing and cache refresh

This is true for the tv.cache; luckily, because of

https://github.com/get-iplayer/get_iplayer/commit/0ad570d54221e53870f56fbafe95f564c84a75e2

after GiP fails to download any radio schedules due to the feeds
being currently broken, it falls back to the AOD feeds
(thank The Lord they still exist!) and hence it populates
the radio.cache. Issuing

perl get_iplayer-299.pl --type=radio -f --force

I get in the end a radio.cache of 13598 entries at
the time of writing, but, alas, covering only the
last 7 days of broadcast, as is noted in the commit message:

>> Only last 7 days can be indexed from AOD feeds,
>> and no future programme listings will be available.

> Downloading with --pid still works,
> but the programme metadata will be unreliable.
> For the moment, just don't use get_iplayer.

...But if you must use GiP, then the --pid method,
as pointed out also in the list, STILL WORKS; the
caveat is imperfect/incomplete file names and file tags...
you can use the --file-prefix switch to manually craft
a correct file title (or just rename a file post fetch)
and use 3rd party MP4 taggers to edit the file tag...

The main issue I have with my radio downloads is
that some of them have wrong embedded thumbnails!
I have to retrieve the correct one by
1. Loading the JSON playlist:
http://www.bbc.co.uk/programmes/<pid>/playlist.json
2. scrolling down to, e.g.

holdingImage": "//ichef.bbci.co.uk/images/ic/976x549/p038jcmv.jpg"

and then compose a URI for the 640x360 thumbnail:

http://ichef.bbci.co.uk/images/ic/640x360/p038jcmv.jpg

A batch file fetches the thumbnail via curl,
and then deletes the wrong one and embeds
the new one via mp4art.exe
If I can manually identify the correct thumbnail,
but GiP cannot while the XML feeds are down,
then I conclude this is a bug; for any party interested,
--pid=b08gwqyf gets tagged with a thumbnail of
Reggie Yates, while the correct one (as shown in
http://www.bbc.co.uk/programmes/b08gwqyf )
is one of Greg James.

> An announcement will be made in the forums
> if/when the BBC data returns.

tellyaddict/tvfan asked in the forums:

> Isn't there already a fall back option built in
> that was added after the last few times this happened?

He was in fact alluding to

https://github.com/get-iplayer/get_iplayer/commit/14d26af99a57cc51c23ace383e7cf3225a23e13e
(and for those not on Windows it also includes:
 https://github.com/get-iplayer/get_iplayer/commit/ee7a04934daeec327cd4bfaddeab6b0225a07b44 )Of course the maintainer is fully awareof its existence, but has withheld referenceto it because it is to be used only as a last resort;it involves web-scraping HTML pages for TV and(along the various limitations cited in the commit'smessage) has a serious toll on the beeb's servers!We, as GiP users, do not want to bring the beeb'sservers down to a crawl...For "academic" reasons,perl get_iplayer-299.pl --type=tv -f --force --ybbcy --refresh-exclude="BBCAlba,BBC Parliament,S4C,CBeebies"took 12min to build a tv.cache of 1086 entries (YMMV);local channels are ignored by default. --ybbcy switch appliesto GiP 2.95+As is the wish of the maintainer, for further directions& guidance keep your eyes open in the Support Forums:https://squarepenguin.co.uk/forums/Kind regards