BBC's iPlayer works when get_iplayer doesn't

[Vangelis forthnet]

On Wed Mar 29 22:51:06 BST 2017, Alan Milewczyk wrote:

> I'm now back in the UK after 4 months in the Philippines

 Hi there, dear Alan :-)
Back in the nest, I guess, for you then...
I do hope things have stayed just as you left them...

> so I'm into catch-up mode while new programmes keep on coming 
> relentlessly.
> Last year it took me about 2 months to catch up on 4 months missed TV!

...At least you do manage to properly listen to/watch your GiP downloads...
I think at some point I stopped fetching progs I realised I will never find
the time to properly listen to (am most a radio downloader); with cheap 
storage
nowadays, I was dangerously bordering digital hoarding...

> get_iplayer --info --pid b08l429m
> (snip)
> INFO: No versions of this programme were selected
> (available versions: audiodescribed)
> At first I wondered if this was an error
> on the part of the BBC team
> (snip)
> I'm wondering am I missing something here?
> Any thoughts?

... Your reported issue intrigued me, hence one
of my (notoriously?) long-ish analyses follows...

TL;DR: The manifestation of your issue is caused by an
unfortunate coincidence of two things:
1. Incomplete metadata on the beeb's side for that
particular episode (YES, the BBC is to blame in part)
2. An otherwise very sane commit in the main GiP
perl script, that was pushed on Jan 6, 2016.

Those with traces of Hercule Poirot inside you, do read on!
======================================================

On Thu Mar 30 04:03:47 BST 2017, tellyaddict wrote:

> You are right, GiP 2.99 only picks up the audiodescribed version
> (snip)
> Version 2.94 can see it but anything after that for some reason can't.
> Also in 2.99 in the option to use alternative metadata
> streams --ybbcy which also sees the missing version.

Many thanks tellyaddict for those additional clues!

Now, under normal conditions (XML feeds up and running),
GiP uses a prog's XML page
"http://www.bbc.co.uk/programmes/<pid>.xml"
to retrieve many metadata, including versions
availability and their vPIDs (used to point to stream data).
For said programme (pid=b08l429m), if you load the page
in a browser and focus on the <versions> element you get:

<versions>
<version canonical="0">
<pid>b08l4283</pid
><duration>1800</duration>
<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04xtjsg</pid>
<duration>1800</duration>
<types>
<type>Store only</type>
</types>
</version>
<version canonical="0">
<pid>p04y79vv</pid>
<duration>1800</duration>
<types>
<type>Dubbed Audio Described</type>
<type>Store only</type>
</types>
</version>
</versions>

So, three (3) versions are present:
v1 = "Store only", vpid=b08l4283 [-> streams exist]
v2 = "Store only", vpid=p04xtjsg [-> no streams]
v3 = "Dubbed Audio Described/Store only", vpid=p04y79vv [-> streams exist]

With GiP 2.94, --info will produce (among other):

title:          EastEnders: 27/03/2017
type:           tv
verpids:        audiodescribed: p04y79vv
verpids:        default: b08l4283
verpids:        store: b08l4283
version:        default
versions:       audiodescribed,default,store

while 2.95 will show:

title:          EastEnders: 27/03/2017
type:           tv
verpids:        audiodescribed: p04y79vv
version:        default
versions:       audiodescribed

where one sees the "store" version not mentioned...
Closely inspecting
"https://github.com/get-iplayer/get_iplayer/compare/v2.94...v2.95"
one can see why; it's because of
"Skip store only version"
https://github.com/get-iplayer/get_iplayer/commit/a5153fc

99.999% of times, the "store only" version lacks any streams,
so rightfully the maintainer pushed the commit to disregard
it; but in this case with "broken" metadata, the first
"store only" version corresponds to the only non-AD, non-SL
version and GiP 2.95+ skips it, taking into account only
the AD flavour...

Workaround 1:
In your version of the main script
(on Windows it's get_iplayer.pl), locate line

next if $type =~ /store/;

(in GiP 2.99 it's line 4286)
and temporarily comment it out (prepend #).
Then you should be able to fetch the non-AD version.

 As to why --ybbcy switch, as reported by tellyaddict,
 also functions as a workaroud,
 1. --ybbcy switch assumes outage of XML metadata pages,
 2. commit "Remove fallback metadata method" (2.98+)
 https://github.com/get-iplayer/get_iplayer/commit/1b5160b
 assumes the same for (and does away with) RDF metadata pages, so
 3. GiP only parses JSON playlist URIs
 INFO: iPlayer metadata URL (JSON) = 
http://www.bbc.co.uk/programmes/b08l429m/playlist.json
 and scrapes the prog's iplayer webpage
 INFO: iPlayer metadata URL (HTML) [default] = 
http://www.bbc.co.uk/iplayer/episode/b08l429m/
 so vpid=b08l4283 is successufuly harvested,
 because the "skip store only version" rule
 doesn't apply here...

 Workaround 2: Issue
 get_iplayer --pid=b08l429m --ybbcy

 CASE CLOSED AND FILED!

 As an exercise for the curious, load
 http://www.bbc.co.uk/programmes/b08l42d7.xml
 and try to understand why GiP 2.99 has no
 problem fetching the non-AD version of
 EastEnders: 28/03/2017

 All the best,
 Vangelis.