BBC's iPlayer works when get_iplayer doesn't

[Alan Milewczyk]

On 31/03/2017 01:23, Vangelis forthnet wrote:
> On Wed Mar 29 22:51:06 BST 2017, Alan Milewczyk wrote:
>
>> I'm now back in the UK after 4 months in the Philippines
>
> Hi there, dear Alan :-)
> Back in the nest, I guess, for you then...
> I do hope things have stayed just as you left them...

Hi again dear Vangelis :-)

Yep back in the UK after a tortuous flight back. It was bad enough being 
on an excruciatingly long 12.5 hour flight from Guangzhou to Amsterdam, 
only to find that the last leg of my journey back to Manchester had been 
cancelled due to weather problems, as had many other flights. In the 
ensuing chaos in the baggage handing area, my checked in luggage was 
"misplaced". I wasn't willing to travel without it, missing replacement 
flights I'd been booked on meaning I ending up spending some 12 hours in 
Schiphol when all I wanted to do was to sleep in my bed. Anyway, it all 
ended well but now I have a leg infection and very painful cellulitis - 
been on the antibiotics for 2 days now and no evidence yet of them 
kicking in! Sigh... :-(
>
>> so I'm into catch-up mode while new programmes keep on coming 
>> relentlessly.
>> Last year it took me about 2 months to catch up on 4 months missed TV!
>
> ...At least you do manage to properly listen to/watch your GiP 
> downloads...
> I think at some point I stopped fetching progs I realised I will never 
> find
> the time to properly listen to (am most a radio downloader); with 
> cheap storage
> nowadays, I was dangerously bordering digital hoarding...

I don't have a serious hoarding issue with the TV programmes although I 
do with radio, but then I am a hoarder (or should it be collector?) :-(
>
>> get_iplayer --info --pid b08l429m
>> (snip)
>> INFO: No versions of this programme were selected
>> (available versions: audiodescribed)
>> At first I wondered if this was an error
>> on the part of the BBC team
>> (snip)
>> I'm wondering am I missing something here?
>> Any thoughts?
>
> ... Your reported issue intrigued me, hence one
> of my (notoriously?) long-ish analyses follows...
>
> TL;DR: The manifestation of your issue is caused by an
> unfortunate coincidence of two things:
> 1. Incomplete metadata on the beeb's side for that
> particular episode (YES, the BBC is to blame in part)
> 2. An otherwise very sane commit in the main GiP
> perl script, that was pushed on Jan 6, 2016.
>
> Those with traces of Hercule Poirot inside you, do read on!

I do admire your forensic abilities regarding software issues. Must 
admit I'm a hardware man and time is a factor in stopping me pursuing 
the nuts and bolts of a program what with my countless other projects 
currently on the go.

> ======================================================
>
> On Thu Mar 30 04:03:47 BST 2017, tellyaddict wrote:
>
>> You are right, GiP 2.99 only picks up the audiodescribed version
>> (snip)
>> Version 2.94 can see it but anything after that for some reason can't.
>> Also in 2.99 in the option to use alternative metadata
>> streams --ybbcy which also sees the missing version.
>
> Many thanks tellyaddict for those additional clues!

Yes belated thanks too to tellyaddict!

>
> Now, under normal conditions (XML feeds up and running),
> GiP uses a prog's XML page
> "http://www.bbc.co.uk/programmes/<pid>.xml"
> to retrieve many metadata, including versions
> availability and their vPIDs (used to point to stream data).
> For said programme (pid=b08l429m), if you load the page
> in a browser and focus on the <versions> element you get:
>
> <versions>
> <version canonical="0">
> <pid>b08l4283</pid
>> <duration>1800</duration>
> <types>
> <type>Store only</type>
> </types>
> </version>
> <version canonical="0">
> <pid>p04xtjsg</pid>
> <duration>1800</duration>
> <types>
> <type>Store only</type>
> </types>
> </version>
> <version canonical="0">
> <pid>p04y79vv</pid>
> <duration>1800</duration>
> <types>
> <type>Dubbed Audio Described</type>
> <type>Store only</type>
> </types>
> </version>
> </versions>
>
> So, three (3) versions are present:
> v1 = "Store only", vpid=b08l4283 [-> streams exist]
> v2 = "Store only", vpid=p04xtjsg [-> no streams]
> v3 = "Dubbed Audio Described/Store only", vpid=p04y79vv [-> streams 
> exist]
>
> With GiP 2.94, --info will produce (among other):
>
> title:          EastEnders: 27/03/2017
> type:           tv
> verpids:        audiodescribed: p04y79vv
> verpids:        default: b08l4283
> verpids:        store: b08l4283
> version:        default
> versions:       audiodescribed,default,store
>
> while 2.95 will show:
>
> title:          EastEnders: 27/03/2017
> type:           tv
> verpids:        audiodescribed: p04y79vv
> version:        default
> versions:       audiodescribed
>
> where one sees the "store" version not mentioned...
> Closely inspecting
> "https://github.com/get-iplayer/get_iplayer/compare/v2.94...v2.95"
> one can see why; it's because of
> "Skip store only version"
> https://github.com/get-iplayer/get_iplayer/commit/a5153fc
>
> 99.999% of times, the "store only" version lacks any streams,
> so rightfully the maintainer pushed the commit to disregard
> it; but in this case with "broken" metadata, the first
> "store only" version corresponds to the only non-AD, non-SL
> version and GiP 2.95+ skips it, taking into account only
> the AD flavour...
>

Ah, I get it
> Workaround 1:
> In your version of the main script
> (on Windows it's get_iplayer.pl), locate line
>
> next if $type =~ /store/;
>
> (in GiP 2.99 it's line 4286)
> and temporarily comment it out (prepend #).
> Then you should be able to fetch the non-AD version.
>
> As to why --ybbcy switch, as reported by tellyaddict,
> also functions as a workaroud,
> 1. --ybbcy switch assumes outage of XML metadata pages,
> 2. commit "Remove fallback metadata method" (2.98+)
> https://github.com/get-iplayer/get_iplayer/commit/1b5160b
> assumes the same for (and does away with) RDF metadata pages, so
> 3. GiP only parses JSON playlist URIs
> INFO: iPlayer metadata URL (JSON) = 
> http://www.bbc.co.uk/programmes/b08l429m/playlist.json
> and scrapes the prog's iplayer webpage
> INFO: iPlayer metadata URL (HTML) [default] = 
> http://www.bbc.co.uk/iplayer/episode/b08l429m/
> so vpid=b08l4283 is successufuly harvested,
> because the "skip store only version" rule
> doesn't apply here...
>
> Workaround 2: Issue
> get_iplayer --pid=b08l429m --ybbcy
>
> CASE CLOSED AND FILED!

Brilliant analysis, Vangelis, thank you.
>
> As an exercise for the curious, load
> http://www.bbc.co.uk/programmes/b08l42d7.xml
> and try to understand why GiP 2.99 has no
> problem fetching the non-AD version of
> EastEnders: 28/03/2017
>

Ok, let's try.... for my failing programme the relevant section in the 
xml was:
<versions><version 
canonical="0"><pid>b08l4283</pid><duration>1800</duration><types><type>Store 
only</type></types></version><version 
canonical="0"><pid>p04xtjsg</pid><duration>1800</duration><types><type>Store 
only</type></types></version>

whereas for your working version it was:
<versions><version 
canonical="1"><pid>b08l42bq</pid><duration>1800</duration><types><type>Original 
version</type></types></version><version 
canonical="0"><pid>p04xtj64</pid><duration>1620</duration><types><type>Store 
only</type></types></version><version 
canonical="0"><pid>p04ybhb3</pid><duration>1800</duration><types><type>Dubbed 
Audio Described</type></types></version></versions>

Sorry I've lost the formatting in the cut and paste but basically my 
programme had only two versions, "Store only" and "Dubbed Audio 
Described" whereas the working programme had three versions, "Original 
version", "Store only" and "Dubbed Audio Described".

Not sure of the significance of <version canonical="1"> in yours and 
<version canonical="0"> in mine, though!

I must admit my eyes have glazed over in the past when looking at xml 
files but now I don't feel as daunted!

Appreciate the analysis as always. Having seen the intolerant attitude 
to queries in "the other place", I'm glad I'm here.

> All the best,
> Vangelis.
>

And to you my friend.... I'm in pain so I'm off back to bed with my 
painkillers.


Alan

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus