Issues with TV program download, One of Us

[Vangelis forthnet]

On Sun Sep 11 14:33:31 BST 2016, artisticforge . wrote:

> attempting to fetch One of Us.;
> http://www.bbc.co.uk/programmes/b07r26py
> (snip)
> Recording:  1365.44MB / 2278.88MB  3400kbps  59.9% 00:36:40 remaining
> Recording:  1367.72MB / 2278.88MB  3400kbps  60.0% 00:36:35 remaining
>
> INFO: Recorded: 1367.82MB in 00:54:55 at  3401kbps to
>( snip) /One_of_Us_-_3._Glenarvon_Loch_b07v6kv1_original.partial.mp4.ts
> (snip)
> I checked the episodes and it appears that they are in fact complete.
> (snip)
> One_of_Us_-_3._Glenarvon_Loch_b07v6kv1_original.mp4:
> Track: 1
> Type: video
> Info: H264 High at 3.2, 3469.600 secs, 3056 kbps, 1280x720 @ 50.000000
> (snip)
> ../get_iplayer-2.96/get_iplayer --force --verbose --type=tv --mode=best --pid=b07v6kv1 
> 2>&1  | tee -a

On Sun Sep 11 19:01:45 BST 2016, Alan Milewczyk wrote:

> I hadn't noticed that all three programmes
> seem to be around 1300MB rather than the 2 Gigs that one
> would expect from one hour recordings at 1280x720 50 fps.
> However, as you say all programmes are complete.
> (snip)
> Puzzled!

Many greetings to you both Terry and Alan!

 I shall stick to data only pertaining to Episode 3 of the series,
as the same applies to the previous two episodes...

 What the original post of terry's documents is in fact
caused by a discrepancy between the file size reported
by the beeb's stream metadata (prior to downloading)
and the actual size of the file extant on the corresponding
CDN server.

 GiP uses the stream metadata to calculate a rough
estimate of the file size to be fetched; terry is using
GiP 2.96 and --modes=best for a TV prog, this
defaults to --tvmode=hvfhd to be tried first for the
download.

If you perform a verbose info query for pid=b07v6kv1:

perl get_iplayer-296w.pl --pid=b07v6kv1 -i -v > VerboseInfo.txt

and you then scroll down the log to the "Found mode"
section for --versions=original (the second set of modes),
you get:

INFO: Found mode hvfhd1: (gip_hvf_iplayer_5380) hls h264 1280x720 5380kbps 
stream (CDN: mf_limelight_uk_hls/2)
INFO: Found mode hvfhd2: (gip_hvf_iplayer_5380) hls h264 1280x720 5380kbps 
stream (CDN: mf_akamai_uk_hls/1)

As you can see, GiP expects the bitrate of the
hvfhd file to be 5380kbps, yet the probe
of the downloaded file, as performed by terry,
reveals the actual bitrate to be:

V=3056kbps + A=128kbps ~ 3185kbps

3185 ~ 60% of 5380, hence GiP reports
the download to be complete at 60% of
the initial size estimate.

There you have it! Nothing to be puzzled by!

The iPlayer team encoded that particular
series (and possibly other programmes,
as hinted by Alan) at a lower bitrare than usual,
however they have not updated the
relevant stream metadata to reflect
that reduced bitrate - no fault of GiP's!

=============================
Note to the developer:

Examining the above verbose log for the
audiodescribed version, vpid=p0471vhc,
reveals not less than six (!) hvfhigh tvmodes:

hvfhigh1: (gip_hvf_iplayer_956) hls h264 704x396 956kbps stream (CDN: 
mf_limelight_uk_hls/2)
hvfhigh2: (gip_hvf_iplayer_956) hls h264 704x396 956kbps stream (CDN: 
mf_akamai_uk_hls/1)
hvfhigh3: (gip_hvf_iplayer_989) hls h264 704x396 989kbps stream (CDN: 
mf_limelight_uk_hls/1)
hvfhigh4: (gip_hvf_iplayer_1757) hls h264 704x396 1757kbps stream (CDN: 
mf_limelight_uk_hls/1)
hvfhigh5: (gip_hvf_iplayer_956) hls h264 704x396 956kbps stream (CDN: 
mf_akamai_uk_hls/1)
hvfhigh6: (gip_hvf_iplayer_956) hls h264 704x396 956kbps stream (CDN: 
mf_limelight_uk_hls/2)

Yes they're all at 704x396 resolution,
but at 3 different bitrates, 1757kbps (possibly 50fps),
989kbps and 956kbps.

Loading
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/iptv-all/vpid/p0471vhc
http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/apple-ipad-hls/vpid/p0471vhc

it appears that the 1757kbps bitrate is indeed at 50fps
with 128kbps audio and is dervided from the "iptv-all" mediaset:

#EXT-X-STREAM-INF:BANDWIDTH=1800000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=704x396,FRAME-RATE=50
vf_b07v6hy4_64e1fd28-440a-42d0-9f97-08792f7cff6f.ism.hlsv2-audio_1=128000-video=1570000.m3u8,

The 989kbps bitrate is at 25fps with 128kbps audio, also
derived from the iptv-all mediaset:

#EXT-X-STREAM-INF:BANDWIDTH=1013000,CODECS="mp4a.40.2,avc1.4D401E",RESOLUTION=704x396,FRAME-RATE=25
vf_b07v6hy4_64e1fd28-440a-42d0-9f97-08792f7cff6f.ism.hlsv2-audio_1=128000-video=827000.m3u8

while the 956kbps bitrate (25fps) with 96kbps audio (HE-AAC)
are all derived from the "apple-ipad-hls" mediaset:

#EXT-X-STREAM-INF:BANDWIDTH=979000,CODECS="mp4a.40.5,avc1.4D401E",RESOLUTION=704x396,FRAME-RATE=25,AUDIO="audio-aach-96",CLOSED-CAPTIONS=NONE
vf_b07v6hy4_64e1fd28-440a-42d0-9f97-08792f7cff6f-audio=96000-video=827000.m3u8

I haven't tried to actually download any of those hvfhigh modes.
I have a gut feeling the above stream detection is bugged;
hvfhigh tvmodes should not have been derived from
iptv-all mediaset, only from the apple-ipad-hls one;
I think what breaks GiP here is the existence
of supplier="mf_bidi_uk_hls" in the available CDNs for iptv-all too!

And the duplication of (CDN: mf_akamai_uk_hls/1)
and (CDN: mf_limelight_uk_hls/2) (four 956kbps
modes detected) can also be put down to CDN bidi
(more likely unsuccesful removal of the unsupported
"mf_bidi_uk_hls" streams from the audiodescribed
version).

Please have a look...
=============================

Regards,
Vangelis.