Programme in Series has `firstbcast' But no `available'.

[Ralph Corderoy]

Hi Vangelis,

> > It seems to have.
>
> > I have done ./get_iplayer --quiet --refresh --refresh-future
>
> "--refresh --refresh-future" won't help you in this case; your
> tv.cache gap for pid=b08nyc9z is for a broadcast date more than a week
> in the past; "--refresh --refresh-future" will index tv content for
> current and next calendar weeks

It used to.  The help for --future in 3.00 still says

    Additionally search future programme schedule if it has been indexed
    (refresh cache with: --refresh --refresh-future).

but the now code does

    if ( $future ) {
	push @schedule_dates, "this_week" unless $opt->{ybbcy};
	push @schedule_dates, "next_week";
    } else {

so with ybbcy permanently set, --future only does next week.  :-(

> do as told to regenerate your past 15 (or more) days worth of the
> tv.cache file...

"as told" is `get_iplayer -f --refresh-limit-tv=15 --index-maxconn=10'.

Since I use sequential indexing, and I'm in no hurry when I refresh, I
ditched the maxconn.  I've now done

    ./get_iplayer --quiet --refresh --refresh-limit-tv=30 \
	--exclude-category children \
	--exclude-channel 'b(?:BBC Alba|CBBC|CBeebies|S4C)b' \
	QEM78eehLjf

    ./get_iplayer --quiet --refresh --refresh-future \
	--exclude-category children \
	--exclude-channel '\b(?:BBC Alba|CBBC|CBeebies|S4C)\b' \
	QEM78eehLjf

BTW, the help for --refresh-limit-tv is

    Number of days of TV programmes to cache. Makes cache updates slow.
    Default: 7 Min: 1 Max: 30

though I noticed a `0' rather than `1' in

    my $limit_days = $opt->{"refreshlimit".${prog_type}} || $opt->{"refreshlimit"};
    $limit_days = 0 if $limit_days < 0;
    $limit_days = 30 if $limit_days > 30;

Also, I think a line break is missing before the second INFO, e.g.

    INFO: Using sequential indexing
    ..................................INFO: Using sequential indexing
    ..................................

> I still fail to grasp the importance it holds for you to have that
> "available" entry in --info for that particular episode

I have existing pre-3.00 scripts that processed the output of --info
--pid and `available' was one of the fields they use.

> Other -i results, (like the long description & modes) which do not
> appear in the cache, are fetched from online metadata feeds, currently
> http://www.bbc.co.uk/programmes/b08pm8cl.json

That's E09, but I get the idea, thanks.

    $ curl -sS http://www.bbc.co.uk/programmes/b08nyc9z.json |
    > jq .programme.first_broadcast_date
    "2017-04-27T10:00:00+01:00"
    $

> "available" is actually the 4th column of the tv.cache file, so if a
> (TV) PID is already indexed, -i --pid=<pid> will include the
> "available" entry, otherwise not.

I've checked tv.cache before and after my above refreshes and the
`available' field has a date-time for all entries.  b08nyc9z now appears
in tv.cache, so that's good.

> The "available" field for an indexed PID was originally populated by
> fetching the value of first_broadcast_date for that PID (from the
> feeds).

I couldn't spot in the code where the `available' that makes it into
tv.cache is obtained from first_broadcast_date.  (Though I did notice
the tv.cache's first line doesn't have a trailing pipe separator, unlike
the data lines.)

> When a PID is not indexed, you'll only get the "firstbcast:" -i
> result, not the "available:" one.

Right, because "on the fly" formatting of non-cache data doesn't output
it.  (Bug?)

This has been very helpful, thanks.  Exploring tv.cache means I can
adjust my script to ditch get_iplayer's --info --pid and pluck what I
want from there far more efficiently.

When reading get_links_schedule(), I wondered if using tv.cache's
modification time as an indicator of the cache being fetched `this_week'
and so not doing that again is a bit flawed if it was previously fetched
around the Sunday night/Monday morning boundary.  W18's data could be
fetched with an mtime of W19, for example.

    $ date -d sunday +'%Y-%m-%d %W %a'
    2017-05-07 18 Sun
    $ date -d monday +'%Y-%m-%d %W %a'
    2017-05-08 19 Mon

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy