From web at audiomisc.co.uk  Tue May  1 01:00:05 2018
From: web at audiomisc.co.uk (Jim web)
Date: Tue, 01 May 2018 09:00:05 +0100
Subject: no  more hslv format  ?
In-Reply-To: <5AE779A9.20973.A9D9328@peter.kirk.isauk.biz>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk> <5AE779A9.20973.A9D9328@peter.kirk.isauk.biz>
Message-ID: <56f14555f7web@audiomisc.co.uk>

My ISP is Orpheus internet. They aren't cheap, but they are excellent in
terms of service, etc. [1] 

I think my actual connection is a repackage of a BT/OR one that has a cap
at a rolling 50GB/30days. But only the time betweem 9am and midnight is
metered. So I can 'fill my boots' outwith that time if I wish. I then
typically get around 50-60 meg download rates since we got FTTC.

Jim

[1] They helpfully support a range of OS's including my actual favourite,
RISC OS. To give an idea of the service, the owner will give clients his
mobile number in case they need help out-of-hours. I once did at the
weekend, phoned him, and he was out sailing. He said he'd fix the problem,
and it was fixed a short time later.

In article <5AE779A9.20973.A9D9328 at peter.kirk.isauk.biz>, Peter S Kirk
<peter.kirk at isauk.biz> wrote:
> Jim,

> What ISP and plan is that?

> On 30 Apr 2018 at 13:56, Jim web Jim web <web at audiomisc.co.uk> wrote:

> > I can avoid a 'cap' as I do all the fetching before the 9am start of
> > our 'metered' period. 



> _______________________________________________ get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From ralph at inputplus.co.uk  Tue May  1 04:31:38 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Tue, 01 May 2018 12:31:38 +0100
Subject: no more hslv format ?
In-Reply-To: <56f0dca351web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
Message-ID: <20180501113138.870F721091@orac.inputplus.co.uk>

Hi Jim,

> I've been discussing the 'loss' of the 1280x720 25fps version with
> someone at the BBC.

I miss those 1 GiB ~= 1 hour ones too.  They were `just right'.

> It has also set me wondering about arranging for gip to fetch to ram
> storage and then convert that into a file on my main disc.

Is this Linux?  get_iplayer here uses the current working directory for
all its large intermediate files.  If that was a `tmpfs' filesystem then
they would all sit in RAM, unless you've swap space and pressure caused
pages from the filesystem to be swapped out.

You could use get_iplayer's --command option to run a command to move
each final file off tmpfs as the download is finished.  Its --output
affects all the intermediate files too, AIUI.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From ralph at inputplus.co.uk  Tue May  1 04:34:30 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Tue, 01 May 2018 12:34:30 +0100
Subject: no more hslv format ?
In-Reply-To: <56f0e48cc7web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <CAF_ssT8Eyw7GuX105X037kTpoNxCQjXjmunzAFN-2qZ64x33XQ@mail.gmail.com>
 <56f0e48cc7web@audiomisc.co.uk>
Message-ID: <20180501113430.4881C21091@orac.inputplus.co.uk>

Hi Jim,

> How do I find out the command gip sends to ffmpeg to do its default
> conversion as things stand?

Try `--verbose';  that should print external commands that are run.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From alan at soulman1949.com  Tue May  1 06:05:45 2018
From: alan at soulman1949.com (Alan Milewczyk)
Date: Tue, 1 May 2018 14:05:45 +0100
Subject: no more hslv format ?
In-Reply-To: <20180501113138.870F721091@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
Message-ID: <9b2c65e8-ee68-71b1-2dfd-83ab9b36ac1c@soulman1949.com>

On 01/05/2018 12:31, Ralph Corderoy wrote:
> Hi Jim,
>
>> I've been discussing the 'loss' of the 1280x720 25fps version with
>> someone at the BBC.
> I miss those 1 GiB ~= 1 hour ones too.  They were `just right'.
Me too! :-(
>
>> It has also set me wondering about arranging for gip to fetch to ram
>> storage and then convert that into a file on my main disc.
> Is this Linux?  get_iplayer here uses the current working directory for
> all its large intermediate files.  If that was a `tmpfs' filesystem then
> they would all sit in RAM, unless you've swap space and pressure caused
> pages from the filesystem to be swapped out.

I'm using Windows 7 not Linux but I wouldn't have a clue how to do that.
> You could use get_iplayer's --command option to run a command to move
> each final file off tmpfs as the download is finished.  Its --output
> affects all the intermediate files too, AIUI.
>
I've run the suggested command line instruction posted earlier and I'm 
afraid it's a very time consuming process - assuming nothing else is 
going on on my PC I get a conversion speed of approx 1.5x/1.6x (down to 
maybe 1.2x if other processes are active). Maybe some of that could be 
due to the relative slowness of hard drives. It would be good to know 
how to modify the code to convert the programme files on the fly.


Regards


Alan


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



From web at audiomisc.co.uk  Tue May  1 05:05:08 2018
From: web at audiomisc.co.uk (Jim web)
Date: Tue, 01 May 2018 13:05:08 +0100
Subject: no more hslv format ?
In-Reply-To: <20180501113138.870F721091@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
Message-ID: <56f15bc592web@audiomisc.co.uk>

In article <20180501113138.870F721091 at orac.inputplus.co.uk>, Ralph
Corderoy <ralph at inputplus.co.uk> wrote:
> Hi Jim,

> > I've been discussing the 'loss' of the 1280x720 25fps version with
> > someone at the BBC.

> I miss those 1 GiB ~= 1 hour ones too.  They were `just right'.

Yes. It seems bonkers to me for the BBC to end them. But I've not had any
reaction to pointing out that they are useful, etc. So I assume they won't
be re-instated. :-/

> > It has also set me wondering about arranging for gip to fetch to ram
> > storage and then convert that into a file on my main disc.

> Is this Linux?  get_iplayer here uses the current working directory for
> all its large intermediate files.  If that was a `tmpfs' filesystem then
> they would all sit in RAM, unless you've swap space and pressure caused
> pages from the filesystem to be swapped out.

> You could use get_iplayer's --command option to run a command to move
> each final file off tmpfs as the download is finished.  Its --output
> affects all the intermediate files too, AIUI.

Yes, Linux. The challenge for me is to work out how to get the fetched file
to go onto the tmpfs, then have that processed - ideally having gip give
ffmpeg relevant details - into a 25fps file on my main disc. i.e. I don't
know enough to specify the commands and either get gip to do this, or add
in the relevant ffmpeg call.

Am I right that by using the --command and also --raw I can then run ffmpeg
to do the conversion and put the result where I want? That sounds about
right for what I have in mind. 

I'll look at what the --verbose gives me as per your other email. Someone
elsewhere has said they can get handbrake to do 50 -> 25 fps fairly well.
But that means running that as well unless I can work out what it is doing
on a command level. And IIUC handrake is using ffmpeg anyway under the
hood...

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From sjcarr at gmail.com  Tue May  1 07:53:08 2018
From: sjcarr at gmail.com (Steven Carr)
Date: Tue, 1 May 2018 16:53:08 +0200
Subject: no more hslv format ?
In-Reply-To: <56f15bc592web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
Message-ID: <CALMep07qQYUu+MXkui-aSNpna5zN6oSmCgTapBGw0OAA=LRQtg@mail.gmail.com>

On 1 May 2018 at 14:05, Jim web <web at audiomisc.co.uk> wrote:
> I'll look at what the --verbose gives me as per your other email. Someone
> elsewhere has said they can get handbrake to do 50 -> 25 fps fairly well.
> But that means running that as well unless I can work out what it is doing
> on a command level. And IIUC handrake is using ffmpeg anyway under the
> hood...

I've just done something similar.

I get get_iplayer to download the files (tvbest with 50fps) and then
dump them into a folder, I then have another script that's going
through those files in that folder and converting them to x265 and
reducing the fps to 25. No issues so far, and results in a filesize of
around 350MB/per hour of TV.

It's using HandbrakeCLI (and fwiw Handbrake doesn't always use ffmpeg,
it also has other encoders built in), I'm encoding to x265 for the
smaller filesize but you can tell it to use x264 instead.

Only downside is the x265 encoding takes a while, pretty much double
the length of the video time on my system, about 1hr45 encoding for a
50-60 minute video, but hey ho, it works, and it's running in the
background via cron so it will spit out the videos when done - it's
not like I was in a hurry to watch them anyway...

For anyone interested the script is here...
https://pastebin.com/cGSxbSPK


From richard22j at zoho.com  Wed May  2 03:50:58 2018
From: richard22j at zoho.com (RS)
Date: Wed, 2 May 2018 11:50:58 +0100
Subject: no more hslv format ?
In-Reply-To: <56f0dca351web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
Message-ID: <60733938-d7ea-4db5-bd1d-2e6f07d25cab@zoho.com>



On 30/04/18 13:56, Jim web wrote:
> I've been discussing the 'loss' of the 1280x720 25fps version with someone
> at the BBC.
> 
> IIUC this stemmed from 'Red Bee' days of yore, and until recently people at
> the BBC had thought they had stopped it long ago. Someone apparently
> noticed recently that it was still available. And then actually disabled
> it. Hence the mysterious recent ending of its availablity.
> 
> I have made the point that the 1280x720 25fps version is useful for people
> with a content 'cap' problem, etc, so will be missed because the 50fps
> version means somewhat bigger files and/or stream rates. But I doubt this
> will cause a rethink.
> 
We have known for some time that the BBC was going to stop using Flash 
at some stage.  Dinky anticipated that change and get_iplayer no longer 
supports Flash.  The last version with Flash capability was v2.99.

HLS is a much more recent innovation than Flash.  get_iplayer now has 
its own built-in downloader which downloads HLS 2 or 3 times as fast as 
Flash.  HLS has been referred to as a legacy mode so it was always 
likely that it would be removed eventually.  The surprise was that HLS 
was removed at the same time as Flash.

It is unrealistic to expect the BBC to restore HLS or Flash.

I have not seen any comments here which suggest that anyone is unhappy 
with HVF.  get_iplayer uses the same built-in downloader as for HLS. 
The BBC's full list of HVF modes is set out in a table in this document.

https://www.bbc.co.uk/rd/blog/2015-07-the-development-of-new-video-factory-profiles-for-bbc-iplayer

What I think a number of people here would like to see is the addition 
of one more High H.264 profile mode to that list, namely 1280x720p 25fps.

The BBC refers to viewing tests it has carried out, "using a range of 
content and included clips, from popular shows such as Strictly Come 
Dancing, East Enders and Top Gear."

It concludes, "At around 3 Mbit/s a 960x540 profile at 50fps will be 
made available to Connected TVs and set top boxes. The Elemental 
encoders are used to convert the 1920x1080 interlaced content to 960x540 
for progressive encoding at 50fps. Although the 960x540 profile has a 
reduced spatial resolution compared to the outgoing 1280x720 at 25fps, 
subjective assessments shows it delivers significantly better pictures 
on TV screens across a wide range of popular content (such as EastEnders 
and Top Gear) due to its higher frame rate. The 50fps, 1280x720 profile, 
however, will be available to those with 5Mbit/s broadband connections.

"Additional lower bit-rate profiles will be made available to computers 
on wifi. This enables video playback to continue, even when an 
individual's available bit-rate is reduced by users sharing a 
connection. These profiles will also cater for similar bit-rate 
restrictions on public wifi connections."

I accept that there is a trade off between resolution and frame rate. 
With differential encoding schemes like H.264, at a higher frame rate 
the changes between frames are smaller, so there is less to encode. 
Even so, for the uncompressed video the resolution at 50fps would need 
to be reduced to 905x510p to give the same bit rate as 1280x720p at 25fps.

For that reason I find it surprising that 960x540p at 50fps "delivers 
significantly better pictures on TV screens across a wide range of 
popular content (such as EastEnders and Top Gear) due to its higher 
frame rate" than 1280x720p at 25fps but the BBC has done the viewing 
tests and I have not.

More importantly BBC Four and to a lesser extent BBC 2 have a lot of 
programmes about paintings, sculpture, architecture and nature where 
there is a lot of fine detail and little motion.  Intuitively such 
programmes would not benefit from the higher frame rate but would 
benefit from the higher resolution.  There is no mention in the blog 
that such programmes were included in the viewing tests, and they ought 
to have been.

Best wishes
Richard





From c.e.macfarlane at macfh.co.uk  Wed May  2 05:06:28 2018
From: c.e.macfarlane at macfh.co.uk (MacFH - C E Macfarlane)
Date: Wed, 2 May 2018 13:06:28 +0100
Subject: no more hslv format ?
In-Reply-To: <60733938-d7ea-4db5-bd1d-2e6f07d25cab@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <60733938-d7ea-4db5-bd1d-2e6f07d25cab@zoho.com>
Message-ID: <9288ffa5-010a-9676-1457-32d7f6658c34@macfh.co.uk>

Please see below ...

On 02/05/2018 11:50, RS wrote:
>
> On 30/04/18 13:56, Jim web wrote:
>> I've been discussing the 'loss' of the 1280x720 25fps version with 
>> someone
>> at the BBC.
>>
>> IIUC this stemmed from 'Red Bee' days of yore, and until recently 
>> people at
>> the BBC had thought they had stopped it long ago. Someone apparently
>> noticed recently that it was still available. And then actually disabled
>> it. Hence the mysterious recent ending of its availablity.
>>
>> I have made the point that the 1280x720 25fps version is useful for 
>> people
>> with a content 'cap' problem, etc, so will be missed because the 50fps
>> version means somewhat bigger files and/or stream rates. But I doubt 
>> this
>> will cause a rethink.
>>
> We have known for some time that the BBC was going to stop using Flash 
> at some stage.? Dinky anticipated that change and get_iplayer no 
> longer supports Flash.? The last version with Flash capability was v2.99.
>
> HLS is a much more recent innovation than Flash.? get_iplayer now has 
> its own built-in downloader which downloads HLS 2 or 3 times as fast 
> as Flash.? HLS has been referred to as a legacy mode so it was always 
> likely that it would be removed eventually.? The surprise was that HLS 
> was removed at the same time as Flash.
>
> It is unrealistic to expect the BBC to restore HLS or Flash.

Yes, it' s unrealistic to expect the BBC to respond to any criticism 
whatsoever, because they never have in the past whenever changes they've 
introduced have broken other people's kit such as Network Media Players 
and Smart TVs.

> I have not seen any comments here which suggest that anyone is unhappy 
> with HVF.? get_iplayer uses the same built-in downloader as for HLS. 
> The BBC's full list of HVF modes is set out in a table in this document.
>
> https://www.bbc.co.uk/rd/blog/2015-07-the-development-of-new-video-factory-profiles-for-bbc-iplayer 
>
>
> What I think a number of people here would like to see is the addition 
> of one more High H.264 profile mode to that list, namely 1280x720p 25fps.

Not *necessarily*? -? I think the problem is more subtle than that. See 
below ...

> The BBC refers to viewing tests it has carried out, "using a range of 
> content and included clips, from popular shows such as Strictly Come 
> Dancing, East Enders and Top Gear."
>
> It concludes, "At around 3 Mbit/s a 960x540 profile at 50fps will be 
> made available to Connected TVs and set top boxes. The Elemental 
> encoders are used to convert the 1920x1080 interlaced content to 
> 960x540 for progressive encoding at 50fps. Although the 960x540 
> profile has a reduced spatial resolution compared to the outgoing 
> 1280x720 at 25fps, subjective assessments shows it delivers 
> significantly better pictures on TV screens across a wide range of 
> popular content (such as EastEnders and Top Gear) due to its higher 
> frame rate. The 50fps, 1280x720 profile, however, will be available to 
> those with 5Mbit/s broadband connections.

That's what I see as the problem, not a single natural history or art or 
historical documentary mentioned in the testing mix.? For such 
programmes, I would suggest that often an increase in spatial resolution 
would be preferable to an increase in temporal resolution.? I don't care 
how popular the above programmes are, I've not watched one of them in 
more than 20 years, and the other two ever.? The BBC is a public service 
broadcaster, and shouldn't only be thinking about what is best for 
populist programming.? They should be doing what is best for the content 
of a particular programme, or if that is too complicated and difficult, 
then, yes, let us have the choice of a 1280x720x25 profile, and let us 
choose which to download.

> I accept that there is a trade off between resolution and frame rate. 
> With differential encoding schemes like H.264, at a higher frame rate 
> the changes between frames are smaller, so there is less to encode. 
> Even so, for the uncompressed video the resolution at 50fps would need 
> to be reduced to 905x510p to give the same bit rate as 1280x720p at 
> 25fps.
>
> For that reason I find it surprising that 960x540p at 50fps "delivers 
> significantly better pictures on TV screens across a wide range of 
> popular content (such as EastEnders and Top Gear) due to its higher 
> frame rate" than 1280x720p at 25fps but the BBC has done the viewing 
> tests and I have not.
>
> More importantly BBC Four and to a lesser extent BBC 2 have a lot of 
> programmes about paintings, sculpture, architecture and nature where 
> there is a lot of fine detail and little motion. Intuitively such 
> programmes would not benefit from the higher frame rate but would 
> benefit from the higher resolution.? There is no mention in the blog 
> that such programmes were included in the viewing tests, and they 
> ought to have been.

Yes, agreed (I wrote the above before I'd read your post entirely, which 
was probably not best practice, for which apologies, but I'm in 
something of a hurry).

But there is also another elephant in the room, which is the capacity of 
their server system to deliver whatever they choose to do.? There's 
little point in increasing either spatial or temporal resolution if the 
resulting download will be compressed to hell on the way out of the 
servers.? I've seen even HD programmes break up into squares, possibly 
because the system can't cope with the throughput at times of peak 
demand, though I ought to point out in fairness that on at least one 
occasion the squares seem to be in the source programme, because 
downloading it a second time gave an identical result!

Regards.


From web at audiomisc.co.uk  Wed May  2 05:16:18 2018
From: web at audiomisc.co.uk (Jim web)
Date: Wed, 02 May 2018 13:16:18 +0100
Subject: no more hslv format ?
In-Reply-To: <60733938-d7ea-4db5-bd1d-2e6f07d25cab@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <60733938-d7ea-4db5-bd1d-2e6f07d25cab@zoho.com>
Message-ID: <56f1e0a122web@audiomisc.co.uk>

In article <60733938-d7ea-4db5-bd1d-2e6f07d25cab at zoho.com>, RS
<richard22j at zoho.com> wrote:


> On 30/04/18 13:56, Jim web wrote:
> > I've been discussing the 'loss' of the 1280x720 25fps version with
> > someone at the BBC.
> > 
> > IIUC this stemmed from 'Red Bee' days of yore, and until recently
> > people at the BBC had thought they had stopped it long ago. Someone
> > apparently noticed recently that it was still available. And then
> > actually disabled it. Hence the mysterious recent ending of its
> > availablity.
> > 
> > I have made the point that the 1280x720 25fps version is useful for
> > people with a content 'cap' problem, etc, so will be missed because
> > the 50fps version means somewhat bigger files and/or stream rates. But
> > I doubt this will cause a rethink.
> > 


> HLS is a much more recent innovation than Flash.  get_iplayer now has
> its own built-in downloader which downloads HLS 2 or 3 times as fast as
> Flash.  HLS has been referred to as a legacy mode so it was always
> likely that it would be removed eventually.  The surprise was that HLS
> was removed at the same time as Flash.

> It is unrealistic to expect the BBC to restore HLS or Flash.

Understood. I've probably been conflating different issues

[snip BBC's comments]


> For that reason I find it surprising that 960x540p at 50fps "delivers
> significantly better pictures on TV screens across a wide range of
> popular content (such as EastEnders and Top Gear) due to its higher
> frame rate" than 1280x720p at 25fps but the BBC has done the viewing
> tests and I have not.

I'm also puzzled by this conclusion by the BBC. But I haven't viewed more
than a few of the 960x540 examples, so my findings may not be typical.

> More importantly BBC Four and to a lesser extent BBC 2 have a lot of
> programmes about paintings, sculpture, architecture and nature where
> there is a lot of fine detail and little motion.  Intuitively such
> programmes would not benefit from the higher frame rate but would
> benefit from the higher resolution.  There is no mention in the blog
> that such programmes were included in the viewing tests, and they ought
> to have been.

I'd agree. I'll see if I can pass forward some of these points. I need
to be able to contact someone further 'up the stack' to ask more
'policy' questions I guess. I know more about the radio side, so
I don't know how I'll get on with the tv side.

[edit] Just sent an email making some of the points. I also added
a comment of my own that I'd *much* prefer "Sky at Night" as the
1280x720 25fps than a lower resolution. Time will tell if I manage
to get someone 'higher up' to be willing to discuss this with me.

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From ralph at inputplus.co.uk  Wed May  2 08:20:15 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Wed, 02 May 2018 16:20:15 +0100
Subject: no more hslv format ?
In-Reply-To: <56f15bc592web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
Message-ID: <20180502152015.50EA621CC3@orac.inputplus.co.uk>

Hi Jim,

> > You could use get_iplayer's --command option to run a command to
> > move each final file off tmpfs as the download is finished.  Its
> > --output affects all the intermediate files too, AIUI.
>
> The challenge for me is to work out how to get the fetched file to go
> onto the tmpfs

Well, `df -t tmpfs' will probably show /tmp is a tmpfs so you could
`--output /tmp' and you should see its intermediate files, and the final
file, only appear there.  `--command' could then move that final file to
the SSD, or run a conversion command that writes to the SSD and then
removes the tmpfs input.

As for altering the ffmpeg command that get_iplayer is using, I'm not
sure that's worthwhile?  It isn't doing any transcoding, just changing
the container format, or splicing in better audio, that kind of thing.
So your `lossy' slow-down ffmpeg from 50 fps to 25 fps won't be a second
lossy one that you'd prefer to combine with the first.  I could be
wrong, not knowing how to have ffmpeg do this conversion.  When you find
out, let the list know.  :-)

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From owen.smith at cantab.net  Wed May  2 11:07:01 2018
From: owen.smith at cantab.net (Owen Smith)
Date: Wed, 2 May 2018 19:07:01 +0100
Subject: no more hslv format ?
In-Reply-To: <20180502152015.50EA621CC3@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
Message-ID: <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>

What do you mean this isn't a lossy transcoding? How can ffmpeg go from 50fps to 25fps without losing anything? The frames are not all complete frames, software can't just throw alternate frames away. Well it could,  but the only way to do that is a full H.264 decode, then discard alternate frames, then a full H.264 encode again which is going to involve loss.

Most frames are not fully present in the original stream, they are interpolated from previous and subsequent frames. You can't throw any of those away, because other frames are interpolated from them. It would need to be a very special original encode which had all even frames only interpolated from other even frames and ditto for odd frames to allow alternate frames to be discarded. And a special encode like that would bloat the file size substantially, almost doubling it I would expect.

-- 
Owen Smith <owen.smith at cantab.net>
Cambridge, UK

> On 2 May 2018, at 16:20, Ralph Corderoy <ralph at inputplus.co.uk> wrote:
> 
> Hi Jim,
> 
>>> You could use get_iplayer's --command option to run a command to
>>> move each final file off tmpfs as the download is finished.  Its
>>> --output affects all the intermediate files too, AIUI.
>> 
>> The challenge for me is to work out how to get the fetched file to go
>> onto the tmpfs
> 
> Well, `df -t tmpfs' will probably show /tmp is a tmpfs so you could
> `--output /tmp' and you should see its intermediate files, and the final
> file, only appear there.  `--command' could then move that final file to
> the SSD, or run a conversion command that writes to the SSD and then
> removes the tmpfs input.
> 
> As for altering the ffmpeg command that get_iplayer is using, I'm not
> sure that's worthwhile?  It isn't doing any transcoding, just changing
> the container format, or splicing in better audio, that kind of thing.
> So your `lossy' slow-down ffmpeg from 50 fps to 25 fps won't be a second
> lossy one that you'd prefer to combine with the first.  I could be
> wrong, not knowing how to have ffmpeg do this conversion.  When you find
> out, let the list know.  :-)
> 
> -- 
> Cheers, Ralph.
> https://plus.google.com/+RalphCorderoy
> 
> _______________________________________________
> get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer



From peter.kirk at isauk.biz  Wed May  2 11:43:48 2018
From: peter.kirk at isauk.biz (Peter S Kirk)
Date: Wed, 02 May 2018 19:43:48 +0100
Subject: no more hslv format ?
In-Reply-To: <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>,
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>,
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
Message-ID: <5AEA14F4.27698.502F38@peter.kirk.isauk.biz>

Many posts back it was mentioned they are not true 50fps, instead each 
frame from a 25fps is duplicated merely to allow BBC to boast about 50fps 
streaming.

On 2 May 2018 at 19:07, Owen Smith Owen Smith <owen.smith at cantab.net> 
wrote:

> What do you mean this isn't a lossy transcoding? How can ffmpeg go from 50fps to 25fps without losing anything? The
> frames are not all complete frames, software can't just throw alternate frames away. Well it could,  but the only way
> to do that is a full H.264 decode, then discard alternate frames, then a full H.264 encode again which is going to
> involve loss.
> 
> Most frames are not fully present in the original stream, they are interpolated from previous and subsequent frames.
> You can't throw any of those away, because other frames are interpolated from them. It would need to be a very special
> original encode which had all even frames only interpolated from other even frames and ditto for odd frames to allow
> alternate frames to be discarded. And a special encode like that would bloat the file size substantially, almost
> doubling it I would expect.




From owen.smith at cantab.net  Wed May  2 12:11:13 2018
From: owen.smith at cantab.net (Owen Smith)
Date: Wed, 2 May 2018 20:11:13 +0100
Subject: no more hslv format ?
In-Reply-To: <5AEA14F4.27698.502F38@peter.kirk.isauk.biz>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <5AEA14F4.27698.502F38@peter.kirk.isauk.biz>
Message-ID: <5AF75F74-2AD3-49B8-A16B-3ACA1580D293@cantab.net>

That doesn't make any difference to the H.264 encoding. At best if you get lucky every odd frame will be marked as being interpolated from the preceding even frame and then no actual changes within the frame. But all it takes is for the encoder to slip by a frame somewhere or to encode an I frame (a complete standalone frame) in the even or odd half that you are trying to discard every one of and then you are screwed.

And if the 50fps file genuinely does have every other frame saying "I'm identical to the previous frame" the stripping these won't save much on the file size. The fact that people complain the 50fps files are double the size of the same resolution 25fps files implies to me this isn't how it has been done. Even if the BBC have only created motion interpolated frames from a 25fps source, they are still encoded in the video stream and you still can't casually toss alternate frames without screwing up the I-P-B frame interpolation.

-- 
Owen Smith <owen.smith at cantab.net>
Cambridge, UK

> On 2 May 2018, at 19:43, Peter S Kirk <peter.kirk at isauk.biz> wrote:
> 
> Many posts back it was mentioned they are not true 50fps, instead each 
> frame from a 25fps is duplicated merely to allow BBC to boast about 50fps 
> streaming.
> 
> On 2 May 2018 at 19:07, Owen Smith Owen Smith <owen.smith at cantab.net> 
> wrote:
> 
>> What do you mean this isn't a lossy transcoding? How can ffmpeg go from 50fps to 25fps without losing anything? The
>> frames are not all complete frames, software can't just throw alternate frames away. Well it could,  but the only way
>> to do that is a full H.264 decode, then discard alternate frames, then a full H.264 encode again which is going to
>> involve loss.
>> 
>> Most frames are not fully present in the original stream, they are interpolated from previous and subsequent frames.
>> You can't throw any of those away, because other frames are interpolated from them. It would need to be a very special
>> original encode which had all even frames only interpolated from other even frames and ditto for odd frames to allow
>> alternate frames to be discarded. And a special encode like that would bloat the file size substantially, almost
>> doubling it I would expect.
> 
> 
> 
> _______________________________________________
> get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer



From ralph at inputplus.co.uk  Wed May  2 14:33:23 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Wed, 02 May 2018 22:33:23 +0100
Subject: no more hslv format ?
In-Reply-To: <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
Message-ID: <20180502213323.48FDA21CC3@orac.inputplus.co.uk>

Hi Owen,

> What do you mean this isn't a lossy transcoding?

Is that aimed at me?

Perhaps if you didn't top post, and instead wrote that under a quote of
mine I'd know to which bit of the two ffmpeg invocations you were
referring!  :-)

> How can ffmpeg go from 50fps to 25fps without losing anything?

I don't think it can, and didn't suggest it could.  It is lossy.  I said
the first, default, one wasn't, and that therefore it wasn't worth
combining this extra, 50->25, one with it, as you would want to if both
were lossy.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From owen.smith at cantab.net  Wed May  2 15:21:27 2018
From: owen.smith at cantab.net (Owen Smith)
Date: Wed, 2 May 2018 23:21:27 +0100
Subject: no more hslv format ?
In-Reply-To: <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
Message-ID: <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>

I'm replying on an iPad, which makes large scale editing of email replies a huge amount of work. So I top post, because it takes only a tenth of the time. If Apple made it only cost me double the amount of time to reply properly I'd do it.

I've been mystified for a while why people talked about "dropping every other frame" as if it were trivial to do, and an email earlier in this chain looked like someone was trying to do that again. I was explaining why that simply is not possible in the general case.

-- 
Owen Smith <owen.smith at cantab.net>
Cambridge, UK

> On 2 May 2018, at 22:33, Ralph Corderoy <ralph at inputplus.co.uk> wrote:
> 
> Hi Owen,
> 
>> What do you mean this isn't a lossy transcoding?
> 
> Is that aimed at me?
> 
> Perhaps if you didn't top post, and instead wrote that under a quote of
> mine I'd know to which bit of the two ffmpeg invocations you were
> referring!  :-)
>> How can ffmpeg go from 50fps to 25fps without losing anything?
> 
> I don't think it can, and didn't suggest it could.  It is lossy.  I said
> the first, default, one wasn't, and that therefore it wasn't worth
> combining this extra, 50->25, one with it, as you would want to if both
> were lossy.
> 
> -- 
> Cheers, Ralph.
> https://plus.google.com/+RalphCorderoy
> 
> _______________________________________________
> get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer



From richard22j at zoho.com  Wed May  2 16:24:48 2018
From: richard22j at zoho.com (RS)
Date: Thu, 3 May 2018 00:24:48 +0100
Subject: no more hslv format ?
In-Reply-To: <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
Message-ID: <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>

On 02/05/18 23:21, Owen Smith wrote:

> 
> I've been mystified for a while why people talked about "dropping every other frame" as if it were trivial to do, and an email earlier in this chain looked like someone was trying to do that again. I was explaining why that simply is not possible in the general case.
> 

If I have caused confusion by talking about dropping alternate frames I 
apologise.  I had come across some posts in another forum which 
suggested it could be done, but I now recognise I was wrong.  What makes 
it worse is that I have since come across a thread in this listserver 
from two years ago where I was asking exactly the same questions, and 
Vangelis pointed out I was wrong and directed me to a Wikipedia article 
on H.264.

It is not possible to change the frame rate using -c:v=copy in ffmpeg; 
it is necessary to re-encode which is why it takes so long.  Inevitably 
there will be losses, added to which the codecs available to us may be 
inferior to those used by the BBC.

I think it was Nick Payne who said he had experimented with re-encoding 
in HEVC (H.265) and found that the file size was the same for 25fps as 
it was for 50fps, which led him to conclude that frames were being 
duplicated to achieve 50fps.

The broadcast signal (at least on satellite) is 1920x1080i at 25fps 
usually with two audio streams, AC3 and NAR.  If the broadcast signal is 
recorded, the resultant file size is about 3GByte/h.

The BBC'S explanation of what it does with the broadcast signal is, "The 
Elemental encoders are used to convert the 1920x1080 interlaced content 
to 960x540 for progressive encoding at 50fps."

It also says, "The 50fps, 1280x720 profile, however, will be available 
to those with 5Mbit/s broadband connections." but it does not explain 
where it comes from or why it cannot generate a 1280x720p 25fps profile.

Best wishes
Richard



From njhp94 at gmail.com  Wed May  2 17:04:47 2018
From: njhp94 at gmail.com (Nick Payne)
Date: Thu, 3 May 2018 10:04:47 +1000
Subject: no more hslv format ?
In-Reply-To: <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
Message-ID: <886de22c-6702-04ed-aba1-7c5526b79f8d@gmail.com>

On 3/05/2018 9:24 AM, RS wrote:
> On 02/05/18 23:21, Owen Smith wrote:
>
>>
>> I've been mystified for a while why people talked about "dropping
>> every other frame" as if it were trivial to do, and an email earlier
>> in this chain looked like someone was trying to do that again. I was
>> explaining why that simply is not possible in the general case.
>>
>
> If I have caused confusion by talking about dropping alternate frames
> I apologise.? I had come across some posts in another forum which
> suggested it could be done, but I now recognise I was wrong.? What
> makes it worse is that I have since come across a thread in this
> listserver from two years ago where I was asking exactly the same
> questions, and Vangelis pointed out I was wrong and directed me to a
> Wikipedia article on H.264.
>
> It is not possible to change the frame rate using -c:v=copy in ffmpeg;
> it is necessary to re-encode which is why it takes so long.?
> Inevitably there will be losses, added to which the codecs available
> to us may be inferior to those used by the BBC.
>
> I think it was Nick Payne who said he had experimented with
> re-encoding in HEVC (H.265) and found that the file size was the same
> for 25fps as it was for 50fps, which led him to conclude that frames
> were being duplicated to achieve 50fps.

Yes, I had some of the coverage of the UK snooker championships where
they provided HLS downloads @ 1280x720 25fps and other coverage was only
available as HVF downloads @ 1280x720 50fps. When I ran these both
through Handbrake with identical settings to convert to HEVC, retaining
the frame rate of the downloaded files, the size reduction for the 50fps
downloads was about twice that for the 25fps downloads, and the size of
the files output by Handbrake was proportional to the length of the
program and not the frame rate. e.g.

4h58m match: D/L size 5.03Gb @ 25fps, output from Handbrake was 1.55Gb
4h44m match: D/L size 10.2Gb @ 50fps, output from Handbrake was 1.48Gb

Nick



From chrisjbrady at gmail.com  Thu May  3 01:28:41 2018
From: chrisjbrady at gmail.com (CJB)
Date: Thu, 3 May 2018 09:28:41 +0100
Subject: All OK
Message-ID: <CAPZzCnP+Vu4jPQa2v-B=-XZXtuqATZ0viXso0Z232z8TzLSPHw@mail.gmail.com>

Just refreshed radio and tv caches using the latest GiP. I thought
that May 1 was some kind of cut-off date? Chris B.


From web at audiomisc.co.uk  Wed May  2 08:49:12 2018
From: web at audiomisc.co.uk (Jim web)
Date: Wed, 02 May 2018 16:49:12 +0100
Subject: no more hslv format ?
In-Reply-To: <20180502152015.50EA621CC3@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
Message-ID: <56f1f41ee0web@audiomisc.co.uk>

In article <20180502152015.50EA621CC3 at orac.inputplus.co.uk>, Ralph
Corderoy <ralph at inputplus.co.uk> wrote:
> >
> > The challenge for me is to work out how to get the fetched file to go
> > onto the tmpfs

> Well, `df -t tmpfs' will probably show /tmp is a tmpfs so you could
> `--output /tmp' and you should see its intermediate files, and the final
> file, only appear there.  `--command' could then move that final file to
> the SSD, or run a conversion command that writes to the SSD and then
> removes the tmpfs input.

Thanks. :-)

> As for altering the ffmpeg command that get_iplayer is using, I'm not
> sure that's worthwhile?  It isn't doing any transcoding, just changing
> the container format, or splicing in better audio, that kind of thing.
> So your `lossy' slow-down ffmpeg from 50 fps to 25 fps won't be a second
> lossy one that you'd prefer to combine with the first.  I could be
> wrong, not knowing how to have ffmpeg do this conversion.  When you find
> out, let the list know.  :-)

My concern is that I only have a total of 8GB of ram at present on the
machine, and the 1280x720 50fps files tend to come in at 2GB or more per
hour. 

To save time and avoid running past 9am (and entering metered time) it
would be quickest to do all the fetches first. But this will mean more
files that I can store on ram on some days. So the wish would be to then
shift or process material and put it on hd.

As things are, I could then have gip+ffmpeg generate the 50 fps mp4 files
(i.e. in the current form) onto hd. This saves some hd wear but means I now
have those (big) files on hd before I then run a 50 -> 25 fps process to
create smaller ones (which will be akin in size to what I got from the
25fps fetching). The result using this approach means I've still done the
fetching as quickly as possible, but have about three times as much data
written to hd at some point.

It would be nice to do the process file by file and in each case only the
'final' 25 fps version gets written to hd. This means the same hd use as
previously. But that either means I need enough ram to hold all the 'temp'
files (masses of ram needed) or doing the process item by item. (slow)

I could use spinning rust I guess. but that seems messy. Otherwise it seems
to want me to buy and fit a lot of ram. :-)

So at present I'm wondering and experimenting. 8-]

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From web at audiomisc.co.uk  Thu May  3 02:23:30 2018
From: web at audiomisc.co.uk (Jim web)
Date: Thu, 03 May 2018 10:23:30 +0100
Subject: no more hslv format ?
In-Reply-To: <20180502152015.50EA621CC3@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
Message-ID: <56f254a551web@audiomisc.co.uk>

In article <20180502152015.50EA621CC3 at orac.inputplus.co.uk>, Ralph
Corderoy <ralph at inputplus.co.uk> wrote:
> > The challenge for me is to work out how to get the fetched file to go
> > onto the tmpfs

> Well, `df -t tmpfs' will probably show /tmp is a tmpfs so you could
> `--output /tmp' and you should see its intermediate files, and the final
> file, only appear there.  `--command' could then move that final file to
> the SSD, or run a conversion command that writes to the SSD and then
> removes the tmpfs input.

the df command listed various 'locations'. These included

tmpfs  815696  1352  814334  /run

none 4078460  0  4078460  /run/shm

none 102400  8  102392  /run/usr

Inside /run/usr I did find a directory for my user id and could use that as
the user. However I don't know what the /run/shm is for, so need to do some
finding out...

I did set up an explict ramdisc on another machine ages ago by adding a
line to the /etc/fstab file. But that fixed the size to 256 MB, which in
this context is tiny. System monitor says I have 8GB of ram.

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From ralph at inputplus.co.uk  Thu May  3 03:35:16 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Thu, 03 May 2018 11:35:16 +0100
Subject: no more hslv format ?
In-Reply-To: <886de22c-6702-04ed-aba1-7c5526b79f8d@gmail.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <886de22c-6702-04ed-aba1-7c5526b79f8d@gmail.com>
Message-ID: <20180503103516.89E7A2126A@orac.inputplus.co.uk>

Hi Nick,

> I had some of the coverage of the UK snooker championships
...
> 4h58m match: D/L size 5.03Gb @ 25fps, output from Handbrake was 1.55Gb
> 4h44m match: D/L size 10.2Gb @ 50fps, output from Handbrake was 1.48Gb

Perhaps many frames(!) of snooker coverage isn't ideal for this
comparison as it often is a static picture of the table.  :-)

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From ralph at inputplus.co.uk  Thu May  3 03:41:03 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Thu, 03 May 2018 11:41:03 +0100
Subject: no more hslv format ?
In-Reply-To: <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
Message-ID: <20180503104103.A38B42126A@orac.inputplus.co.uk>

Hi Richard,

> I think it was Nick Payne who said he had experimented with
> re-encoding in HEVC (H.265) and found that the file size was the same
> for 25fps as it was for 50fps, which led him to conclude that frames
> were being duplicated to achieve 50fps.

ffmpeg can produce PNGs, one per frame, and convert only a few specific
seconds to avoid tens of thousands of them, and then adjacent frames
could be compared in pairs to see if they are indeed identical;  compare
`ab', `bc', `cd'...

There's a couple of variants for comparing images.

    compare foo.png bar.png x:
    gm compare -highlight-style assign foo.png bar.png -file x:

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From steved424 at gmail.com  Thu May  3 04:18:59 2018
From: steved424 at gmail.com (Steve Dodd)
Date: Thu, 3 May 2018 12:18:59 +0100
Subject: no more hslv format ?
In-Reply-To: <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
Message-ID: <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>

On Thu, May 3, 2018 at 12:24 AM, RS <richard22j at zoho.com> wrote:

> I think it was Nick Payne who said he had experimented with re-encoding in HEVC (H.265) and found that the file size was the same for 25fps as it was for 50fps, which led him to conclude that frames were being duplicated to achieve 50fps.

Is it possible it depends on the source material? From the BBC quote
earlier it sounds like their "source" material is still mostly HD
interlaced, but perhaps some of their sources are all also
progressive, and those ones get doubled identical frames (which might
be a logical result of feeding 25p material into something that's
designed to interpolate interlaced fields into frames)?

S.


From ilainftw at gmail.com  Thu May  3 05:35:02 2018
From: ilainftw at gmail.com (iz)
Date: Thu, 3 May 2018 13:35:02 +0100
Subject: no more hslv format ?
In-Reply-To: <56f1f41ee0web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <56f1f41ee0web@audiomisc.co.uk>
Message-ID: <CAF_ssT_0x-8CB3cxZuxMURtHT4fmADD1ETnPEq3NLoLwMtBtTw@mail.gmail.com>

On 2 May 2018 at 16:49, Jim web <web at audiomisc.co.uk> wrote:
>
> As things are, I could then have gip+ffmpeg generate the 50 fps mp4 files
> (i.e. in the current form) onto hd. This saves some hd wear but means I now
> have those (big) files on hd before I then run a 50 -> 25 fps process to
> create smaller ones (which will be akin in size to what I got from the
> 25fps fetching). The result using this approach means I've still done the
> fetching as quickly as possible, but have about three times as much data
> written to hd at some point.

You may be able to limit the whole process to only two file writes
with a little scripting and XML parsing. Use --raw with GiP  to save
the .ts file directly to drive or to drive via ram disk. The ram disk
probably wouldn't be of much use since I doubt your downloads are
limited by disk write speed. Also use --metadata to save programme
metadata in an XML file and --thumbnail to save the cover art in a JPG
file. Write a script to re-encode the file to 25fps MP4 and at the
same time add metadata tags with ffmpeg, using the XML file and JPG
file as input to construction of the ffmpeg command string. You'll
need ffmpeg 4.0 or higher to add cover art to MP4.


From richard22j at zoho.com  Thu May  3 13:02:54 2018
From: richard22j at zoho.com (RS)
Date: Thu, 3 May 2018 21:02:54 +0100
Subject: no more hslv format ?
In-Reply-To: <20180503104103.A38B42126A@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <20180503104103.A38B42126A@orac.inputplus.co.uk>
Message-ID: <8c8d1bd9-1265-7838-1339-0087cf1d64e4@zoho.com>

On 03/05/18 11:41, Ralph Corderoy wrote:
> 
> ffmpeg can produce PNGs, one per frame, and convert only a few specific
> seconds to avoid tens of thousands of them, and then adjacent frames
> could be compared in pairs to see if they are indeed identical;  compare
> `ab', `bc', `cd'...
> 
> There's a couple of variants for comparing images.
> 
>      compare foo.png bar.png x:
>      gm compare -highlight-style assign foo.png bar.png -file x:
> 
Hi Ralph

Would you first need to convert the H.264 or H.265 to raw video?  If one 
PNG is of an I-frame and the next is a P-frame or B-frame they are bound 
to be different.

Best wishes
Richard



From richard22j at zoho.com  Thu May  3 13:37:36 2018
From: richard22j at zoho.com (RS)
Date: Thu, 3 May 2018 21:37:36 +0100
Subject: no more hslv format ?
In-Reply-To: <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>
Message-ID: <923678f6-f950-1ec8-cbf4-565a3719c621@zoho.com>

On 03/05/18 12:18, Steve Dodd wrote:

> Is it possible it depends on the source material? From the BBC quote
> earlier it sounds like their "source" material is still mostly HD
> interlaced, but perhaps some of their sources are all also
> progressive, and those ones get doubled identical frames (which might
> be a logical result of feeding 25p material into something that's
> designed to interpolate interlaced fields into frames)?
> 
The satellite broadcasts I have seen from the BBC and other broadcasters 
are 1920x1080i25 for HD and 720x576i25 or 704x576i25 for SD.  Everything 
that is broadcast is in an interlaced format at some stage, however it 
is generated.  Some programmes are only produced for the iPlayer, so 
they may be generated in a different way.

Best wishes
Richard




From web at audiomisc.co.uk  Thu May  3 06:17:54 2018
From: web at audiomisc.co.uk (Jim web)
Date: Thu, 03 May 2018 14:17:54 +0100
Subject: no more hslv format ?
In-Reply-To: <CAF_ssT_0x-8CB3cxZuxMURtHT4fmADD1ETnPEq3NLoLwMtBtTw@mail.gmail.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <56f1f41ee0web@audiomisc.co.uk>
 <CAF_ssT_0x-8CB3cxZuxMURtHT4fmADD1ETnPEq3NLoLwMtBtTw@mail.gmail.com>
Message-ID: <56f26a1af3web@audiomisc.co.uk>

In article
<CAF_ssT_0x-8CB3cxZuxMURtHT4fmADD1ETnPEq3NLoLwMtBtTw at mail.gmail.com>, iz
<ilainftw at gmail.com> wrote:
> On 2 May 2018 at 16:49, Jim web <web at audiomisc.co.uk> wrote:
> >
> > As things are, I could then have gip+ffmpeg generate the 50 fps mp4
> > files (i.e. in the current form) onto hd. This saves some hd wear but
> > means I now have those (big) files on hd before I then run a 50 -> 25
> > fps process to create smaller ones (which will be akin in size to what
> > I got from the 25fps fetching). The result using this approach means
> > I've still done the fetching as quickly as possible, but have about
> > three times as much data written to hd at some point.

> You may be able to limit the whole process to only two file writes with
> a little scripting and XML parsing. 

FWIW as things stand I run the processes via a simple proglet I wrote in
'C'. (I use it as a ROX-Filer app, but it can also work via CLI.) I can
drag-and-drop a text file onto the program's icon. The file specifies a
'mode' and a pid per line, and the program then proceeds to chug though
them as I make breakfast. If needed, the file I give it could also say if
any particular reprocessing need be done 'now' or left to later, etc.

So I'd almost certainly do this via a 'C' approach because I've become used
to 'C' over the years. The trick is knowing what commands to launch, etc.

Modifying where output goes, making the output -raw, etc, is OK as I know
how to do that - once I've sorted out unknowns like if the -raw is going to
ramdisc or somewhere else, when reprocessing is done, etc.

Thus, yes, I am thinking it should be possible to do this via running 'one
program'. But...

Truth is, having finally allowed people to prise FORTRAN out of my cold
dead hands a decade or more ago, I'm now settled on 'C'. :-)

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From dave at lambley.me.uk  Fri May  4 15:50:00 2018
From: dave at lambley.me.uk (Dave Lambley)
Date: Fri, 4 May 2018 23:50:00 +0100
Subject: no more hslv format ?
In-Reply-To: <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
Message-ID: <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>

On 3 May 2018 at 00:24, RS <richard22j at zoho.com> wrote:
> On 02/05/18 23:21, Owen Smith wrote:
>
>>
>> I've been mystified for a while why people talked about "dropping every
>> other frame" as if it were trivial to do, and an email earlier in this chain
>> looked like someone was trying to do that again. I was explaining why that
>> simply is not possible in the general case.
>>
>
> If I have caused confusion by talking about dropping alternate frames I
> apologise.  I had come across some posts in another forum which suggested it
> could be done, but I now recognise I was wrong.  What makes it worse is that
> I have since come across a thread in this listserver from two years ago
> where I was asking exactly the same questions, and Vangelis pointed out I
> was wrong and directed me to a Wikipedia article on H.264.
>
> It is not possible to change the frame rate using -c:v=copy in ffmpeg; it is
> necessary to re-encode which is why it takes so long.  Inevitably there will
> be losses, added to which the codecs available to us may be inferior to
> those used by the BBC.
>
> I think it was Nick Payne who said he had experimented with re-encoding in
> HEVC (H.265) and found that the file size was the same for 25fps as it was
> for 50fps, which led him to conclude that frames were being duplicated to
> achieve 50fps.
>
> The broadcast signal (at least on satellite) is 1920x1080i at 25fps usually
> with two audio streams, AC3 and NAR.  If the broadcast signal is recorded,
> the resultant file size is about 3GByte/h.
>
> The BBC'S explanation of what it does with the broadcast signal is, "The
> Elemental encoders are used to convert the 1920x1080 interlaced content to
> 960x540 for progressive encoding at 50fps."

Searching turns up this,

http://dpp-assets.s3.amazonaws.com/wp-content/uploads/specs/bbc/TechnicalDeliveryStandardsBBCFile.pdf

... which tells you that 25fps interlaced is what programme makers
will be supplying for HD programmes.

> It also says, "The 50fps, 1280x720 profile, however, will be available to
> those with 5Mbit/s broadband connections." but it does not explain where it
> comes from or why it cannot generate a 1280x720p 25fps profile.

They would remove the interlace to bring the framerate to 50fps before
encoding. There are many methods of doing so, ffmpeg can do this for
example.

Dave


From richard22j at zoho.com  Fri May  4 16:29:51 2018
From: richard22j at zoho.com (RS)
Date: Sat, 5 May 2018 00:29:51 +0100
Subject: no more hslv format ?
In-Reply-To: <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>
Message-ID: <9f0b6f84-7e6a-e2f8-1d25-a2e7fde17510@zoho.com>

On 04/05/18 23:50, Dave Lambley wrote:

> 
> They would remove the interlace to bring the framerate to 50fps before
> encoding. There are many methods of doing so, ffmpeg can do this for
> example.
> 
This article

https://superuser.com/questions/253691/how-to-convert-108050i-72050p-using-ffmpeg

says, "Just like interlacing progressive content divides each frame into 
2 fields, thus doubling the frame rate, the common method of 
deinterlacing is to combine each 2 fields into 1 frame, which reduces 
the rate by 2, thus taking 50 into 25fps. You can of course double each 
final frame, but that does not provide any benefit."

Do you disagree?

Best wishes
Richard




From web at audiomisc.co.uk  Sat May  5 02:07:10 2018
From: web at audiomisc.co.uk (Jim web)
Date: Sat, 05 May 2018 10:07:10 +0100
Subject: no more hslv format ?
In-Reply-To: <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>
Message-ID: <56f35ad297web@audiomisc.co.uk>

In article
<CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow at mail.gmail.com>, Dave
Lambley <dave at lambley.me.uk> wrote:

> Searching turns up this,

> http://dpp-assets.s3.amazonaws.com/wp-content/uploads/specs/bbc/TechnicalDeliveryStandardsBBCFile.pdf

> ... which tells you that 25fps interlaced is what programme makers will
> be supplying for HD programmes.

Thats' quite interesting. It seems to confirm a feeling I had that the
input might often be 25fps. Is there a conflation or mix up between frames
and fields somewhere?

Jim

-- 
Electronics  https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/intro/electron.htm
Armstrong Audio  http://www.audiomisc.co.uk/Armstrong/armstrong.html
biog http://jcgl.orpheusweb.co.uk/history/ups_and_downs.html
Audio Misc  http://www.audiomisc.co.uk/index.html



From dave at lambley.me.uk  Sat May  5 16:38:46 2018
From: dave at lambley.me.uk (Dave Lambley)
Date: Sun, 6 May 2018 00:38:46 +0100
Subject: no more hslv format ?
In-Reply-To: <9f0b6f84-7e6a-e2f8-1d25-a2e7fde17510@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAPCK=o8mEqu3oE1KK5O3YOpsD=czK52k4-J-N6=og-SzM=qtow@mail.gmail.com>
 <9f0b6f84-7e6a-e2f8-1d25-a2e7fde17510@zoho.com>
Message-ID: <CAPCK=o9kVNoXtJX0eScCyfbBv+oTXNe0is4gnq2gj3-7NpTN7Q@mail.gmail.com>

On 5 May 2018 at 00:29, RS <richard22j at zoho.com> wrote:
> On 04/05/18 23:50, Dave Lambley wrote:
>
>>
>> They would remove the interlace to bring the framerate to 50fps before
>> encoding. There are many methods of doing so, ffmpeg can do this for
>> example.
>>
> This article
>
> https://superuser.com/questions/253691/how-to-convert-108050i-72050p-using-ffmpeg
>
> says, "Just like interlacing progressive content divides each frame into 2
> fields, thus doubling the frame rate, the common method of deinterlacing is
> to combine each 2 fields into 1 frame, which reduces the rate by 2, thus
> taking 50 into 25fps. You can of course double each final frame, but that
> does not provide any benefit."
>
> Do you disagree?

eg., https://ffmpeg.org/ffmpeg-filters.html#bwdif (and other) filters
can give you good quality 50fps frames, without duplication, from
interlaced 25fps video. I have no idea what the BBC use, but they have
plenty of choice.

Dave


From ralph at inputplus.co.uk  Sun May  6 10:02:13 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Sun, 06 May 2018 18:02:13 +0100
Subject: no more hslv format ?
In-Reply-To: <8c8d1bd9-1265-7838-1339-0087cf1d64e4@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <20180503104103.A38B42126A@orac.inputplus.co.uk>
 <8c8d1bd9-1265-7838-1339-0087cf1d64e4@zoho.com>
Message-ID: <20180506170213.EA90B21402@orac.inputplus.co.uk>

Hi Richard,

> > ffmpeg can produce PNGs, one per frame, and convert only a few
> > specific seconds to avoid tens of thousands of them
>
> Would you first need to convert the H.264 or H.265 to raw video?  If
> one PNG is of an I-frame and the next is a P-frame or B-frame they are
> bound to be different.

No, AIUI ffmpeg(1) produces a PNG that's the whole assembled picture,
just as if one hit `Pause' when watching TV.  Any indication of how that
was built up from the input video encoding is lost.

    ffmpeg -i foo.mp4 -t 1 foo-%04d.png

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From ralph at inputplus.co.uk  Mon May  7 09:00:41 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Mon, 07 May 2018 17:00:41 +0100
Subject: no more hslv format ?
In-Reply-To: <56f1f41ee0web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <56f1f41ee0web@audiomisc.co.uk>
Message-ID: <20180507160041.60AAD21953@orac.inputplus.co.uk>

Hi Jim,

> My concern is that I only have a total of 8GB of ram at present on the
> machine, and the 1280x720 50fps files tend to come in at 2GB or more
> per hour. 

Yes, you'd need to finish one download off, clearing RAM disk, before
starting the next.

> To save time and avoid running past 9am (and entering metered time) it
> would be quickest to do all the fetches first.

I too have metered broadband, but have automated use of it begin at
00:03 using timeout(1) to ensure it is killed by my 08:00 cut-off time
if it didn't complete for some reason.  Have you considered something
similar, perhaps using get_iplayer's `--pvr-queue', etc., to add
programmes of interest, and then telling it to `run' some or all of the
queue items?

> but have about three times as much data written to hd at some point.

Yes, with my preferred choices, get_iplayer's writes to disk would be

    downloaded video
    downloaded audio
    video and audio combined into MP4
    tagged MP4

so 3(v+a), and I'm not converting to 25 fps.

> I could use spinning rust I guess. but that seems messy.

Do you have it already available in the machine to use?  That seems an
easy compromise.

You could also look at your SSD's specifications and whether you're
striving too hard to avoid what might not be a problem with modern ones
and their built-in wear levelling.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From christopher at custommade.org.uk  Mon May  7 09:09:41 2018
From: christopher at custommade.org.uk (Christopher Woods)
Date: Mon, 07 May 2018 17:09:41 +0100
Subject: no more hslv format ?
In-Reply-To: <923678f6-f950-1ec8-cbf4-565a3719c621@zoho.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>
 <923678f6-f950-1ec8-cbf4-565a3719c621@zoho.com>
Message-ID: <1633b5d9d88.2790.ba2ca43ed0d8fe55db1ebf01ab8dfa50@custommade.org.uk>


On 3 May 2018 21:38:23 RS <richard22j at zoho.com> wrote:

> On 03/05/18 12:18, Steve Dodd wrote:
>
>> Is it possible it depends on the source material? From the BBC quote
>> earlier it sounds like their "source" material is still mostly HD
>> interlaced, but perhaps some of their sources are all also
>> progressive, and those ones get doubled identical frames (which might
>> be a logical result of feeding 25p material into something that's
>> designed to interpolate interlaced fields into frames)?
> The satellite broadcasts I have seen from the BBC and other broadcasters
> are 1920x1080i25 for HD and 720x576i25 or 704x576i25 for SD.  Everything
> that is broadcast is in an interlaced format at some stage, however it
> is generated.  Some programmes are only produced for the iPlayer, so
> they may be generated in a different way.


A handful of HD is still 1440x1080i25 :( the non-square pixel format is a 
great cheat for bandwidth saving. The chroma compression on current 
generation terrestrial broadcasting also makes everything look bad. Once 
you watch uncompressed HD-SDI you never want anything else :) Some SD is 
way below what we would accept as SD resolution as well, all in the name of 
cost saving.


Steve is right on the doubled technique, this is the filmic look you see on 
documentaries and dramas. That source material will be likely be captured 
as progressive frames, then upconverted to 50 fields per second, 
'progressive segmented frame' format (PsF) in the edit.

The same image is stored across every two fields that constitute one frame, 
meaning you're being shown one effective whole full resolution image, 25 
times a second.

With standard interlaced footage, you're being shown overlapping effective 
'images' at half-resolution, 50 times a second. Persistence of vision 
(well, nowadays, flat panel processing circuitry) bob deinterlaces the 
fields to produce 50 images per second.


However, TVs sometimes do this artificially by taking true progressive 
sources and interpolating the footage to a higher refresh rate (100/200 
Hz). This inherently looks dire and should be disabled.

On some TVs with poorly written image DSP (or badly chosen settings) you'll 
see the picture suddenly go 'filmic' as the deinterlace method changes to 
blend, which literally blends together pairs of fields to produce 25 frames.

Some explainers of fields, frames and progressive segmented format interlaced:

https://en.m.wikipedia.org/wiki/Progressive_segmented_frame

https://wolfcrow.com/blog/understanding-terminology-progressive-segmented-frames-psf/
https://wolfcrow.com/blog/understanding-terminology-progressive-frames-interlaced-frames-and-the-field-rate/




From ralph at inputplus.co.uk  Mon May  7 09:15:58 2018
From: ralph at inputplus.co.uk (Ralph Corderoy)
Date: Mon, 07 May 2018 17:15:58 +0100
Subject: no more hslv format ?
In-Reply-To: <56f254a551web@audiomisc.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <56f254a551web@audiomisc.co.uk>
Message-ID: <20180507161558.42CFE21953@orac.inputplus.co.uk>

Hi Jim,

> > Well, `df -t tmpfs' will probably show /tmp is a tmpfs so you could
> > `--output /tmp' and you should see its intermediate files, and the
> > final file, only appear there.  `--command' could then move that
> > final file to the SSD, or run a conversion command that writes to
> > the SSD and then removes the tmpfs input.
>
> the df command listed various 'locations'. These included
>
>     tmpfs  815696   1352  814334   /run
>     none   4078460  0     4078460  /run/shm
>     none   102400   8     102392   /run/usr

/run/shm is half your RAM, as I'd expect.  I'd also expect /tmp to be
there.  What does `df -T /tmp' show?  Perhaps that's using SSD for all
temporary files and you might not want that given your concerns.

> Inside /run/usr I did find a directory for my user id and could use
> that as the user.

But it's deliberately constrained to be small.

> However I don't know what the /run/shm is for, so need to do some
> finding out...

POSIX's communication methods of shared memory and semaphores;
shm_overview(7), and sem_overview(7).  Again, not what you want.

> I did set up an explict ramdisc on another machine ages ago by adding
> a line to the /etc/fstab file. But that fixed the size to 256 MB,
> which in this context is tiny. System monitor says I have 8GB of ram.

That's the right idea.  Here, the system has /tmp be a tmpfs that is
half the RAM, like the `shm' one above.  They share one single half.
`systemctl status tmp.mount' is what creates it on some systems these
days, not an /etc/fstab entry.

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy


From lucywalker46 at yahoo.com  Mon May  7 10:13:51 2018
From: lucywalker46 at yahoo.com (Lucy Walker)
Date: Mon, 7 May 2018 18:13:51 +0100
Subject: no more hslv format ?
In-Reply-To: <1633b5d9d88.2790.ba2ca43ed0d8fe55db1ebf01ab8dfa50@custommade.org.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>
 <923678f6-f950-1ec8-cbf4-565a3719c621@zoho.com>
 <1633b5d9d88.2790.ba2ca43ed0d8fe55db1ebf01ab8dfa50@custommade.org.uk>
Message-ID: <84ce6e55-a59f-e44e-f463-fdd27b801ca5@yahoo.com>



On 07/05/2018 17:09, Christopher Woods wrote:
>
> Steve is right on the doubled technique, this is the filmic look you 
> see on documentaries and dramas. That source material will be likely 
> be captured as progressive frames, then upconverted to 50 fields per 
> second, 'progressive segmented frame' format (PsF) in the edit.
>

At least one "yoof soap" shoots 1080p25 shuttered to 1/50th - looks like 
shit, but the head of cameras is a fucking moron. No more processing 
than that


From christopher at custommade.org.uk  Mon May  7 17:32:22 2018
From: christopher at custommade.org.uk (Christopher Woods)
Date: Tue, 08 May 2018 01:32:22 +0100
Subject: no more hslv format ?
In-Reply-To: <84ce6e55-a59f-e44e-f463-fdd27b801ca5@yahoo.com>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <CAA64434-19F6-41A1-89D9-EE9B6CF62E7E@cantab.net>
 <20180502213323.48FDA21CC3@orac.inputplus.co.uk>
 <1E7C7F28-56D7-4CB2-933B-F78A945D7257@cantab.net>
 <df37d77d-6b15-d00e-0d11-63e4770ec0a5@zoho.com>
 <CAJ6XMjGw26ahfHtq09O5sFzf+RM3hy9zuZa6GyM7as6eyDgdFg@mail.gmail.com>
 <923678f6-f950-1ec8-cbf4-565a3719c621@zoho.com>
 <1633b5d9d88.2790.ba2ca43ed0d8fe55db1ebf01ab8dfa50@custommade.org.uk>
 <84ce6e55-a59f-e44e-f463-fdd27b801ca5@yahoo.com>
Message-ID: <1633d29d5f0.2790.ba2ca43ed0d8fe55db1ebf01ab8dfa50@custommade.org.uk>

Curious - 25p @ 1/50th is the generally accepted 180? rule, unless they're 
doing weird stylistic stuff in post? Or does it all just look smooshy and 
motion is really badly defined?

Personally I wish people would shoot 50p and frame drop instead of shoot 
25p, I think it's more flexible.

(Making me feel old now. The last yoof soap I watched regularly was Byker 
Grove)

On 7 May 2018 18:14:40 Lucy Walker <lucywalker46 at yahoo.com> wrote:

> On 07/05/2018 17:09, Christopher Woods wrote:
>>
>> Steve is right on the doubled technique, this is the filmic look you
>> see on documentaries and dramas. That source material will be likely
>> be captured as progressive frames, then upconverted to 50 fields per
>> second, 'progressive segmented frame' format (PsF) in the edit.
>>
>
> At least one "yoof soap" shoots 1080p25 shuttered to 1/50th - looks like
> shit, but the head of cameras is a fucking moron. No more processing
> than that
>
> _______________________________________________
> get_iplayer mailing list
> get_iplayer at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/get_iplayer





From gasssmith at gmail.com  Tue May  8 02:36:17 2018
From: gasssmith at gmail.com (Geoff Smith)
Date: Tue, 8 May 2018 10:36:17 +0100
Subject: 3.14.0 released
Message-ID: <CAJdt=oGfiO6z9S0ZCQEEY924x9hd5KUUnc9H14FFK+_us45CrQ@mail.gmail.com>

https://github.com/get-iplayer/get_iplayer_win32/releases/tag/3.14.0


From richard22j at zoho.com  Wed May  9 03:49:04 2018
From: richard22j at zoho.com (RS)
Date: Wed, 9 May 2018 11:49:04 +0100
Subject: no more hslv format ?
In-Reply-To: <20180507160041.60AAD21953@orac.inputplus.co.uk>
References: <6EFB8831284C4E2DBDB1A9649709CCC7@oldpc>
 <56f0dca351web@audiomisc.co.uk>
 <20180501113138.870F721091@orac.inputplus.co.uk>
 <56f15bc592web@audiomisc.co.uk>
 <20180502152015.50EA621CC3@orac.inputplus.co.uk>
 <56f1f41ee0web@audiomisc.co.uk>
 <20180507160041.60AAD21953@orac.inputplus.co.uk>
Message-ID: <f880c80a-8a38-aca3-78b7-09997ae8e0b1@zoho.com>

On 07/05/18 17:00, Ralph Corderoy wrote:

> I too have metered broadband, but have automated use of it begin at
> 00:03 using timeout(1) to ensure it is killed by my 08:00 cut-off time
> if it didn't complete for some reason.  Have you considered something
> similar, perhaps using get_iplayer's `--pvr-queue', etc., to add
> programmes of interest, and then telling it to `run' some or all of the
> queue items?
> 
Hi Jim and Ralph

Are you sure metered broadband is the best deal for you?  What you said 
about unlimited overnight use reminded me of a contract I had with 
plusnet which was unlimited from midnight to 0800.  Out of curiosity I 
tried to find the prices to compare.  It wasn't easy.  Sky won't tell 
you unless you have an account, but I did find the archived prices for 
plusnet Essentials.

Interestingly for new contracts there are only two metered offers.  They 
are both for Sky fibre and are more expensive than unmetered offers.

Although the old metered contracts sound cheap the difference is that it 
was necessary to pay separately for the line rental.  According to BT's 
website its monthly line rental is now ?18.99.  The Post Office is 
offering unlimited ADSL2+ for ?17/month including line rental so it is 
effectively paying you ?2/month to take its unlimited broadband.

Those prices suggest it may be a good idea to review your contract.

Best wishes
Richard




From richard22j at zoho.com  Sun May 13 03:56:35 2018
From: richard22j at zoho.com (RS)
Date: Sun, 13 May 2018 11:56:35 +0100
Subject: Is there a time limit on length of downloads?
Message-ID: <59dd9018-f8ab-3a42-807a-ec0a0c711cc9@zoho.com>

This is probably a ffmpeg problem, but I mention it here in case it is a 
problem with what get_iplayer is passing to ffmpeg.  I have downloaded 
the Red Button coverage of the cross-country day at Badminton (p065zgf4) 
using hvfxsd3 from Akamai.  The resultant .mp4 file is 4.6GByte and the 
duration is 5h 56min.  The .mp4 file plays fine in VLC, and if I seek to 
a different part of the file it does so within 500ms.

During the conversion by ffmpeg I got a large number of error messages like

[mpegts @ 0x56028af28400] Invalid timestamps stream=1, pts=160090200, 
dts=160092000, size=7261
[mp4 @ 0x56028af58360] Invalid DTS: 159190200 PTS: 159188400 in output 
stream 0:0, replacing by guess
[mp4 @ 0x56028af58360] Invalid DTS: 159192000 PTS: 159190200 in output 
stream 0:0, replacing by guess
[mpegts @ 0x56028af28400] Invalid timestamps stream=1, pts=160093800, 
dts=160095600, size=1747
[mpegts @ 0x56028af28400] Invalid timestamps stream=1, pts=160095600, 
dts=160097400, size=7676
[mp4 @ 0x56028af58360] Invalid DTS: 159195600 PTS: 159193800 in output 
stream 0:0, replacing by guess

Is there a limit on the length of recording get_iplayer can handle, or 
is it a ffmpeg problem?

We were recently discussing the withdrawal of HLS modes.  When the HLShd 
mode existed it was never made available for Red Button channels.  What 
I could not remember was whether other HD modes were produced.  In this 
case all the HVF modes including HVFhd were present, even though the Red 
Button is usually a SD channel.  get_iplayer's estimate of hvfhd file 
size was 15290MByte!




From gasssmith at gmail.com  Sun May 13 05:41:57 2018
From: gasssmith at gmail.com (Geoff Smith)
Date: Sun, 13 May 2018 13:41:57 +0100
Subject: Is there a time limit on length of downloads?
In-Reply-To: <59dd9018-f8ab-3a42-807a-ec0a0c711cc9@zoho.com>
References: <59dd9018-f8ab-3a42-807a-ec0a0c711cc9@zoho.com>
Message-ID: <CAJdt=oH2MGRvQpGTdiXcuRg8eAe19nVVK9TvbtTA+Nz0v_EkFg@mail.gmail.com>

https://www.bbc.co.uk/programmes/p065zgf4
says duration 6h 10m

On 13/05/2018, RS <richard22j at zoho.com> wrote:
> I have downloaded
> the Red Button coverage of the cross-country day at Badminton (p065zgf4)
> using hvfxsd3 from Akamai.  The resultant .mp4 file is 4.6GByte and the
> duration is 5h 56min.  The .mp4 file plays fine in VLC, and if I seek to
> a different part of the file it does so within 500ms.


From gasssmith at gmail.com  Sun May 13 05:56:10 2018
From: gasssmith at gmail.com (Geoff Smith)
Date: Sun, 13 May 2018 13:56:10 +0100
Subject: Is there a time limit on length of downloads?
In-Reply-To: <CAJdt=oH2MGRvQpGTdiXcuRg8eAe19nVVK9TvbtTA+Nz0v_EkFg@mail.gmail.com>
References: <59dd9018-f8ab-3a42-807a-ec0a0c711cc9@zoho.com>
 <CAJdt=oH2MGRvQpGTdiXcuRg8eAe19nVVK9TvbtTA+Nz0v_EkFg@mail.gmail.com>
Message-ID: <CAJdt=oFtkoNTmiykTef=bgUEJwYnn54ntNL+F8DMbdNHi5RDTQ@mail.gmail.com>

Ignore that, it's irrelevant.

On 13/05/2018, Geoff Smith <gasssmith at gmail.com> wrote:
> https://www.bbc.co.uk/programmes/p065zgf4
> says duration 6h 10m
>
> On 13/05/2018, RS <richard22j at zoho.com> wrote:
>> I have downloaded
>> the Red Button coverage of the cross-country day at Badminton (p065zgf4)
>> using hvfxsd3 from Akamai.  The resultant .mp4 file is 4.6GByte and the
>> duration is 5h 56min.  The .mp4 file plays fine in VLC, and if I seek to
>> a different part of the file it does so within 500ms.
>


