get_iplayer Download Olympics In Chunks Tutorial

MS jmstanfield at gmail.com
Tue Aug 7 11:46:19 EDT 2012


Hi all,

Matthew Kerle emailed me directly asking for help to download Olympic 
event sessions in chunks. I decided to write a tutorial for him and for 
the list.

The tutorial is pasted online here:

http://pastebin.com/R1NQhswH

You can view it online and there's also a download (plain text) link 
near the top.

I shall also append the tutorial below but it'll probably view better on 
the web or downloaded in plain text as it'll be best viewed with a fixed 
width font.

Anyway all comments would be welcome, mistakes spotted, improvements, 
alternatives, and such like.

Hope it helps at least a few people as it ended up taking me a couple of 
hours.

Best wishes,

Matthew



         get_iplayer Download Olympics In Chunks Guide
         ---------------------------------------------

This is for Linux users, other OSes may vary.

Linux software used: get_iplayer, VLC, ffmpeg, and avidemux.


This tutorial is how to download Olympic event sessions in chunks using 
get_iplayer, and then to do a rough and ready edit (not slick with nice 
transitions or anything), and to end up with a well-formed .mp4 file. It 
is not easy and it is time consuming, but worked well for me.


Step 1:

Download the get_iplayer Olympics fix version from this page:

http://git.infradead.org/get_iplayer.git

"Allow Olympics live streams to work using new mediaselector/5 URLs"

Click on 'snapshot', save to disk, then extract the archive.

Either replace your existing get_iplayer with this version, or always 
run this version by using the full path to the new get_iplayer script, 
or from within its own directory using ./get_iplayer notation.

e.g.

Using './' notation:

cd /home/user/Scripts/Get-IPlayer/get_iplayer-c3aafce/
./get_iplayer 
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w316y" etc

Using full path:

/home/user/Scripts/Get-IPlayer/get_iplayer-c3aafce/get_iplayer 
"http://www.bbc.co.uk/..." etc


Step 2:

Get your event's video link from:

http://www.bbc.co.uk/sport/olympics/2012/live-video
http://www.bbc.co.uk/sport/olympics/2012/schedule-results

The bottom link above has the full Olympics schedule, click on the 
event/day and select "Catch Up" to get to the video page.

e.g.

Men's Individual Foil heats and quarter-finals
http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr

By the way, wait until the event's session has actually finished before 
starting the chunk download.


Step 3:

If the session you are downloading is long, the chances are you will 
need to download in chunks. There is a 4GB video file size limit (not 
due to a file system limitation but to 32 bit memory addressing limit). 
However with the software versions I've got, I rarely got as far as 4GB 
with each chunk, typically my downloads were of 1.5 to 2.5 hours in 
video length before RTMPDump crashed - but that's ok the files left 
after the crash work fine. Sometimes the crash happened sooner, when the 
downloaded chunk was less than an hour I just deleted the file and 
started that chunk again. NOTE: 90 mins of BBC Olympics 1280x720 video 
is a bit over 2GB of file size.

Keep notes about your downloaded chunks and ALWAYS REMEMBER to rename 
your files or get_iplayer will try to resume them which will not work 
(at least didn't for me), and unless you rename them you'll lose track 
of what you've got and which is which.

There will be lots of time calculation needed, the Time Calculator on 
this page was useful: http://www.grun1.com/utils/timeCalc.html


Download the first chunk, no start time necessary:

./get_iplayer 
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force 
-o "/big/storage/location/"

Download stopped at 5260 secs [Find this in the terminal, you may need 
to scroll up above errors.]

5260 = 87 mins 40 secs = 01:27:40

Rename the file: Mens_Team_Foil_final_00.00.00-01.27.40.mp4.flv

Check the video. I used VLC and skipped through it using 
<CTRL><RIGHT_ARROW> / <SHIFT><RIGHT_ARROW> / <ALT><RIGHT_ARROW> to jump 
quickly through the video, VLC allows you to manually set both the jump 
size lengths and the keys to use for them in this settings location:

VLC Prefs --> Show All --> Interface --> Hotkeys Setting


Download the second chunk, start time is now necessary:

The first chunk stopped at 01:27:40. You can start the next chunk with a 
few mins overlap, say at 01:25:00 (stick to whole minutes and the 
calculations will be easier). However this ignores the editing process - 
it may be that between 01:25:00 and 01:27:40 there is continuous 
sporting action and no convenient join/edit point for later. My 
technique was to look in the video for a specific join point, a 5-15 min 
break, or some kind of pause in the action. Let's say I found a 
convenient 5 min break while viewing in VLC at 01:20:25, so I'll start 
the next chunk at 01:20:00 (believe me whole mins will make your life 
easier).

./get_iplayer 
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force 
--start "01:20:00" -o "/big/storage/location/"

Download stopped at 12285 secs. This is ACCUMULATIVE, it includes the 
start time of 01:20:00.

Calculate length of chunk:

Started at:
01:20:00 = 80 mins
80 * 60 = 4800 secs

Duration of chunk is: Stop Pos - Start Pos
Duration of chunk is: 12285 - 4800 = 7485 seconds

7485 / 60 = 124.75
7485 mod 60 = 45
124 mins 45 secs = 02:04:45

Use VLC to check that the chunk is 02:04:45 in length.

Chunk finish time in relation to the overall video is therefore:
Chunk Start At + Chunk Duration
01:20:00 + 02:04:45 = 03:24:45

Rename the file: Mens_Team_Foil_final_01.20.00-03.24.45.mp4.flv


Download the third chunk and subsequent chunks...:
[DO NOT SKIP READING THIS BIT - IT IS NOT QUITE THE SAME AS DOWNLOADING 
THE 2ND CHUNK.]

Now you need to find a convenient start point again, so view the 2nd 
chunk. VLC will specify the time from 00:00:00, when in terms of the 
overall you're actually starting at 01.20.00. Let's say you find a good 
start-the-next-chunk start point at approx. 10 mins from the actual end 
of the 2nd chunk, at 01:55:00 (as reported by VLC).

Start next chunk at:
Pos Start of 2nd Chunk + Pos into 2nd Chunk
01.20.00 + 01:55:00 = 3:15:00

./get_iplayer 
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force 
--start "03:15:00" -o "/big/storage/location/"

Download stopped at...

Repeat this process till all the chunks are done. I deleted any chunks 
that downloaded less than an hour of video and started them again on the 
command line (but you must remember to delete the small chunks or 
get_iplayer will try appending to the chunk which won't work). Sometimes 
I had to restart a chunk several times.


Step 4:

Edit and remux - this will fix the video timecodes and give you nice 
.mp4 files.

Now you need to view your chunks and work out the points at which you 
want each chunk to start and stop, so that they will all fit nicely 
together - I also used this process to edit out long breaks in the 
action, this meant I sometimes used the same chunk to create 2 different 
new files (clearly with different start and end points). By doing so I 
turned 9 hours of video into 6.5 hours of actual action.

I created a table of what I needed to do in order to edit out the breaks 
and remove overlaps in the chunks. The start and end times are from the 
actual times shown in VLC of each chunk, the overall time is now no 
longer an issue.

ffmpeg needs a start point and then the duration (not the end point), so 
I used a Time Difference Calculator to get the duration. Put the end pos 
in the top text box and the start pos in the bottom text box: 
http://www.grun1.com/utils/timeDiff.cfm

NOTE: When creating your own table, note that ffmpeg is not perfectly 
accurate to the exact second, for instance the 2nd entry in the table 
below resulted in a file output by ffmpeg which was 00:57:34 in duration 
and not 00:57:35, 1 second went AWOL. So choose your edit points wisely, 
start a second or 2 early, finish a second or 2 late. I was only doing a 
rough 'just get it all without the breaks and overlaps' edit.

Fencing - Men's Indiv. Foil 1st Session (I also had to do the 2nd session):

Note the multiple use of Chunk_1 and Chunk_2 to remove long breaks.

------------------------------------------------------------
DL Chunk    Start       End         Duration    Output
------------------------------------------------------------
Chunk_1     00:00:00    00:40:00    00:40:00    Output_1.mp4
Chunk_1     01:21:00    02:18:35    00:57:35    Output_2.mp4
Chunk_2     00:11:00    00:29:00    00:18:00    Output_3.mp4
Chunk_2     00:37:00    00:53:00    00:16:00    Output_4.mp4
Chunk_3     00:17:30    01:32:30    01:15:00    Output_5.mp4
Chunk_4     00:16:30    00:52:30    00:36:00    Output_6.mp4
------------------------------------------------------------
Total duration of all output files: 4:02:35
------------------------------------------------------------

The basic ffmpeg command is as follows - PARAMETER ORDER IS IMPORTANT:

ffmpeg -i INFILE.mp4.flv -ss StartAt -t Duration -vcodec copy -acodec 
copy Output_1.mp4

-i    = input file
-ss   = start pos time
-t    = duration (not end pos time)

For -ss and -t you can use seconds or "hh:mm:ss". e.g. these 2 command 
lines below do the same thing, they start 10 mins from the beginning and 
provide a video of the next 1 hour, put another way they do 00:10:00 to 
01:10:00.

ffmpeg -i infile.mp4.flv -ss 600 -t 3600 -vcodec copy -acodec copy 
Output_1.mp4
ffmpeg -i infile.mp4.flv -ss "00:10:00" -t "01:00:00" -vcodec copy 
-acodec copy Output_1.mp4

The above table translated into these commands:

e.g.
ffmpeg -i DL_Chunk_01.mp4.flv -ss "START_AT" -t "DURATION" -vcodec copy 
-acodec copy Output.mp4

I actually did (though I've renamed the input file names below for clarity):

ffmpeg -i DL_Chunk_01.mp4.flv -ss "00:00:00" -t "00:40:00" -vcodec copy 
-acodec copy Output_1.mp4
ffmpeg -i DL_Chunk_01.mp4.flv -ss "01:21:00" -t "00:57:35" -vcodec copy 
-acodec copy Output_2.mp4
ffmpeg -i DL_Chunk_02.mp4.flv -ss "00:11:00" -t "00:18:00" -vcodec copy 
-acodec copy Output_3.mp4
ffmpeg -i DL_Chunk_02.mp4.flv -ss "00:37:00" -t "00:16:00" -vcodec copy 
-acodec copy Output_4.mp4
ffmpeg -i DL_Chunk_03.mp4.flv -ss "00:17:30" -t "01:15:00" -vcodec copy 
-acodec copy Output_5.mp4
ffmpeg -i DL_Chunk_04.mp4.flv -ss "00:16:30" -t "00:36:00" -vcodec copy 
-acodec copy Output_6.mp4

View and check all the output files using VLC.


Step 5:

Joining together the output files from step 4 is very easy.

Start Avidemux:

a) On the left side set video to 'copy', audio to 'copy', and format to 
'MP4'.

b) Click 'open' at the top and select your first output file from step 
4, in my example this would be 'Output_1.mp4'.

Avidemux will say:

"H.264 detected - If the file is using B-frames as reference it can lead 
to a crash or stuttering. Avidemux can use another mode which is safe 
but YOU WILL LOSE FRAME ACCURACY. Do you want to use that mode?"

Select 'Use Safe Mode'.

I got a big green screen after the video had loaded, click play (and 
then stop after a while) if you're worried that this is not normal, it 
frightened the pants off me, the green screen should immediately be 
replaced by your video playing.

c) Now go to the File menu and click 'Append', select your next output 
file, in my example this would be 'Output_2.mp4'. Select 'Use Safe Mode' 
again. Then select 'Append' again for each of your remaining output 
files (in order) from step 4, of course selecting 'Use Safe Mode' each 
time. You can keep an eye on the overall time length at the bottom and 
watch it grow as you append each file and check that it is growing by 
the right amount each time if you're worried.

d) When all the files have been appended... double check on the left 
side that video is still set to 'copy', audio to 'copy', and format to 
'MP4' as per step 5 a). Then select the file menu, then 'save', then 
'save video' and chose a save location and a name for your file, e.g. 
'Full_Final_Edit.mp4' (add the '.mp4' bit at the end of the filename as 
this is not automatically added). Avidemux will now join all the files 
together nicely, muxing into a nice stable, .mp4 file. It may take a 
while, depending on your hard disk's speed and length of the video, 6.5 
hours took my speedy desktop PC about 5 mins (but I have a nice new big 
and fast hard disk).


That's it - end of tutorial.



More information about the get_iplayer mailing list