get_iplayer Download Olympics In Chunks Tutorial
MS
jmstanfield at gmail.com
Tue Aug 7 11:46:19 EDT 2012
Hi all,
Matthew Kerle emailed me directly asking for help to download Olympic
event sessions in chunks. I decided to write a tutorial for him and for
the list.
The tutorial is pasted online here:
http://pastebin.com/R1NQhswH
You can view it online and there's also a download (plain text) link
near the top.
I shall also append the tutorial below but it'll probably view better on
the web or downloaded in plain text as it'll be best viewed with a fixed
width font.
Anyway all comments would be welcome, mistakes spotted, improvements,
alternatives, and such like.
Hope it helps at least a few people as it ended up taking me a couple of
hours.
Best wishes,
Matthew
get_iplayer Download Olympics In Chunks Guide
---------------------------------------------
This is for Linux users, other OSes may vary.
Linux software used: get_iplayer, VLC, ffmpeg, and avidemux.
This tutorial is how to download Olympic event sessions in chunks using
get_iplayer, and then to do a rough and ready edit (not slick with nice
transitions or anything), and to end up with a well-formed .mp4 file. It
is not easy and it is time consuming, but worked well for me.
Step 1:
Download the get_iplayer Olympics fix version from this page:
http://git.infradead.org/get_iplayer.git
"Allow Olympics live streams to work using new mediaselector/5 URLs"
Click on 'snapshot', save to disk, then extract the archive.
Either replace your existing get_iplayer with this version, or always
run this version by using the full path to the new get_iplayer script,
or from within its own directory using ./get_iplayer notation.
e.g.
Using './' notation:
cd /home/user/Scripts/Get-IPlayer/get_iplayer-c3aafce/
./get_iplayer
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w316y" etc
Using full path:
/home/user/Scripts/Get-IPlayer/get_iplayer-c3aafce/get_iplayer
"http://www.bbc.co.uk/..." etc
Step 2:
Get your event's video link from:
http://www.bbc.co.uk/sport/olympics/2012/live-video
http://www.bbc.co.uk/sport/olympics/2012/schedule-results
The bottom link above has the full Olympics schedule, click on the
event/day and select "Catch Up" to get to the video page.
e.g.
Men's Individual Foil heats and quarter-finals
http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr
By the way, wait until the event's session has actually finished before
starting the chunk download.
Step 3:
If the session you are downloading is long, the chances are you will
need to download in chunks. There is a 4GB video file size limit (not
due to a file system limitation but to 32 bit memory addressing limit).
However with the software versions I've got, I rarely got as far as 4GB
with each chunk, typically my downloads were of 1.5 to 2.5 hours in
video length before RTMPDump crashed - but that's ok the files left
after the crash work fine. Sometimes the crash happened sooner, when the
downloaded chunk was less than an hour I just deleted the file and
started that chunk again. NOTE: 90 mins of BBC Olympics 1280x720 video
is a bit over 2GB of file size.
Keep notes about your downloaded chunks and ALWAYS REMEMBER to rename
your files or get_iplayer will try to resume them which will not work
(at least didn't for me), and unless you rename them you'll lose track
of what you've got and which is which.
There will be lots of time calculation needed, the Time Calculator on
this page was useful: http://www.grun1.com/utils/timeCalc.html
Download the first chunk, no start time necessary:
./get_iplayer
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force
-o "/big/storage/location/"
Download stopped at 5260 secs [Find this in the terminal, you may need
to scroll up above errors.]
5260 = 87 mins 40 secs = 01:27:40
Rename the file: Mens_Team_Foil_final_00.00.00-01.27.40.mp4.flv
Check the video. I used VLC and skipped through it using
<CTRL><RIGHT_ARROW> / <SHIFT><RIGHT_ARROW> / <ALT><RIGHT_ARROW> to jump
quickly through the video, VLC allows you to manually set both the jump
size lengths and the keys to use for them in this settings location:
VLC Prefs --> Show All --> Interface --> Hotkeys Setting
Download the second chunk, start time is now necessary:
The first chunk stopped at 01:27:40. You can start the next chunk with a
few mins overlap, say at 01:25:00 (stick to whole minutes and the
calculations will be easier). However this ignores the editing process -
it may be that between 01:25:00 and 01:27:40 there is continuous
sporting action and no convenient join/edit point for later. My
technique was to look in the video for a specific join point, a 5-15 min
break, or some kind of pause in the action. Let's say I found a
convenient 5 min break while viewing in VLC at 01:20:25, so I'll start
the next chunk at 01:20:00 (believe me whole mins will make your life
easier).
./get_iplayer
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force
--start "01:20:00" -o "/big/storage/location/"
Download stopped at 12285 secs. This is ACCUMULATIVE, it includes the
start time of 01:20:00.
Calculate length of chunk:
Started at:
01:20:00 = 80 mins
80 * 60 = 4800 secs
Duration of chunk is: Stop Pos - Start Pos
Duration of chunk is: 12285 - 4800 = 7485 seconds
7485 / 60 = 124.75
7485 mod 60 = 45
124 mins 45 secs = 02:04:45
Use VLC to check that the chunk is 02:04:45 in length.
Chunk finish time in relation to the overall video is therefore:
Chunk Start At + Chunk Duration
01:20:00 + 02:04:45 = 03:24:45
Rename the file: Mens_Team_Foil_final_01.20.00-03.24.45.mp4.flv
Download the third chunk and subsequent chunks...:
[DO NOT SKIP READING THIS BIT - IT IS NOT QUITE THE SAME AS DOWNLOADING
THE 2ND CHUNK.]
Now you need to find a convenient start point again, so view the 2nd
chunk. VLC will specify the time from 00:00:00, when in terms of the
overall you're actually starting at 01.20.00. Let's say you find a good
start-the-next-chunk start point at approx. 10 mins from the actual end
of the 2nd chunk, at 01:55:00 (as reported by VLC).
Start next chunk at:
Pos Start of 2nd Chunk + Pos into 2nd Chunk
01.20.00 + 01:55:00 = 3:15:00
./get_iplayer
"http://www.bbc.co.uk/sport/olympics/2012/live-video/p00w2xsr" --force
--start "03:15:00" -o "/big/storage/location/"
Download stopped at...
Repeat this process till all the chunks are done. I deleted any chunks
that downloaded less than an hour of video and started them again on the
command line (but you must remember to delete the small chunks or
get_iplayer will try appending to the chunk which won't work). Sometimes
I had to restart a chunk several times.
Step 4:
Edit and remux - this will fix the video timecodes and give you nice
.mp4 files.
Now you need to view your chunks and work out the points at which you
want each chunk to start and stop, so that they will all fit nicely
together - I also used this process to edit out long breaks in the
action, this meant I sometimes used the same chunk to create 2 different
new files (clearly with different start and end points). By doing so I
turned 9 hours of video into 6.5 hours of actual action.
I created a table of what I needed to do in order to edit out the breaks
and remove overlaps in the chunks. The start and end times are from the
actual times shown in VLC of each chunk, the overall time is now no
longer an issue.
ffmpeg needs a start point and then the duration (not the end point), so
I used a Time Difference Calculator to get the duration. Put the end pos
in the top text box and the start pos in the bottom text box:
http://www.grun1.com/utils/timeDiff.cfm
NOTE: When creating your own table, note that ffmpeg is not perfectly
accurate to the exact second, for instance the 2nd entry in the table
below resulted in a file output by ffmpeg which was 00:57:34 in duration
and not 00:57:35, 1 second went AWOL. So choose your edit points wisely,
start a second or 2 early, finish a second or 2 late. I was only doing a
rough 'just get it all without the breaks and overlaps' edit.
Fencing - Men's Indiv. Foil 1st Session (I also had to do the 2nd session):
Note the multiple use of Chunk_1 and Chunk_2 to remove long breaks.
------------------------------------------------------------
DL Chunk Start End Duration Output
------------------------------------------------------------
Chunk_1 00:00:00 00:40:00 00:40:00 Output_1.mp4
Chunk_1 01:21:00 02:18:35 00:57:35 Output_2.mp4
Chunk_2 00:11:00 00:29:00 00:18:00 Output_3.mp4
Chunk_2 00:37:00 00:53:00 00:16:00 Output_4.mp4
Chunk_3 00:17:30 01:32:30 01:15:00 Output_5.mp4
Chunk_4 00:16:30 00:52:30 00:36:00 Output_6.mp4
------------------------------------------------------------
Total duration of all output files: 4:02:35
------------------------------------------------------------
The basic ffmpeg command is as follows - PARAMETER ORDER IS IMPORTANT:
ffmpeg -i INFILE.mp4.flv -ss StartAt -t Duration -vcodec copy -acodec
copy Output_1.mp4
-i = input file
-ss = start pos time
-t = duration (not end pos time)
For -ss and -t you can use seconds or "hh:mm:ss". e.g. these 2 command
lines below do the same thing, they start 10 mins from the beginning and
provide a video of the next 1 hour, put another way they do 00:10:00 to
01:10:00.
ffmpeg -i infile.mp4.flv -ss 600 -t 3600 -vcodec copy -acodec copy
Output_1.mp4
ffmpeg -i infile.mp4.flv -ss "00:10:00" -t "01:00:00" -vcodec copy
-acodec copy Output_1.mp4
The above table translated into these commands:
e.g.
ffmpeg -i DL_Chunk_01.mp4.flv -ss "START_AT" -t "DURATION" -vcodec copy
-acodec copy Output.mp4
I actually did (though I've renamed the input file names below for clarity):
ffmpeg -i DL_Chunk_01.mp4.flv -ss "00:00:00" -t "00:40:00" -vcodec copy
-acodec copy Output_1.mp4
ffmpeg -i DL_Chunk_01.mp4.flv -ss "01:21:00" -t "00:57:35" -vcodec copy
-acodec copy Output_2.mp4
ffmpeg -i DL_Chunk_02.mp4.flv -ss "00:11:00" -t "00:18:00" -vcodec copy
-acodec copy Output_3.mp4
ffmpeg -i DL_Chunk_02.mp4.flv -ss "00:37:00" -t "00:16:00" -vcodec copy
-acodec copy Output_4.mp4
ffmpeg -i DL_Chunk_03.mp4.flv -ss "00:17:30" -t "01:15:00" -vcodec copy
-acodec copy Output_5.mp4
ffmpeg -i DL_Chunk_04.mp4.flv -ss "00:16:30" -t "00:36:00" -vcodec copy
-acodec copy Output_6.mp4
View and check all the output files using VLC.
Step 5:
Joining together the output files from step 4 is very easy.
Start Avidemux:
a) On the left side set video to 'copy', audio to 'copy', and format to
'MP4'.
b) Click 'open' at the top and select your first output file from step
4, in my example this would be 'Output_1.mp4'.
Avidemux will say:
"H.264 detected - If the file is using B-frames as reference it can lead
to a crash or stuttering. Avidemux can use another mode which is safe
but YOU WILL LOSE FRAME ACCURACY. Do you want to use that mode?"
Select 'Use Safe Mode'.
I got a big green screen after the video had loaded, click play (and
then stop after a while) if you're worried that this is not normal, it
frightened the pants off me, the green screen should immediately be
replaced by your video playing.
c) Now go to the File menu and click 'Append', select your next output
file, in my example this would be 'Output_2.mp4'. Select 'Use Safe Mode'
again. Then select 'Append' again for each of your remaining output
files (in order) from step 4, of course selecting 'Use Safe Mode' each
time. You can keep an eye on the overall time length at the bottom and
watch it grow as you append each file and check that it is growing by
the right amount each time if you're worried.
d) When all the files have been appended... double check on the left
side that video is still set to 'copy', audio to 'copy', and format to
'MP4' as per step 5 a). Then select the file menu, then 'save', then
'save video' and chose a save location and a name for your file, e.g.
'Full_Final_Edit.mp4' (add the '.mp4' bit at the end of the filename as
this is not automatically added). Avidemux will now join all the files
together nicely, muxing into a nice stable, .mp4 file. It may take a
while, depending on your hard disk's speed and length of the video, 6.5
hours took my speedy desktop PC about 5 mins (but I have a nice new big
and fast hard disk).
That's it - end of tutorial.
More information about the get_iplayer
mailing list