Compare commits

...

23 Commits

Author SHA1 Message Date
Philipp Hagemeister
ef357c4bf2 Bump version number 2011-09-15 18:48:29 +02:00
Philipp Hagemeister
5260e68f64 Add format fallback 2011-09-15 18:47:36 +02:00
Philipp Hagemeister
6a1ca41e17 Update README 2011-09-15 12:45:56 +02:00
Philipp Hagemeister
c99dcbd2d6 Merge remote-tracking branch 'rmanola/master' (Closes: #124)
Add option to specify mp3 quality and prevent the video from being deleted
2011-09-15 12:43:27 +02:00
Dominik Heidler
da0db53a75 added option to get the available formats for a video (Closes #106) 2011-09-15 12:20:03 +02:00
Kegan
c52b01f326 Added ability to download worst quality video file only. (Closes #113) 2011-09-15 12:14:16 +02:00
Kegan
36597dc40f Updated to stamp extracted audio file with HTTP last modified date. 2011-09-15 12:04:54 +02:00
Philipp Hagemeister
9b4556c469 New option --skip-download (Closes #162) 2011-09-15 11:36:49 +02:00
Philipp Hagemeister
f3098c4d8a --list-extractors (Closes #161) 2011-09-15 11:03:29 +02:00
Philipp Hagemeister
bdb3f7a769 Simplify suitable 2011-09-15 10:06:14 +02:00
Philipp Hagemeister
afb5b55de6 Proper warning if xml.etree.ElementTree is not available 2011-09-15 09:59:03 +02:00
Philipp Hagemeister
c23cec29a3 Update LATEST_VERSION (and wait for a script to do it so I do not forget ;) ) 2011-09-14 23:03:01 +02:00
Philipp Hagemeister
e5b9fac281 Bump version number 2011-09-14 22:55:26 +02:00
Philipp Hagemeister
08c1d0d3bc Update README 2011-09-14 22:55:09 +02:00
Anand Babu Periasamy
20e91e8375 Add --match-title and --reject-title (Closes #132) 2011-09-14 22:54:51 +02:00
Philipp Hagemeister
f9c6878714 Support for The Escapist 2011-09-14 22:26:53 +02:00
Philipp Hagemeister
8c5dc3ad40 Simplify IE index 2011-09-14 21:39:41 +02:00
Philipp Hagemeister
1d2e86aed9 Decapitalize options in README for consistency with youtube-dl --help 2011-09-14 21:20:23 +02:00
Philipp Hagemeister
a2f7e3a5bb Clarify usage 2011-09-14 21:19:33 +02:00
Philipp Hagemeister
f2a3a3522c typo in README 2011-09-14 21:18:22 +02:00
Philipp Hagemeister
b487ef0833 Fully implement comedycentral downloader 2011-09-14 21:17:05 +02:00
Philipp Hagemeister
d0922f29a3 Update LATEST_VERSION (oops) 2011-09-14 00:04:46 +02:00
rmanola
18b7f87409 Added option to allow different audio encoding qualities and to allow specify whether erase or not the video when it's need to extract the audio. 2011-06-06 17:46:37 -07:00
3 changed files with 345 additions and 224 deletions

View File

@@ -1 +1 @@
2011.09.13 2011.09.16

View File

@@ -1,7 +1,7 @@
# youtube-dl # youtube-dl
## USAGE ## USAGE
youtube-dl [OPTIONS] URL youtube-dl [options] url [url...]
## DESCRIPTION ## DESCRIPTION
**youtube-dl** is a small command-line program to download videos from **youtube-dl** is a small command-line program to download videos from
@@ -17,9 +17,17 @@ which means you can modify it, redistribute it or use it however you like.
-i, --ignore-errors continue on download errors -i, --ignore-errors continue on download errors
-r, --rate-limit LIMIT download rate limit (e.g. 50k or 44.6m) -r, --rate-limit LIMIT download rate limit (e.g. 50k or 44.6m)
-R, --retries RETRIES number of retries (default is 10) -R, --retries RETRIES number of retries (default is 10)
--dump-user-agent display the current browser identification
--list-extractors List all supported extractors and the URLs they
would handle
### Video Selection:
--playlist-start NUMBER playlist video to start at (default is 1) --playlist-start NUMBER playlist video to start at (default is 1)
--playlist-end NUMBER playlist video to end at (default is last) --playlist-end NUMBER playlist video to end at (default is last)
--dump-user-agent display the current browser identification --match-title REGEX download only matching titles (regex or caseless
sub-string)
--reject-title REGEX skip download for matching titles (regex or
caseless sub-string)
### Filesystem Options: ### Filesystem Options:
-t, --title use title in file name -t, --title use title in file name
@@ -38,12 +46,15 @@ which means you can modify it, redistribute it or use it however you like.
### Verbosity / Simulation Options: ### Verbosity / Simulation Options:
-q, --quiet activates quiet mode -q, --quiet activates quiet mode
-s, --simulate do not download video -s, --simulate do not download the video and do not write anything
to disk
--skip-download do not download the video
-g, --get-url simulate, quiet but print URL -g, --get-url simulate, quiet but print URL
-e, --get-title simulate, quiet but print title -e, --get-title simulate, quiet but print title
--get-thumbnail simulate, quiet but print thumbnail URL --get-thumbnail simulate, quiet but print thumbnail URL
--get-description simulate, quiet but print video description --get-description simulate, quiet but print video description
--get-filename simulate, quiet but print output filename --get-filename simulate, quiet but print output filename
--get-format simulate, quiet but print output format
--no-progress do not print progress bar --no-progress do not print progress bar
--console-title display progress in console titlebar --console-title display progress in console titlebar
@@ -61,6 +72,9 @@ which means you can modify it, redistribute it or use it however you like.
--extract-audio convert video files to audio-only files (requires --extract-audio convert video files to audio-only files (requires
ffmpeg and ffprobe) ffmpeg and ffprobe)
--audio-format FORMAT "best", "aac" or "mp3"; best by default --audio-format FORMAT "best", "aac" or "mp3"; best by default
--audio-quality QUALITY ffmpeg audio bitrate specification, 128k by default
-k, --keep-video keeps the video file on disk after the post-
processing; the video is erased by default
## FAQ ## FAQ
@@ -68,7 +82,7 @@ which means you can modify it, redistribute it or use it however you like.
Most people asking this question are not aware that youtube-dl now defaults to downloading the highest available quality as reported by YouTube, which will be 1080p or 720p in some cases, so you no longer need the -b option. For some specific videos, maybe YouTube does not report them to be available in a specific high quality format you''re interested in. In that case, simply request it with the -f option and youtube-dl will try to download it. Most people asking this question are not aware that youtube-dl now defaults to downloading the highest available quality as reported by YouTube, which will be 1080p or 720p in some cases, so you no longer need the -b option. For some specific videos, maybe YouTube does not report them to be available in a specific high quality format you''re interested in. In that case, simply request it with the -f option and youtube-dl will try to download it.
### I get HTTP error 402 when trying to download a video. What''s this? ### I get HTTP error 402 when trying to download a video. What's this?
Apparently YouTube requires you to pass a CAPTCHA test if you download too much. We''re [considering to provide a way to let you solve the CAPTCHA](https://github.com/rg3/youtube-dl/issues/154), but at the moment, your best course of action is pointing a webbrowser to the youtube URL, solving the CAPTCHA, and restart youtube-dl. Apparently YouTube requires you to pass a CAPTCHA test if you download too much. We''re [considering to provide a way to let you solve the CAPTCHA](https://github.com/rg3/youtube-dl/issues/154), but at the moment, your best course of action is pointing a webbrowser to the youtube URL, solving the CAPTCHA, and restart youtube-dl.

View File

@@ -15,7 +15,7 @@ __author__ = (
) )
__license__ = 'Public Domain' __license__ = 'Public Domain'
__version__ = '2011.09.14' __version__ = '2011.09.16'
UPDATE_URL = 'https://raw.github.com/rg3/youtube-dl/master/youtube-dl' UPDATE_URL = 'https://raw.github.com/rg3/youtube-dl/master/youtube-dl'
@@ -23,6 +23,7 @@ import cookielib
import datetime import datetime
import gzip import gzip
import htmlentitydefs import htmlentitydefs
import HTMLParser
import httplib import httplib
import locale import locale
import math import math
@@ -65,8 +66,8 @@ except ImportError:
try: try:
import xml.etree.ElementTree import xml.etree.ElementTree
except ImportError: # Python<2.5 except ImportError: # Python<2.5: Not officially supported, but let it slip
pass # Not officially supported, but let it slip warnings.warn('xml.etree.ElementTree support is missing. Consider upgrading to Python >= 2.5 if you get related errors.')
std_headers = { std_headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:5.0.1) Gecko/20100101 Firefox/5.0.1', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:5.0.1) Gecko/20100101 Firefox/5.0.1',
@@ -437,6 +438,8 @@ class FileDownloader(object):
noprogress: Do not print the progress bar. noprogress: Do not print the progress bar.
playliststart: Playlist item to start at. playliststart: Playlist item to start at.
playlistend: Playlist item to end at. playlistend: Playlist item to end at.
matchtitle: Download only matching titles.
rejecttitle: Reject downloads for matching titles.
logtostderr: Log messages to stderr instead of stdout. logtostderr: Log messages to stderr instead of stdout.
consoletitle: Display progress in console window's titlebar. consoletitle: Display progress in console window's titlebar.
nopart: Do not use temporary .part files. nopart: Do not use temporary .part files.
@@ -622,11 +625,12 @@ class FileDownloader(object):
return return
filetime = timeconvert(timestr) filetime = timeconvert(timestr)
if filetime is None: if filetime is None:
return return filetime
try: try:
os.utime(filename, (time.time(), filetime)) os.utime(filename, (time.time(), filetime))
except: except:
pass pass
return filetime
def report_writedescription(self, descfn): def report_writedescription(self, descfn):
""" Report that the description file is being written """ """ Report that the description file is being written """
@@ -694,8 +698,7 @@ class FileDownloader(object):
def process_info(self, info_dict): def process_info(self, info_dict):
"""Process a single dictionary returned by an InfoExtractor.""" """Process a single dictionary returned by an InfoExtractor."""
filename = self.prepare_filename(info_dict) filename = self.prepare_filename(info_dict)
# Do nothing else if in simulate mode
if self.params.get('simulate', False):
# Forced printings # Forced printings
if self.params.get('forcetitle', False): if self.params.get('forcetitle', False):
print info_dict['title'].encode(preferredencoding(), 'xmlcharrefreplace') print info_dict['title'].encode(preferredencoding(), 'xmlcharrefreplace')
@@ -707,11 +710,26 @@ class FileDownloader(object):
print info_dict['description'].encode(preferredencoding(), 'xmlcharrefreplace') print info_dict['description'].encode(preferredencoding(), 'xmlcharrefreplace')
if self.params.get('forcefilename', False) and filename is not None: if self.params.get('forcefilename', False) and filename is not None:
print filename.encode(preferredencoding(), 'xmlcharrefreplace') print filename.encode(preferredencoding(), 'xmlcharrefreplace')
if self.params.get('forceformat', False):
print info_dict['format'].encode(preferredencoding(), 'xmlcharrefreplace')
# Do nothing else if in simulate mode
if self.params.get('simulate', False):
return return
if filename is None: if filename is None:
return return
matchtitle=self.params.get('matchtitle',False)
rejecttitle=self.params.get('rejecttitle',False)
title=info_dict['title'].encode(preferredencoding(), 'xmlcharrefreplace')
if matchtitle and not re.search(matchtitle, title, re.IGNORECASE):
self.to_screen(u'[download] "%s" title did not match pattern "%s"' % (title, matchtitle))
return
if rejecttitle and re.search(rejecttitle, title, re.IGNORECASE):
self.to_screen(u'[download] "%s" title matched reject pattern "%s"' % (title, rejecttitle))
return
if self.params.get('nooverwrites', False) and os.path.exists(filename): if self.params.get('nooverwrites', False) and os.path.exists(filename):
self.to_stderr(u'WARNING: file exists and will be skipped') self.to_stderr(u'WARNING: file exists and will be skipped')
return return
@@ -755,8 +773,10 @@ class FileDownloader(object):
self.trouble(u'ERROR: Cannot write metadata to JSON file ' + infofn) self.trouble(u'ERROR: Cannot write metadata to JSON file ' + infofn)
return return
if not self.params.get('skip_download', False):
try: try:
success = self._do_download(filename, info_dict['url'].encode('utf-8'), info_dict.get('player_url', None)) success,add_data = self._do_download(filename, info_dict['url'].encode('utf-8'), info_dict.get('player_url', None))
info_dict.update(add_data)
except (OSError, IOError), err: except (OSError, IOError), err:
raise UnavailableVideoError raise UnavailableVideoError
except (urllib2.URLError, httplib.HTTPException, socket.error), err: except (urllib2.URLError, httplib.HTTPException, socket.error), err:
@@ -822,7 +842,7 @@ class FileDownloader(object):
# Download using rtmpdump. rtmpdump returns exit code 2 when # Download using rtmpdump. rtmpdump returns exit code 2 when
# the connection was interrumpted and resuming appears to be # the connection was interrumpted and resuming appears to be
# possible. This is part of rtmpdump's normal usage, AFAIK. # possible. This is part of rtmpdump's normal usage, AFAIK.
basic_args = ['rtmpdump'] + [[], ['-W', player_url]][player_url is not None] + ['-r', url, '-o', tmpfilename] basic_args = ['rtmpdump', '-q'] + [[], ['-W', player_url]][player_url is not None] + ['-r', url, '-o', tmpfilename]
retval = subprocess.call(basic_args + [[], ['-e', '-k', '1']][self.params.get('continuedl', False)]) retval = subprocess.call(basic_args + [[], ['-e', '-k', '1']][self.params.get('continuedl', False)])
while retval == 2 or retval == 1: while retval == 2 or retval == 1:
prevsize = os.path.getsize(tmpfilename) prevsize = os.path.getsize(tmpfilename)
@@ -832,6 +852,11 @@ class FileDownloader(object):
cursize = os.path.getsize(tmpfilename) cursize = os.path.getsize(tmpfilename)
if prevsize == cursize and retval == 1: if prevsize == cursize and retval == 1:
break break
# Some rtmp streams seem abort after ~ 99.8%. Don't complain for those
if prevsize == cursize and retval == 2 and cursize > 1024:
self.to_screen(u'\r[rtmpdump] Could not download the whole video. This can happen for some advertisements.')
retval = 0
break
if retval == 0: if retval == 0:
self.to_screen(u'\r[rtmpdump] %s bytes' % os.path.getsize(tmpfilename)) self.to_screen(u'\r[rtmpdump] %s bytes' % os.path.getsize(tmpfilename))
self.try_rename(tmpfilename, filename) self.try_rename(tmpfilename, filename)
@@ -971,10 +996,11 @@ class FileDownloader(object):
self.try_rename(tmpfilename, filename) self.try_rename(tmpfilename, filename)
# Update file modification time # Update file modification time
filetime = None
if self.params.get('updatetime', True): if self.params.get('updatetime', True):
self.try_utime(filename, data.info().get('last-modified', None)) filetime = self.try_utime(filename, data.info().get('last-modified', None))
return True return True, {'filetime': filetime}
class InfoExtractor(object): class InfoExtractor(object):
@@ -1007,9 +1033,8 @@ class InfoExtractor(object):
description: One-line video description. description: One-line video description.
Subclasses of this one should re-define the _real_initialize() and Subclasses of this one should re-define the _real_initialize() and
_real_extract() methods, as well as the suitable() static method. _real_extract() methods and define a _VALID_URL regexp.
Probably, they should also be instantiated and added to the main Probably, they should also be added to the list of extractors.
downloader.
""" """
_ready = False _ready = False
@@ -1020,10 +1045,9 @@ class InfoExtractor(object):
self._ready = False self._ready = False
self.set_downloader(downloader) self.set_downloader(downloader)
@staticmethod def suitable(self, url):
def suitable(url):
"""Receives a URL and returns True if suitable for this IE.""" """Receives a URL and returns True if suitable for this IE."""
return False return re.match(self._VALID_URL, url) is not None
def initialize(self): def initialize(self):
"""Initializes an instance (authentication, etc).""" """Initializes an instance (authentication, etc)."""
@@ -1069,10 +1093,7 @@ class YoutubeIE(InfoExtractor):
'43': 'webm', '43': 'webm',
'45': 'webm', '45': 'webm',
} }
IE_NAME = u'youtube'
@staticmethod
def suitable(url):
return (re.match(YoutubeIE._VALID_URL, url) is not None)
def report_lang(self): def report_lang(self):
"""Report attempt to set language.""" """Report attempt to set language."""
@@ -1299,16 +1320,24 @@ class YoutubeIE(InfoExtractor):
if len(existing_formats) == 0: if len(existing_formats) == 0:
self._downloader.trouble(u'ERROR: no known formats available for video') self._downloader.trouble(u'ERROR: no known formats available for video')
return return
if req_format is None: if req_format is None or req_format == 'best':
video_url_list = [(existing_formats[0], url_map[existing_formats[0]])] # Best quality video_url_list = [(existing_formats[0], url_map[existing_formats[0]])] # Best quality
elif req_format == '-1': elif req_format == 'worst':
video_url_list = [(existing_formats[len(existing_formats)-1], url_map[existing_formats[len(existing_formats)-1]])] # worst quality
elif req_format in ('-1', 'all'):
video_url_list = [(f, url_map[f]) for f in existing_formats] # All formats video_url_list = [(f, url_map[f]) for f in existing_formats] # All formats
else: else:
# Specific format # Specific formats. We pick the first in a slash-delimeted sequence.
if req_format not in url_map: # For example, if '1/2/3/4' is requested and '2' and '4' are available, we pick '2'.
req_formats = req_format.split('/')
video_url_list = None
for rf in req_formats:
if rf in url_map:
video_url_list = [(rf, url_map[rf])]
break
if video_url_list is None:
self._downloader.trouble(u'ERROR: requested format not available') self._downloader.trouble(u'ERROR: requested format not available')
return return
video_url_list = [(req_format, url_map[req_format])] # Specific format
else: else:
self._downloader.trouble(u'ERROR: no conn or url_encoded_fmt_stream_map information found in video info') self._downloader.trouble(u'ERROR: no conn or url_encoded_fmt_stream_map information found in video info')
return return
@@ -1346,15 +1375,12 @@ class MetacafeIE(InfoExtractor):
_DISCLAIMER = 'http://www.metacafe.com/family_filter/' _DISCLAIMER = 'http://www.metacafe.com/family_filter/'
_FILTER_POST = 'http://www.metacafe.com/f/index.php?inputType=filter&controllerGroup=user' _FILTER_POST = 'http://www.metacafe.com/f/index.php?inputType=filter&controllerGroup=user'
_youtube_ie = None _youtube_ie = None
IE_NAME = u'metacafe'
def __init__(self, youtube_ie, downloader=None): def __init__(self, youtube_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._youtube_ie = youtube_ie self._youtube_ie = youtube_ie
@staticmethod
def suitable(url):
return (re.match(MetacafeIE._VALID_URL, url) is not None)
def report_disclaimer(self): def report_disclaimer(self):
"""Report disclaimer retrieval.""" """Report disclaimer retrieval."""
self._downloader.to_screen(u'[metacafe] Retrieving disclaimer') self._downloader.to_screen(u'[metacafe] Retrieving disclaimer')
@@ -1488,14 +1514,11 @@ class DailymotionIE(InfoExtractor):
"""Information Extractor for Dailymotion""" """Information Extractor for Dailymotion"""
_VALID_URL = r'(?i)(?:https?://)?(?:www\.)?dailymotion\.[a-z]{2,3}/video/([^_/]+)_([^/]+)' _VALID_URL = r'(?i)(?:https?://)?(?:www\.)?dailymotion\.[a-z]{2,3}/video/([^_/]+)_([^/]+)'
IE_NAME = u'dailymotion'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(DailymotionIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[dailymotion] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[dailymotion] %s: Downloading webpage' % video_id)
@@ -1582,14 +1605,11 @@ class GoogleIE(InfoExtractor):
"""Information extractor for video.google.com.""" """Information extractor for video.google.com."""
_VALID_URL = r'(?:http://)?video\.google\.(?:com(?:\.au)?|co\.(?:uk|jp|kr|cr)|ca|de|es|fr|it|nl|pl)/videoplay\?docid=([^\&]+).*' _VALID_URL = r'(?:http://)?video\.google\.(?:com(?:\.au)?|co\.(?:uk|jp|kr|cr)|ca|de|es|fr|it|nl|pl)/videoplay\?docid=([^\&]+).*'
IE_NAME = u'video.google'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(GoogleIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[video.google] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[video.google] %s: Downloading webpage' % video_id)
@@ -1692,14 +1712,11 @@ class PhotobucketIE(InfoExtractor):
"""Information extractor for photobucket.com.""" """Information extractor for photobucket.com."""
_VALID_URL = r'(?:http://)?(?:[a-z0-9]+\.)?photobucket\.com/.*[\?\&]current=(.*\.flv)' _VALID_URL = r'(?:http://)?(?:[a-z0-9]+\.)?photobucket\.com/.*[\?\&]current=(.*\.flv)'
IE_NAME = u'photobucket'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(PhotobucketIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[photobucket] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[photobucket] %s: Downloading webpage' % video_id)
@@ -1777,14 +1794,11 @@ class YahooIE(InfoExtractor):
# _VPAGE_URL matches only the extractable '/watch/' URLs # _VPAGE_URL matches only the extractable '/watch/' URLs
_VALID_URL = r'(?:http://)?(?:[a-z]+\.)?video\.yahoo\.com/(?:watch|network)/([0-9]+)(?:/|\?v=)([0-9]+)(?:[#\?].*)?' _VALID_URL = r'(?:http://)?(?:[a-z]+\.)?video\.yahoo\.com/(?:watch|network)/([0-9]+)(?:/|\?v=)([0-9]+)(?:[#\?].*)?'
_VPAGE_URL = r'(?:http://)?video\.yahoo\.com/watch/([0-9]+)/([0-9]+)(?:[#\?].*)?' _VPAGE_URL = r'(?:http://)?video\.yahoo\.com/watch/([0-9]+)/([0-9]+)(?:[#\?].*)?'
IE_NAME = u'video.yahoo'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(YahooIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[video.yahoo] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[video.yahoo] %s: Downloading webpage' % video_id)
@@ -1933,14 +1947,11 @@ class VimeoIE(InfoExtractor):
# _VALID_URL matches Vimeo URLs # _VALID_URL matches Vimeo URLs
_VALID_URL = r'(?:https?://)?(?:(?:www|player).)?vimeo\.com/(?:groups/[^/]+/)?(?:videos?/)?([0-9]+)' _VALID_URL = r'(?:https?://)?(?:(?:www|player).)?vimeo\.com/(?:groups/[^/]+/)?(?:videos?/)?([0-9]+)'
IE_NAME = u'vimeo'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(VimeoIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[vimeo] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[vimeo] %s: Downloading webpage' % video_id)
@@ -2047,13 +2058,12 @@ class VimeoIE(InfoExtractor):
class GenericIE(InfoExtractor): class GenericIE(InfoExtractor):
"""Generic last-resort information extractor.""" """Generic last-resort information extractor."""
_VALID_URL = r'.*'
IE_NAME = u'generic'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return True
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'WARNING: Falling back on generic information extractor.') self._downloader.to_screen(u'WARNING: Falling back on generic information extractor.')
@@ -2147,21 +2157,18 @@ class GenericIE(InfoExtractor):
class YoutubeSearchIE(InfoExtractor): class YoutubeSearchIE(InfoExtractor):
"""Information Extractor for YouTube search queries.""" """Information Extractor for YouTube search queries."""
_VALID_QUERY = r'ytsearch(\d+|all)?:[\s\S]+' _VALID_URL = r'ytsearch(\d+|all)?:[\s\S]+'
_TEMPLATE_URL = 'http://www.youtube.com/results?search_query=%s&page=%s&gl=US&hl=en' _TEMPLATE_URL = 'http://www.youtube.com/results?search_query=%s&page=%s&gl=US&hl=en'
_VIDEO_INDICATOR = r'href="/watch\?v=.+?"' _VIDEO_INDICATOR = r'href="/watch\?v=.+?"'
_MORE_PAGES_INDICATOR = r'(?m)>\s*Next\s*</a>' _MORE_PAGES_INDICATOR = r'(?m)>\s*Next\s*</a>'
_youtube_ie = None _youtube_ie = None
_max_youtube_results = 1000 _max_youtube_results = 1000
IE_NAME = u'youtube:search'
def __init__(self, youtube_ie, downloader=None): def __init__(self, youtube_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._youtube_ie = youtube_ie self._youtube_ie = youtube_ie
@staticmethod
def suitable(url):
return (re.match(YoutubeSearchIE._VALID_QUERY, url) is not None)
def report_download_page(self, query, pagenum): def report_download_page(self, query, pagenum):
"""Report attempt to download playlist page with given number.""" """Report attempt to download playlist page with given number."""
query = query.decode(preferredencoding()) query = query.decode(preferredencoding())
@@ -2171,7 +2178,7 @@ class YoutubeSearchIE(InfoExtractor):
self._youtube_ie.initialize() self._youtube_ie.initialize()
def _real_extract(self, query): def _real_extract(self, query):
mobj = re.match(self._VALID_QUERY, query) mobj = re.match(self._VALID_URL, query)
if mobj is None: if mobj is None:
self._downloader.trouble(u'ERROR: invalid search query "%s"' % query) self._downloader.trouble(u'ERROR: invalid search query "%s"' % query)
return return
@@ -2239,21 +2246,18 @@ class YoutubeSearchIE(InfoExtractor):
class GoogleSearchIE(InfoExtractor): class GoogleSearchIE(InfoExtractor):
"""Information Extractor for Google Video search queries.""" """Information Extractor for Google Video search queries."""
_VALID_QUERY = r'gvsearch(\d+|all)?:[\s\S]+' _VALID_URL = r'gvsearch(\d+|all)?:[\s\S]+'
_TEMPLATE_URL = 'http://video.google.com/videosearch?q=%s+site:video.google.com&start=%s&hl=en' _TEMPLATE_URL = 'http://video.google.com/videosearch?q=%s+site:video.google.com&start=%s&hl=en'
_VIDEO_INDICATOR = r'videoplay\?docid=([^\&>]+)\&' _VIDEO_INDICATOR = r'videoplay\?docid=([^\&>]+)\&'
_MORE_PAGES_INDICATOR = r'<span>Next</span>' _MORE_PAGES_INDICATOR = r'<span>Next</span>'
_google_ie = None _google_ie = None
_max_google_results = 1000 _max_google_results = 1000
IE_NAME = u'video.google:search'
def __init__(self, google_ie, downloader=None): def __init__(self, google_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._google_ie = google_ie self._google_ie = google_ie
@staticmethod
def suitable(url):
return (re.match(GoogleSearchIE._VALID_QUERY, url) is not None)
def report_download_page(self, query, pagenum): def report_download_page(self, query, pagenum):
"""Report attempt to download playlist page with given number.""" """Report attempt to download playlist page with given number."""
query = query.decode(preferredencoding()) query = query.decode(preferredencoding())
@@ -2263,7 +2267,7 @@ class GoogleSearchIE(InfoExtractor):
self._google_ie.initialize() self._google_ie.initialize()
def _real_extract(self, query): def _real_extract(self, query):
mobj = re.match(self._VALID_QUERY, query) mobj = re.match(self._VALID_URL, query)
if mobj is None: if mobj is None:
self._downloader.trouble(u'ERROR: invalid search query "%s"' % query) self._downloader.trouble(u'ERROR: invalid search query "%s"' % query)
return return
@@ -2331,21 +2335,18 @@ class GoogleSearchIE(InfoExtractor):
class YahooSearchIE(InfoExtractor): class YahooSearchIE(InfoExtractor):
"""Information Extractor for Yahoo! Video search queries.""" """Information Extractor for Yahoo! Video search queries."""
_VALID_QUERY = r'yvsearch(\d+|all)?:[\s\S]+' _VALID_URL = r'yvsearch(\d+|all)?:[\s\S]+'
_TEMPLATE_URL = 'http://video.yahoo.com/search/?p=%s&o=%s' _TEMPLATE_URL = 'http://video.yahoo.com/search/?p=%s&o=%s'
_VIDEO_INDICATOR = r'href="http://video\.yahoo\.com/watch/([0-9]+/[0-9]+)"' _VIDEO_INDICATOR = r'href="http://video\.yahoo\.com/watch/([0-9]+/[0-9]+)"'
_MORE_PAGES_INDICATOR = r'\s*Next' _MORE_PAGES_INDICATOR = r'\s*Next'
_yahoo_ie = None _yahoo_ie = None
_max_yahoo_results = 1000 _max_yahoo_results = 1000
IE_NAME = u'video.yahoo:search'
def __init__(self, yahoo_ie, downloader=None): def __init__(self, yahoo_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._yahoo_ie = yahoo_ie self._yahoo_ie = yahoo_ie
@staticmethod
def suitable(url):
return (re.match(YahooSearchIE._VALID_QUERY, url) is not None)
def report_download_page(self, query, pagenum): def report_download_page(self, query, pagenum):
"""Report attempt to download playlist page with given number.""" """Report attempt to download playlist page with given number."""
query = query.decode(preferredencoding()) query = query.decode(preferredencoding())
@@ -2355,7 +2356,7 @@ class YahooSearchIE(InfoExtractor):
self._yahoo_ie.initialize() self._yahoo_ie.initialize()
def _real_extract(self, query): def _real_extract(self, query):
mobj = re.match(self._VALID_QUERY, query) mobj = re.match(self._VALID_URL, query)
if mobj is None: if mobj is None:
self._downloader.trouble(u'ERROR: invalid search query "%s"' % query) self._downloader.trouble(u'ERROR: invalid search query "%s"' % query)
return return
@@ -2429,15 +2430,12 @@ class YoutubePlaylistIE(InfoExtractor):
_VIDEO_INDICATOR = r'/watch\?v=(.+?)&' _VIDEO_INDICATOR = r'/watch\?v=(.+?)&'
_MORE_PAGES_INDICATOR = r'(?m)>\s*Next\s*</a>' _MORE_PAGES_INDICATOR = r'(?m)>\s*Next\s*</a>'
_youtube_ie = None _youtube_ie = None
IE_NAME = u'youtube:playlist'
def __init__(self, youtube_ie, downloader=None): def __init__(self, youtube_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._youtube_ie = youtube_ie self._youtube_ie = youtube_ie
@staticmethod
def suitable(url):
return (re.match(YoutubePlaylistIE._VALID_URL, url) is not None)
def report_download_page(self, playlist_id, pagenum): def report_download_page(self, playlist_id, pagenum):
"""Report attempt to download playlist page with given number.""" """Report attempt to download playlist page with given number."""
self._downloader.to_screen(u'[youtube] PL %s: Downloading page #%s' % (playlist_id, pagenum)) self._downloader.to_screen(u'[youtube] PL %s: Downloading page #%s' % (playlist_id, pagenum))
@@ -2507,15 +2505,12 @@ class YoutubeUserIE(InfoExtractor):
_GDATA_URL = 'http://gdata.youtube.com/feeds/api/users/%s/uploads?max-results=%d&start-index=%d' _GDATA_URL = 'http://gdata.youtube.com/feeds/api/users/%s/uploads?max-results=%d&start-index=%d'
_VIDEO_INDICATOR = r'/watch\?v=(.+?)&' _VIDEO_INDICATOR = r'/watch\?v=(.+?)&'
_youtube_ie = None _youtube_ie = None
IE_NAME = u'youtube:user'
def __init__(self, youtube_ie, downloader=None): def __init__(self, youtube_ie, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
self._youtube_ie = youtube_ie self._youtube_ie = youtube_ie
@staticmethod
def suitable(url):
return (re.match(YoutubeUserIE._VALID_URL, url) is not None)
def report_download_page(self, username, start_index): def report_download_page(self, username, start_index):
"""Report attempt to download user page.""" """Report attempt to download user page."""
self._downloader.to_screen(u'[youtube] user %s: Downloading video ids from %d to %d' % self._downloader.to_screen(u'[youtube] user %s: Downloading video ids from %d to %d' %
@@ -2593,14 +2588,11 @@ class DepositFilesIE(InfoExtractor):
"""Information extractor for depositfiles.com""" """Information extractor for depositfiles.com"""
_VALID_URL = r'(?:http://)?(?:\w+\.)?depositfiles.com/(?:../(?#locale))?files/(.+)' _VALID_URL = r'(?:http://)?(?:\w+\.)?depositfiles.com/(?:../(?#locale))?files/(.+)'
IE_NAME = u'DepositFiles'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(DepositFilesIE._VALID_URL, url) is not None)
def report_download_webpage(self, file_id): def report_download_webpage(self, file_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[DepositFiles] %s: Downloading webpage' % file_id) self._downloader.to_screen(u'[DepositFiles] %s: Downloading webpage' % file_id)
@@ -2680,14 +2672,11 @@ class FacebookIE(InfoExtractor):
'highqual': 'mp4', 'highqual': 'mp4',
'lowqual': 'mp4', 'lowqual': 'mp4',
} }
IE_NAME = u'facebook'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(FacebookIE._VALID_URL, url) is not None)
def _reporter(self, message): def _reporter(self, message):
"""Add header and report message.""" """Add header and report message."""
self._downloader.to_screen(u'[facebook] %s' % message) self._downloader.to_screen(u'[facebook] %s' % message)
@@ -2853,6 +2842,8 @@ class FacebookIE(InfoExtractor):
return return
if req_format is None: if req_format is None:
video_url_list = [(existing_formats[0], url_map[existing_formats[0]])] # Best quality video_url_list = [(existing_formats[0], url_map[existing_formats[0]])] # Best quality
elif req_format == 'worst':
video_url_list = [(existing_formats[len(existing_formats)-1], url_map[existing_formats[len(existing_formats)-1]])] # worst quality
elif req_format == '-1': elif req_format == '-1':
video_url_list = [(f, url_map[f]) for f in existing_formats] # All formats video_url_list = [(f, url_map[f]) for f in existing_formats] # All formats
else: else:
@@ -2893,10 +2884,7 @@ class BlipTVIE(InfoExtractor):
_VALID_URL = r'^(?:https?://)?(?:\w+\.)?blip\.tv(/.+)$' _VALID_URL = r'^(?:https?://)?(?:\w+\.)?blip\.tv(/.+)$'
_URL_EXT = r'^.*\.([a-z0-9]+)$' _URL_EXT = r'^.*\.([a-z0-9]+)$'
IE_NAME = u'blip.tv'
@staticmethod
def suitable(url):
return (re.match(BlipTVIE._VALID_URL, url) is not None)
def report_extraction(self, file_id): def report_extraction(self, file_id):
"""Report information extraction.""" """Report information extraction."""
@@ -2968,14 +2956,11 @@ class MyVideoIE(InfoExtractor):
"""Information Extractor for myvideo.de.""" """Information Extractor for myvideo.de."""
_VALID_URL = r'(?:http://)?(?:www\.)?myvideo\.de/watch/([0-9]+)/([^?/]+).*' _VALID_URL = r'(?:http://)?(?:www\.)?myvideo\.de/watch/([0-9]+)/([^?/]+).*'
IE_NAME = u'myvideo'
def __init__(self, downloader=None): def __init__(self, downloader=None):
InfoExtractor.__init__(self, downloader) InfoExtractor.__init__(self, downloader)
@staticmethod
def suitable(url):
return (re.match(MyVideoIE._VALID_URL, url) is not None)
def report_download_webpage(self, video_id): def report_download_webpage(self, video_id):
"""Report webpage download.""" """Report webpage download."""
self._downloader.to_screen(u'[myvideo] %s: Downloading webpage' % video_id) self._downloader.to_screen(u'[myvideo] %s: Downloading webpage' % video_id)
@@ -3043,11 +3028,8 @@ class MyVideoIE(InfoExtractor):
class ComedyCentralIE(InfoExtractor): class ComedyCentralIE(InfoExtractor):
"""Information extractor for The Daily Show and Colbert Report """ """Information extractor for The Daily Show and Colbert Report """
_VALID_URL = r'^(:(?P<shortname>tds|thedailyshow|cr|colbert|colbertnation|colbertreport))|(https?://)?(www\.)(?P<showname>thedailyshow|colbertnation)\.com/full-episodes/(?P<episode>.*)$' _VALID_URL = r'^(:(?P<shortname>tds|thedailyshow|cr|colbert|colbertnation|colbertreport))|(https?://)?(www\.)?(?P<showname>thedailyshow|colbertnation)\.com/full-episodes/(?P<episode>.*)$'
IE_NAME = u'comedycentral'
@staticmethod
def suitable(url):
return (re.match(ComedyCentralIE._VALID_URL, url) is not None)
def report_extraction(self, episode_id): def report_extraction(self, episode_id):
self._downloader.to_screen(u'[comedycentral] %s: Extracting information' % episode_id) self._downloader.to_screen(u'[comedycentral] %s: Extracting information' % episode_id)
@@ -3055,6 +3037,9 @@ class ComedyCentralIE(InfoExtractor):
def report_config_download(self, episode_id): def report_config_download(self, episode_id):
self._downloader.to_screen(u'[comedycentral] %s: Downloading configuration' % episode_id) self._downloader.to_screen(u'[comedycentral] %s: Downloading configuration' % episode_id)
def report_index_download(self, episode_id):
self._downloader.to_screen(u'[comedycentral] %s: Downloading show index' % episode_id)
def report_player_url(self, episode_id): def report_player_url(self, episode_id):
self._downloader.to_screen(u'[comedycentral] %s: Determining player URL' % episode_id) self._downloader.to_screen(u'[comedycentral] %s: Determining player URL' % episode_id)
@@ -3102,36 +3087,38 @@ class ComedyCentralIE(InfoExtractor):
return return
epTitle = mobj.group('episode') epTitle = mobj.group('episode')
mMovieParams = re.findall('<param name="movie" value="(http://media.mtvnservices.com/(.*?:episode:([^:]*):)(.*?))"/>', html) mMovieParams = re.findall('<param name="movie" value="(http://media.mtvnservices.com/([^"]*episode.*?:.*?))"/>', html)
if len(mMovieParams) == 0: if len(mMovieParams) == 0:
self._downloader.trouble(u'ERROR: unable to find Flash URL in webpage ' + url) self._downloader.trouble(u'ERROR: unable to find Flash URL in webpage ' + url)
return return
show_id = mMovieParams[0][2]
ACT_COUNT = { # TODO: Detect this dynamically
'thedailyshow.com': 4,
'colbertnation.com': 3,
}.get(show_id, 4)
OFFSET = {
'thedailyshow.com': 1,
'colbertnation.com': 1,
}.get(show_id, 1)
first_player_url = mMovieParams[0][0] playerUrl_raw = mMovieParams[0][0]
startMediaNum = int(mMovieParams[0][3]) + OFFSET
movieId = mMovieParams[0][1]
playerReq = urllib2.Request(first_player_url)
self.report_player_url(epTitle) self.report_player_url(epTitle)
try: try:
playerResponse = urllib2.urlopen(playerReq) urlHandle = urllib2.urlopen(playerUrl_raw)
playerUrl = urlHandle.geturl()
except (urllib2.URLError, httplib.HTTPException, socket.error), err: except (urllib2.URLError, httplib.HTTPException, socket.error), err:
self._downloader.trouble(u'ERROR: unable to download player: %s' % unicode(err)) self._downloader.trouble(u'ERROR: unable to find out player URL: ' + unicode(err))
return return
player_url = playerResponse.geturl()
for actNum in range(ACT_COUNT): uri = mMovieParams[0][1]
mediaNum = startMediaNum + actNum indexUrl = 'http://shadow.comedycentral.com/feeds/video_player/mrss/?' + urllib.urlencode({'uri': uri})
mediaId = movieId + str(mediaNum) self.report_index_download(epTitle)
try:
indexXml = urllib2.urlopen(indexUrl).read()
except (urllib2.URLError, httplib.HTTPException, socket.error), err:
self._downloader.trouble(u'ERROR: unable to download episode index: ' + unicode(err))
return
idoc = xml.etree.ElementTree.fromstring(indexXml)
itemEls = idoc.findall('.//item')
for itemEl in itemEls:
mediaId = itemEl.findall('./guid')[0].text
shortMediaId = mediaId.split(':')[-1]
showId = mediaId.split(':')[-2].replace('.com', '')
officialTitle = itemEl.findall('./title')[0].text
officialDate = itemEl.findall('./pubDate')[0].text
configUrl = ('http://www.comedycentral.com/global/feeds/entertainment/media/mediaGenEntertainment.jhtml?' + configUrl = ('http://www.comedycentral.com/global/feeds/entertainment/media/mediaGenEntertainment.jhtml?' +
urllib.urlencode({'uri': mediaId})) urllib.urlencode({'uri': mediaId}))
configReq = urllib2.Request(configUrl) configReq = urllib2.Request(configUrl)
@@ -3149,7 +3136,7 @@ class ComedyCentralIE(InfoExtractor):
turls.append(finfo) turls.append(finfo)
if len(turls) == 0: if len(turls) == 0:
self._downloader.trouble(u'\nERROR: unable to download ' + str(mediaNum) + ': No videos found') self._downloader.trouble(u'\nERROR: unable to download ' + mediaId + ': No videos found')
continue continue
# For now, just pick the highest bitrate # For now, just pick the highest bitrate
@@ -3157,28 +3144,112 @@ class ComedyCentralIE(InfoExtractor):
self._downloader.increment_downloads() self._downloader.increment_downloads()
effTitle = show_id.replace('.com', '') + '-' + epTitle effTitle = showId + '-' + epTitle
info = { info = {
'id': str(mediaNum), 'id': shortMediaId,
'url': video_url, 'url': video_url,
'uploader': show_id, 'uploader': showId,
'upload_date': 'NA', 'upload_date': officialDate,
'title': effTitle, 'title': effTitle,
'stitle': self._simplify_title(effTitle), 'stitle': self._simplify_title(effTitle),
'ext': 'mp4', 'ext': 'mp4',
'format': format, 'format': format,
'thumbnail': None, 'thumbnail': None,
'description': 'TODO: Not yet supported', 'description': officialTitle,
'player_url': player_url 'player_url': playerUrl
} }
try: try:
self._downloader.process_info(info) self._downloader.process_info(info)
except UnavailableVideoError, err: except UnavailableVideoError, err:
self._downloader.trouble(u'\nERROR: unable to download ' + str(mediaNum)) self._downloader.trouble(u'\nERROR: unable to download ' + mediaId)
continue continue
class EscapistIE(InfoExtractor):
"""Information extractor for The Escapist """
_VALID_URL = r'^(https?://)?(www\.)escapistmagazine.com/videos/view/(?P<showname>[^/]+)/(?P<episode>[^/?]+)[/?].*$'
IE_NAME = u'escapist'
def report_extraction(self, showName):
self._downloader.to_screen(u'[escapist] %s: Extracting information' % showName)
def report_config_download(self, showName):
self._downloader.to_screen(u'[escapist] %s: Downloading configuration' % showName)
def _simplify_title(self, title):
res = re.sub(ur'(?u)([^%s]+)' % simple_title_chars, ur'_', title)
res = res.strip(ur'_')
return res
def _real_extract(self, url):
htmlParser = HTMLParser.HTMLParser()
mobj = re.match(self._VALID_URL, url)
if mobj is None:
self._downloader.trouble(u'ERROR: invalid URL: %s' % url)
return
showName = mobj.group('showname')
videoId = mobj.group('episode')
self.report_extraction(showName)
try:
webPage = urllib2.urlopen(url).read()
except (urllib2.URLError, httplib.HTTPException, socket.error), err:
self._downloader.trouble(u'ERROR: unable to download webpage: ' + unicode(err))
return
descMatch = re.search('<meta name="description" content="([^"]*)"', webPage)
description = htmlParser.unescape(descMatch.group(1))
imgMatch = re.search('<meta property="og:image" content="([^"]*)"', webPage)
imgUrl = htmlParser.unescape(imgMatch.group(1))
playerUrlMatch = re.search('<meta property="og:video" content="([^"]*)"', webPage)
playerUrl = htmlParser.unescape(playerUrlMatch.group(1))
configUrlMatch = re.search('config=(.*)$', playerUrl)
configUrl = urllib2.unquote(configUrlMatch.group(1))
self.report_config_download(showName)
try:
configJSON = urllib2.urlopen(configUrl).read()
except (urllib2.URLError, httplib.HTTPException, socket.error), err:
self._downloader.trouble(u'ERROR: unable to download configuration: ' + unicode(err))
return
# Technically, it's JavaScript, not JSON
configJSON = configJSON.replace("'", '"')
try:
config = json.loads(configJSON)
except (ValueError,), err:
self._downloader.trouble(u'ERROR: Invalid JSON in configuration file: ' + unicode(err))
return
playlist = config['playlist']
videoUrl = playlist[1]['url']
self._downloader.increment_downloads()
info = {
'id': videoId,
'url': videoUrl,
'uploader': showName,
'upload_date': None,
'title': showName,
'stitle': self._simplify_title(showName),
'ext': 'flv',
'format': 'flv',
'thumbnail': imgUrl,
'description': description,
'player_url': playerUrl,
}
try:
self._downloader.process_info(info)
except UnavailableVideoError, err:
self._downloader.trouble(u'\nERROR: unable to download ' + videoId)
class PostProcessor(object): class PostProcessor(object):
"""Post Processor class. """Post Processor class.
@@ -3228,11 +3299,13 @@ class PostProcessor(object):
class FFmpegExtractAudioPP(PostProcessor): class FFmpegExtractAudioPP(PostProcessor):
def __init__(self, downloader=None, preferredcodec=None): def __init__(self, downloader=None, preferredcodec=None, preferredquality=None, keepvideo=False):
PostProcessor.__init__(self, downloader) PostProcessor.__init__(self, downloader)
if preferredcodec is None: if preferredcodec is None:
preferredcodec = 'best' preferredcodec = 'best'
self._preferredcodec = preferredcodec self._preferredcodec = preferredcodec
self._preferredquality = preferredquality
self._keepvideo = keepvideo
@staticmethod @staticmethod
def get_audio_codec(path): def get_audio_codec(path):
@@ -3281,12 +3354,16 @@ class FFmpegExtractAudioPP(PostProcessor):
# MP3 otherwise. # MP3 otherwise.
acodec = 'libmp3lame' acodec = 'libmp3lame'
extension = 'mp3' extension = 'mp3'
more_opts = ['-ab', '128k'] more_opts = []
if self._preferredquality is not None:
more_opts += ['-ab', self._preferredquality]
else: else:
# We convert the audio (lossy) # We convert the audio (lossy)
acodec = {'mp3': 'libmp3lame', 'aac': 'aac'}[self._preferredcodec] acodec = {'mp3': 'libmp3lame', 'aac': 'aac'}[self._preferredcodec]
extension = self._preferredcodec extension = self._preferredcodec
more_opts = ['-ab', '128k'] more_opts = []
if self._preferredquality is not None:
more_opts += ['-ab', self._preferredquality]
if self._preferredcodec == 'aac': if self._preferredcodec == 'aac':
more_opts += ['-f', 'adts'] more_opts += ['-f', 'adts']
@@ -3299,6 +3376,14 @@ class FFmpegExtractAudioPP(PostProcessor):
self._downloader.to_stderr(u'WARNING: error running ffmpeg') self._downloader.to_stderr(u'WARNING: error running ffmpeg')
return None return None
# Try to update the date time for extracted audio file.
if information.get('filetime') is not None:
try:
os.utime(new_path, (time.time(), information['filetime']))
except:
self._downloader.to_stderr(u'WARNING: Cannot update utime of audio file')
if not self._keepvideo:
try: try:
os.remove(path) os.remove(path)
except (IOError, OSError): except (IOError, OSError):
@@ -3381,7 +3466,7 @@ def parseOpts():
kw = { kw = {
'version' : __version__, 'version' : __version__,
'formatter' : fmt, 'formatter' : fmt,
'usage' : '%prog [options] url...', 'usage' : '%prog [options] url [url...]',
'conflict_handler' : 'resolve', 'conflict_handler' : 'resolve',
} }
@@ -3389,6 +3474,7 @@ def parseOpts():
# option groups # option groups
general = optparse.OptionGroup(parser, 'General Options') general = optparse.OptionGroup(parser, 'General Options')
selection = optparse.OptionGroup(parser, 'Video Selection')
authentication = optparse.OptionGroup(parser, 'Authentication Options') authentication = optparse.OptionGroup(parser, 'Authentication Options')
video_format = optparse.OptionGroup(parser, 'Video Format Options') video_format = optparse.OptionGroup(parser, 'Video Format Options')
postproc = optparse.OptionGroup(parser, 'Post-processing Options') postproc = optparse.OptionGroup(parser, 'Post-processing Options')
@@ -3407,13 +3493,19 @@ def parseOpts():
dest='ratelimit', metavar='LIMIT', help='download rate limit (e.g. 50k or 44.6m)') dest='ratelimit', metavar='LIMIT', help='download rate limit (e.g. 50k or 44.6m)')
general.add_option('-R', '--retries', general.add_option('-R', '--retries',
dest='retries', metavar='RETRIES', help='number of retries (default is 10)', default=10) dest='retries', metavar='RETRIES', help='number of retries (default is 10)', default=10)
general.add_option('--playlist-start',
dest='playliststart', metavar='NUMBER', help='playlist video to start at (default is 1)', default=1)
general.add_option('--playlist-end',
dest='playlistend', metavar='NUMBER', help='playlist video to end at (default is last)', default=-1)
general.add_option('--dump-user-agent', general.add_option('--dump-user-agent',
action='store_true', dest='dump_user_agent', action='store_true', dest='dump_user_agent',
help='display the current browser identification', default=False) help='display the current browser identification', default=False)
general.add_option('--list-extractors',
action='store_true', dest='list_extractors',
help='List all supported extractors and the URLs they would handle', default=False)
selection.add_option('--playlist-start',
dest='playliststart', metavar='NUMBER', help='playlist video to start at (default is 1)', default=1)
selection.add_option('--playlist-end',
dest='playlistend', metavar='NUMBER', help='playlist video to end at (default is last)', default=-1)
selection.add_option('--match-title', dest='matchtitle', metavar='REGEX',help='download only matching titles (regex or caseless sub-string)')
selection.add_option('--reject-title', dest='rejecttitle', metavar='REGEX',help='skip download for matching titles (regex or caseless sub-string)')
authentication.add_option('-u', '--username', authentication.add_option('-u', '--username',
dest='username', metavar='USERNAME', help='account username') dest='username', metavar='USERNAME', help='account username')
@@ -3426,7 +3518,7 @@ def parseOpts():
video_format.add_option('-f', '--format', video_format.add_option('-f', '--format',
action='store', dest='format', metavar='FORMAT', help='video format code') action='store', dest='format', metavar='FORMAT', help='video format code')
video_format.add_option('--all-formats', video_format.add_option('--all-formats',
action='store_const', dest='format', help='download all available video formats', const='-1') action='store_const', dest='format', help='download all available video formats', const='all')
video_format.add_option('--max-quality', video_format.add_option('--max-quality',
action='store', dest='format_limit', metavar='FORMAT', help='highest quality format to download') action='store', dest='format_limit', metavar='FORMAT', help='highest quality format to download')
@@ -3434,7 +3526,9 @@ def parseOpts():
verbosity.add_option('-q', '--quiet', verbosity.add_option('-q', '--quiet',
action='store_true', dest='quiet', help='activates quiet mode', default=False) action='store_true', dest='quiet', help='activates quiet mode', default=False)
verbosity.add_option('-s', '--simulate', verbosity.add_option('-s', '--simulate',
action='store_true', dest='simulate', help='do not download video', default=False) action='store_true', dest='simulate', help='do not download the video and do not write anything to disk', default=False)
verbosity.add_option('--skip-download',
action='store_true', dest='skip_download', help='do not download the video', default=False)
verbosity.add_option('-g', '--get-url', verbosity.add_option('-g', '--get-url',
action='store_true', dest='geturl', help='simulate, quiet but print URL', default=False) action='store_true', dest='geturl', help='simulate, quiet but print URL', default=False)
verbosity.add_option('-e', '--get-title', verbosity.add_option('-e', '--get-title',
@@ -3448,6 +3542,9 @@ def parseOpts():
verbosity.add_option('--get-filename', verbosity.add_option('--get-filename',
action='store_true', dest='getfilename', action='store_true', dest='getfilename',
help='simulate, quiet but print output filename', default=False) help='simulate, quiet but print output filename', default=False)
verbosity.add_option('--get-format',
action='store_true', dest='getformat',
help='simulate, quiet but print output format', default=False)
verbosity.add_option('--no-progress', verbosity.add_option('--no-progress',
action='store_true', dest='noprogress', help='do not print progress bar', default=False) action='store_true', dest='noprogress', help='do not print progress bar', default=False)
verbosity.add_option('--console-title', verbosity.add_option('--console-title',
@@ -3489,9 +3586,14 @@ def parseOpts():
help='convert video files to audio-only files (requires ffmpeg and ffprobe)') help='convert video files to audio-only files (requires ffmpeg and ffprobe)')
postproc.add_option('--audio-format', metavar='FORMAT', dest='audioformat', default='best', postproc.add_option('--audio-format', metavar='FORMAT', dest='audioformat', default='best',
help='"best", "aac" or "mp3"; best by default') help='"best", "aac" or "mp3"; best by default')
postproc.add_option('--audio-quality', metavar='QUALITY', dest='audioquality', default='128K',
help='ffmpeg audio bitrate specification, 128k by default')
postproc.add_option('-k', '--keep-video', action='store_true', dest='keepvideo', default=False,
help='keeps the video file on disk after the post-processing; the video is erased by default')
parser.add_option_group(general) parser.add_option_group(general)
parser.add_option_group(selection)
parser.add_option_group(filesystem) parser.add_option_group(filesystem)
parser.add_option_group(verbosity) parser.add_option_group(verbosity)
parser.add_option_group(video_format) parser.add_option_group(video_format)
@@ -3502,6 +3604,36 @@ def parseOpts():
return parser, opts, args return parser, opts, args
def gen_extractors():
""" Return a list of an instance of every supported extractor.
The order does matter; the first extractor matched is the one handling the URL.
"""
youtube_ie = YoutubeIE()
google_ie = GoogleIE()
yahoo_ie = YahooIE()
return [
youtube_ie,
MetacafeIE(youtube_ie),
DailymotionIE(),
YoutubePlaylistIE(youtube_ie),
YoutubeUserIE(youtube_ie),
YoutubeSearchIE(youtube_ie),
google_ie,
GoogleSearchIE(google_ie),
PhotobucketIE(),
yahoo_ie,
YahooSearchIE(yahoo_ie),
DepositFilesIE(),
FacebookIE(),
BlipTVIE(),
VimeoIE(),
MyVideoIE(),
ComedyCentralIE(),
EscapistIE(),
GenericIE()
]
def main(): def main():
parser, opts, args = parseOpts() parser, opts, args = parseOpts()
@@ -3521,12 +3653,6 @@ def main():
print std_headers['User-Agent'] print std_headers['User-Agent']
sys.exit(0) sys.exit(0)
# General configuration
cookie_processor = urllib2.HTTPCookieProcessor(jar)
opener = urllib2.build_opener(urllib2.ProxyHandler(), cookie_processor, YoutubeDLHandler())
urllib2.install_opener(opener)
socket.setdefaulttimeout(300) # 5 minutes should be enough (famous last words)
# Batch file verification # Batch file verification
batchurls = [] batchurls = []
if opts.batchfile is not None: if opts.batchfile is not None:
@@ -3542,6 +3668,23 @@ def main():
sys.exit(u'ERROR: batch file could not be read') sys.exit(u'ERROR: batch file could not be read')
all_urls = batchurls + args all_urls = batchurls + args
# General configuration
cookie_processor = urllib2.HTTPCookieProcessor(jar)
opener = urllib2.build_opener(urllib2.ProxyHandler(), cookie_processor, YoutubeDLHandler())
urllib2.install_opener(opener)
socket.setdefaulttimeout(300) # 5 minutes should be enough (famous last words)
extractors = gen_extractors()
if opts.list_extractors:
for ie in extractors:
print(ie.IE_NAME)
matchedUrls = filter(lambda url: ie.suitable(url), all_urls)
all_urls = filter(lambda url: url not in matchedUrls, all_urls)
for mu in matchedUrls:
print(u' ' + mu)
sys.exit(0)
# Conflicting, missing and erroneous options # Conflicting, missing and erroneous options
if opts.usenetrc and (opts.username is not None or opts.password is not None): if opts.usenetrc and (opts.username is not None or opts.password is not None):
parser.error(u'using .netrc conflicts with giving username/password') parser.error(u'using .netrc conflicts with giving username/password')
@@ -3579,39 +3722,20 @@ def main():
if opts.audioformat not in ['best', 'aac', 'mp3']: if opts.audioformat not in ['best', 'aac', 'mp3']:
parser.error(u'invalid audio format specified') parser.error(u'invalid audio format specified')
# Information extractors
youtube_ie = YoutubeIE()
metacafe_ie = MetacafeIE(youtube_ie)
dailymotion_ie = DailymotionIE()
youtube_pl_ie = YoutubePlaylistIE(youtube_ie)
youtube_user_ie = YoutubeUserIE(youtube_ie)
youtube_search_ie = YoutubeSearchIE(youtube_ie)
google_ie = GoogleIE()
google_search_ie = GoogleSearchIE(google_ie)
photobucket_ie = PhotobucketIE()
yahoo_ie = YahooIE()
yahoo_search_ie = YahooSearchIE(yahoo_ie)
deposit_files_ie = DepositFilesIE()
facebook_ie = FacebookIE()
bliptv_ie = BlipTVIE()
vimeo_ie = VimeoIE()
myvideo_ie = MyVideoIE()
comedycentral_ie = ComedyCentralIE()
generic_ie = GenericIE()
# File downloader # File downloader
fd = FileDownloader({ fd = FileDownloader({
'usenetrc': opts.usenetrc, 'usenetrc': opts.usenetrc,
'username': opts.username, 'username': opts.username,
'password': opts.password, 'password': opts.password,
'quiet': (opts.quiet or opts.geturl or opts.gettitle or opts.getthumbnail or opts.getdescription or opts.getfilename), 'quiet': (opts.quiet or opts.geturl or opts.gettitle or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat),
'forceurl': opts.geturl, 'forceurl': opts.geturl,
'forcetitle': opts.gettitle, 'forcetitle': opts.gettitle,
'forcethumbnail': opts.getthumbnail, 'forcethumbnail': opts.getthumbnail,
'forcedescription': opts.getdescription, 'forcedescription': opts.getdescription,
'forcefilename': opts.getfilename, 'forcefilename': opts.getfilename,
'simulate': (opts.simulate or opts.geturl or opts.gettitle or opts.getthumbnail or opts.getdescription or opts.getfilename), 'forceformat': opts.getformat,
'simulate': opts.simulate,
'skip_download': (opts.skip_download or opts.simulate or opts.geturl or opts.gettitle or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat),
'format': opts.format, 'format': opts.format,
'format_limit': opts.format_limit, 'format_limit': opts.format_limit,
'outtmpl': ((opts.outtmpl is not None and opts.outtmpl.decode(preferredencoding())) 'outtmpl': ((opts.outtmpl is not None and opts.outtmpl.decode(preferredencoding()))
@@ -3638,32 +3762,15 @@ def main():
'updatetime': opts.updatetime, 'updatetime': opts.updatetime,
'writedescription': opts.writedescription, 'writedescription': opts.writedescription,
'writeinfojson': opts.writeinfojson, 'writeinfojson': opts.writeinfojson,
'matchtitle': opts.matchtitle,
'rejecttitle': opts.rejecttitle,
}) })
fd.add_info_extractor(youtube_search_ie) for extractor in extractors:
fd.add_info_extractor(youtube_pl_ie) fd.add_info_extractor(extractor)
fd.add_info_extractor(youtube_user_ie)
fd.add_info_extractor(metacafe_ie)
fd.add_info_extractor(dailymotion_ie)
fd.add_info_extractor(youtube_ie)
fd.add_info_extractor(google_ie)
fd.add_info_extractor(google_search_ie)
fd.add_info_extractor(photobucket_ie)
fd.add_info_extractor(yahoo_ie)
fd.add_info_extractor(yahoo_search_ie)
fd.add_info_extractor(deposit_files_ie)
fd.add_info_extractor(facebook_ie)
fd.add_info_extractor(bliptv_ie)
fd.add_info_extractor(vimeo_ie)
fd.add_info_extractor(myvideo_ie)
fd.add_info_extractor(comedycentral_ie)
# This must come last since it's the
# fallback if none of the others work
fd.add_info_extractor(generic_ie)
# PostProcessors # PostProcessors
if opts.extractaudio: if opts.extractaudio:
fd.add_post_processor(FFmpegExtractAudioPP(preferredcodec=opts.audioformat)) fd.add_post_processor(FFmpegExtractAudioPP(preferredcodec=opts.audioformat, preferredquality=opts.audioquality, keepvideo=opts.keepvideo))
# Update version # Update version
if opts.update_self: if opts.update_self: