release 2016.09.03

[ChangeLog] Actualize
Credit @C4K3 for #10536
2016-09-03 01:46:41 +07:00 · 2016-09-03 01:45:17 +07:00 · 2016-09-03 01:42:33 +07:00 · 2016-09-03 01:17:15 +07:00 · 2016-09-02 23:53:17 +07:00 · 2016-09-02 23:50:42 +07:00
153 changed files with 4845 additions and 2691 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -6,8 +6,8 @@
 ---
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.08.07*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.09.03*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.08.07**
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.09.03**
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
@@ -35,7 +35,7 @@ $ youtube-dl -v <your command line>
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2016.08.07
+[debug] youtube-dl version 2016.09.03
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/4
+++ b/4
@@ -179,3 +179,7 @@ Jakub Adam Wieczorek
 Aleksandar Topuzović
 Nehal Patel
 Rob van Bekkum
 Petr Zvoníček
 Pratyush Singh
 Aleksander Nitecki
 Sebastian Blunt
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -46,7 +46,7 @@ Make sure that someone has not already opened the issue you're trying to open. S
 ###  Why are existing options not enough?
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#synopsis). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
 ###  Is there enough context in your bug report?
--- a/227
+++ b/227
@@ -1,3 +1,230 @@
 version 2016.09.03
 Core
 * Restore usage of NAME attribute from EXT-X-MEDIA tag for formats codes in
  _extract_m3u8_formats (#10522)
 * Handle semicolon in mimetype2ext
 Extractors
 + [youtube] Add support for rental videos' previews (#10532)
 * [youtube:playlist] Fallback to video extraction for video/playlist URLs when
  no playlist is actually served (#10537)
 + [drtv] Add support for dr.dk/nyheder (#10536)
 + [facebook:plugins:video] Add extractor (#10530)
 + [go] Add extractor for *.go.com sites
 * [adobepass] Check for authz_token expiration (#10527)
 * [nytimes] improve extraction
 * [thestar] Fix extraction (#10465)
 * [glide] Fix extraction (#10478)
 - [exfm] Remove extractor (#10482)
 * [youporn] Fix categories and tags extraction (#10521)
 + [curiositystream] Add extractor for app.curiositystream.com
 - [thvideo] Remove extractor (#10464)
 * [movingimage] Fix for the new site name (#10466)
 + [cbs] Add support for once formats (#10515)
 * [limelight] Skip ism snd duplicate manifests
 + [porncom] Extract categories and tags (#10510)
 + [facebook] Extract timestamp (#10508)
 + [yahoo] Extract more formats
 version 2016.08.31
 Extractors
 * [soundcloud] Fix URL regular expression to avoid clashes with sets (#10505)
 * [bandcamp:album] Fix title extraction (#10455)
 * [pyvideo] Fix extraction (#10468)
 + [ctv] Add support for tsn.ca, bnn.ca and thecomedynetwork.ca (#10016)
 * [9c9media] Extract more metadata
 * [9c9media] Fix multiple stacks extraction (#10016)
 * [adultswim] Improve video info extraction (#10492)
 * [vodplatform] Improve embed regular expression
 - [played] Remove extractor (#10470)
 + [tbs] Add extractor for tbs.com and tntdrama.com (#10222)
 + [cartoonnetwork] Add extractor for cartoonnetwork.com (#10110)
 * [adultswim] Rework in terms of turner extractor
 * [cnn] Rework in terms of turner extractor
 * [nba] Rework in terms of turner extractor
 + [turner] Add base extractor for Turner Broadcasting System based sites
 * [bilibili] Fix extraction (#10375)
 * [openload] Fix extraction (#10408)
 version 2016.08.28
 Core
 + Add warning message that ffmpeg doesn't support SOCKS
 * Improve thumbnail sorting
 + Extract formats from #EXT-X-MEDIA tags in _extract_m3u8_formats
 * Fill IV with leading zeros for IVs shorter than 16 octets in hlsnative
 + Add ac-3 to the list of audio codecs in parse_codecs
 Extractors
 * [periscope:user] Fix extraction (#10453)
 * [douyutv] Fix extraction (#10153, #10318, #10444)
 + [nhk:vod] Add extractor for www3.nhk.or.jp on demand (#4437, #10424)
 - [trutube] Remove extractor (#10438)
 + [usanetwork] Add extractor for usanetwork.com
 * [crackle] Fix extraction (#10333)
 * [spankbang] Fix description and uploader extraction (#10339)
 * [discoverygo] Detect cable provider restricted videos (#10425)
 + [cbc] Add support for watch.cbc.ca
 * [kickstarter] Silent the warning for og:description (#10415)
 * [mtvservices:embedded] Fix extraction for the new 'edge' player (#10363)
 version 2016.08.24.1
 Extractors
 + [pluralsight] Add support for subtitles (#9681)
 version 2016.08.24
 Extractors
 * [youtube] Fix authentication (#10392)
 * [openload] Fix extraction (#10408)
 + [bravotv] Add support for Adobe Pass (#10407)
 * [bravotv] Fix clip info extraction (#10407)
 * [eagleplatform] Improve embedded videos detection (#10409)
 * [awaan] Fix extraction
 * [mtvservices:embedded] Update config URL
 + [abc:iview] Add extractor (#6148)
 version 2016.08.22
 Core
 * Improve formats and subtitles extension auto calculation
 + Recognize full unit names in parse_filesize
 + Add support for m3u8 manifests in HTML5 multimedia tags
 * Fix octal/hexadecimal number detection in js_to_json
 Extractors
 + [ivi] Add support for 720p and 1080p
 + [charlierose] Add new extractor (#10382)
 * [1tv] Fix extraction (#9249)
 * [twitch] Renew authentication
 * [kaltura] Improve subtitles extension calculation
 + [zingmp3] Add support for video clips
 * [zingmp3] Fix extraction (#10041)
 * [kaltura] Improve subtitles extraction (#10279)
 * [cultureunplugged] Fix extraction (#10330)
 + [cnn] Add support for money.cnn.com (#2797)
 * [cbsnews] Fix extraction (#10362)
 * [cbs] Fix extraction (#10393)
 + [litv] Support 'promo' URLs (#10385)
 * [snotr] Fix extraction (#10338)
 * [n-tv.de] Fix extraction (#10331)
 * [globo:article] Relax URL and video id regular expressions (#10379)
 version 2016.08.19
 Core
 - Remove output template description from --help
 * Recognize lowercase units in parse_filesize
 Extractors
 + [porncom] Add extractor for porn.com (#2251, #10251)
 + [generic] Add support for DBTV embeds
 * [vk:wallpost] Fix audio extraction for new site layout
 * [vk] Fix authentication
 + [hgtvcom:show] Add extractor for hgtv.com shows (#10365)
 + [discoverygo] Add support for another GO network sites
 version 2016.08.17
 Core
 + Add _get_netrc_login_info
 Extractors
 * [mofosex] Extract all formats (#10335)
 + [generic] Add support for vbox7 embeds
 + [vbox7] Add support for embed URLs
 + [viafree] Add extractor (#10358)
 + [mtg] Add support for viafree URLs (#10358)
 * [theplatform] Extract all subtitles per language
 + [xvideos] Fix HLS extraction (#10356)
 + [amcnetworks] Add extractor
 + [bbc:playlist] Add support for pagination (#10349)
 + [fxnetworks] Add extractor (#9462)
 * [cbslocal] Fix extraction for SendtoNews-based videos
 * [sendtonews] Fix extraction
 * [jwplatform] Extract video id from JWPlayer data
 - [zippcast] Remove extractor (#10332)
 + [viceland] Add extractor (#8799)
 + [adobepass] Add base extractor for Adobe Pass Authentication
 * [life:embed] Improve extraction
 * [vgtv] Detect geo restricted videos (#10348)
 + [uplynk] Add extractor
 * [xiami] Fix extraction (#10342)
 version 2016.08.13
 Core
 * Show progress for curl external downloader
 * Forward more options to curl external downloader
 Extractors
 * [pbs] Fix description extraction
 * [franceculture] Fix extraction (#10324)
 * [pornotube] Fix extraction (#10322)
 * [4tube] Fix metadata extraction (#10321)
 * [imgur] Fix width and height extraction (#10325)
 * [expotv] Improve extraction
 + [vbox7] Fix extraction (#10309)
 - [tapely] Remove extractor (#10323)
 * [muenchentv] Fix extraction (#10313)
 + [24video] Add support for .me and .xxx TLDs
 * [24video] Fix comment count extraction
 * [sunporno] Add support for embed URLs
 * [sunporno] Fix metadata extraction (#10316)
 + [hgtv] Add extractor for hgtv.ca (#3999)
 - [pbs] Remove request to unavailable API
 + [pbs] Add support for high quality HTTP formats
 + [crunchyroll] Add support for HLS formats (#10301)
 version 2016.08.12
 Core
 * Subtitles are now written as is. Newline conversions are disabled. (#10268)
 + Recognize more formats in unified_timestamp
 Extractors
 - [goldenmoustache] Remove extractor (#10298)
 * [drtuber] Improve title extraction
 * [drtuber] Make dislike count optional (#10297)
 * [chirbit] Fix extraction (#10296)
 * [francetvinfo] Relax URL regular expression
 * [rtlnl] Relax URL regular expression (#10282)
 * [formula1] Relax URL regular expression (#10283)
 * [wat] Improve extraction (#10281)
 * [ctsnews] Fix extraction
 version 2016.08.10
 Core
 * Make --metadata-from-title non fatal when title does not match the pattern
 * Introduce options for randomized sleep before each download
  --min-sleep-interval and --max-sleep-interval (#9930)
 * Respect default in _search_json_ld
 Extractors
 + [uol] Add extractor for uol.com.br (#4263)
 * [rbmaradio] Fix extraction and extract all formats (#10242)
 + [sonyliv] Add extractor for sonyliv.com (#10258)
 * [aparat] Fix extraction
 * [cwtv] Extract HTTP formats
 + [rozhlas] Add extractor for prehravac.rozhlas.cz (#10253)
 * [kuwo:singer] Fix extraction
 version 2016.08.07
 Core
--- a/README.md
+++ b/README.md
@@ -201,32 +201,8 @@ which means you can modify it, redistribute it or use it however you like.
    -a, --batch-file FILE            File containing URLs to download ('-' for
                                     stdin)
    --id                             Use only video ID in file name
-    -o, --output TEMPLATE            Output filename template. Use %(title)s to
+    -o, --output TEMPLATE            Output filename template, see the "OUTPUT
-                                     get the title, %(uploader)s for the
+                                     TEMPLATE" for all the info
                                     uploader name, %(uploader_id)s for the
                                     uploader nickname if different,
                                     %(autonumber)s to get an automatically
                                     incremented number, %(ext)s for the
                                     filename extension, %(format)s for the
                                     format description (like "22 - 1280x720" or
                                     "HD"), %(format_id)s for the unique id of
                                     the format (like YouTube's itags: "137"),
                                     %(upload_date)s for the upload date
                                     (YYYYMMDD), %(extractor)s for the provider
                                     (youtube, metacafe, etc), %(id)s for the
                                     video id, %(playlist_title)s,
                                     %(playlist_id)s, or %(playlist)s (=title if
                                     present, ID otherwise) for the playlist the
                                     video is in, %(playlist_index)s for the
                                     position in the playlist. %(height)s and
                                     %(width)s for the width and height of the
                                     video format. %(resolution)s for a textual
                                     description of the resolution of the video
                                     format. %% for a literal percent. Use - to
                                     output to stdout. Can also be used to
                                     download to a different directory, for
                                     example with -o '/my/downloads/%(uploader)s
                                     /%(title)s-%(id)s.%(ext)s' .
    --autonumber-size NUMBER         Specify the number of digits in
                                     %(autonumber)s when it is present in output
                                     filename template or --auto-number option
@@ -330,7 +306,15 @@ which means you can modify it, redistribute it or use it however you like.
                                     bidirectional text support. Requires bidiv
                                     or fribidi executable in PATH
    --sleep-interval SECONDS         Number of seconds to sleep before each
-                                     download.
+                                     download when used alone or a lower bound
                                     of a range for randomized sleep before each
                                     download (minimum possible number of
                                     seconds to sleep) when used along with
                                     --max-sleep-interval.
    --max-sleep-interval SECONDS     Upper bound of a range for randomized sleep
                                     before each download (maximum possible
                                     number of seconds to sleep). Must only be
                                     used along with --min-sleep-interval.
 ## Video Format Options:
    -f, --format FORMAT              Video format code, see the "FORMAT
@@ -428,11 +412,19 @@ You can configure youtube-dl by placing any supported command line option to a c
 For example, with the following configuration file youtube-dl will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
 ```
 -x
 --no-mtime
 --proxy 127.0.0.1:3128
 -o ~/Movies/%(title)s.%(ext)s
 # Lines starting with # are comments
 # Always extract audio
 -x
 # Do not copy the mtime
 --no-mtime
 # Use this proxy
 --proxy 127.0.0.1:3128
 # Save all videos under Movies directory in your home directory
 -o ~/Movies/%(title)s.%(ext)s
 ```
 Note that options in configuration file are just the same options aka switches used in regular command line calls thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`.
@@ -661,7 +653,11 @@ $ youtube-dl -f 'best[filesize<50M]'
 # Download best format available via direct link over HTTP/HTTPS protocol
 $ youtube-dl -f '(bestvideo+bestaudio/best)[protocol^=http]'
 # Download the best video format and the best audio format without merging them
 $ youtube-dl -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s'
 ```
 Note that in the last example, an output template is recommended as bestvideo and bestaudio may have the same file name.
 # VIDEO SELECTION
@@ -742,7 +738,7 @@ Videos or video formats streamed via RTMP protocol can only be downloaded when [
 ### I have downloaded a video but how can I play it?
-Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org) or [mplayer](http://www.mplayerhq.hu/).
+Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org/) or [mplayer](http://www.mplayerhq.hu/).
 ### I extracted a video URL with `-g`, but it does not play on another machine / in my webbrowser.
@@ -824,10 +820,32 @@ Either prepend `http://www.youtube.com/watch?v=` or separate the ID from the opt
 ### How do I pass cookies to youtube-dl?
-Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`. Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows, `LF` (`\n`) for Linux and `CR` (`\r`) for Mac OS. `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
+Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`.
 In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [Export Cookies](https://addons.mozilla.org/en-US/firefox/addon/export-cookies/) (for Firefox).
 Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows, `LF` (`\n`) for Linux and `CR` (`\r`) for Mac OS. `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
 Passing cookies to youtube-dl is a good way to workaround login when a particular extractor does not implement it explicitly. Another use case is working around [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) some websites require you to solve in particular cases in order to get access (e.g. YouTube, CloudFlare).
 ### How do I stream directly to media player?
 You will first need to tell youtube-dl to stream media to stdout with `-o -`, and also tell your media player to read from stdin (it must be capable of this for streaming) and then pipe former to latter. For example, streaming to [vlc](http://www.videolan.org/) can be achieved with:
    youtube-dl -o - "http://www.youtube.com/watch?v=BaW_jenozKcj" | vlc -
 ### How do I download only new videos from a playlist?
 Use download-archive feature. With this feature you should initially download the complete playlist with `--download-archive /path/to/download/archive/file.txt` that will record identifiers of all the videos in a special file. Each subsequent run with the same `--download-archive` will download only new videos and skip all videos that have been downloaded before. Note that only successful downloads are recorded in the file.
 For example, at first,
    youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
 will download the complete `PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re` playlist and create a file `archive.txt`. Each subsequent run will only download new videos if any:
    youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
 ### Can you add support for this anime video site, or site which shows current movies for free?
 As a matter of policy (as well as legality), youtube-dl does not include support for services that specialize in infringing copyright. As a rule of thumb, if you cannot easily find a video that the service is quite obviously allowed to distribute (i.e. that has been uploaded by the creator, the creator's distributor, or is published under a free license), the service is probably unfit for inclusion to youtube-dl.
@@ -1196,7 +1214,7 @@ Make sure that someone has not already opened the issue you're trying to open. S
 ###  Why are existing options not enough?
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#synopsis). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
 ###  Is there enough context in your bug report?
--- a/devscripts/prepare_manpage.py
+++ b/devscripts/prepare_manpage.py
@@ -54,17 +54,21 @@ def filter_options(readme):
        if in_options:
            if line.lstrip().startswith('-'):
-                option, description = re.split(r'\s{2,}', line.lstrip())
+                split = re.split(r'\s{2,}', line.lstrip())
-                split_option = option.split(' ')
+                # Description string may start with `-` as well. If there is
                # only one piece then it's a description bit not an option.
                if len(split) > 1:
                    option, description = split
                    split_option = option.split(' ')
-                if not split_option[-1].startswith('-'):  # metavar
+                    if not split_option[-1].startswith('-'):  # metavar
-                    option = ' '.join(split_option[:-1] + ['*%s*' % split_option[-1]])
+                        option = ' '.join(split_option[:-1] + ['*%s*' % split_option[-1]])
-                # Pandoc's definition_lists. See http://pandoc.org/README.html
+                    # Pandoc's definition_lists. See http://pandoc.org/README.html
-                # for more information.
+                    # for more information.
-                ret += '\n%s\n:   %s\n' % (option, description)
+                    ret += '\n%s\n:   %s\n' % (option, description)
-            else:
+                    continue
-                ret += line.lstrip() + '\n'
+            ret += line.lstrip() + '\n'
        else:
            ret += line + '\n'
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@@ -13,9 +13,12 @@
 - **5min**
 - **8tracks**
 - **91porn**
 - **9c9media**
 - **9c9media:stack**
 - **9gag**
 - **9now.com.au**
 - **abc.net.au**
 - **abc.net.au:iview**
 - **Abc7News**
 - **abcnews**
 - **abcnews:video**
@@ -35,6 +38,7 @@
 - **AlJazeera**
 - **Allocine**
 - **AlphaPorno**
 - **AMCNetworks**
 - **AnimeOnDemand**
 - **anitube.se**
 - **AnySex**
@@ -65,6 +69,10 @@
 - **audiomack**
 - **audiomack:album**
 - **auroravid**: AuroraVid
 - **AWAAN**
 - **awaan:live**
 - **awaan:season**
 - **awaan:video**
 - **Azubu**
 - **AzubuLive**
 - **BaiduVideo**: 百度视频
@@ -109,8 +117,11 @@
 - **Canvas**
 - **CarambaTV**
 - **CarambaTVPage**
- - **CBC**
+ - **CartoonNetwork**
- - **CBCPlayer**
+ - **cbc.ca**
 - **cbc.ca:player**
 - **cbc.ca:watch**
 - **cbc.ca:watch:video**
 - **CBS**
 - **CBSInteractive**
 - **CBSLocal**
@@ -120,6 +131,7 @@
 - **CDA**
 - **CeskaTelevize**
 - **channel9**: Channel 9
 - **CharlieRose**
 - **Chaturbate**
 - **Chilloutzone**
 - **chirbit**
@@ -159,6 +171,8 @@
 - **CTVNews**
 - **culturebox.francetvinfo.fr**
 - **CultureUnplugged**
 - **curiositystream**
 - **curiositystream:collection**
 - **CWTV**
 - **DailyMail**
 - **dailymotion**
@@ -170,10 +184,6 @@
 - **daum.net:playlist**
 - **daum.net:user**
 - **DBTV**
 - **DCN**
 - **dcn:live**
 - **dcn:season**
 - **dcn:video**
 - **DctpTv**
 - **DeezerPlaylist**
 - **defense.gouv.fr**
@@ -215,11 +225,11 @@
 - **EsriVideo**
 - **Europa**
 - **EveryonesMixtape**
 - **exfm**: ex.fm
 - **ExpoTV**
 - **ExtremeTube**
 - **EyedoTV**
 - **facebook**
 - **FacebookPluginsVideo**
 - **faz.net**
 - **fc2**
 - **Fczenit**
@@ -238,7 +248,6 @@
 - **FoxSports**
 - **france2.fr:generation-quoi**
 - **FranceCulture**
 - **FranceCultureEmission**
 - **FranceInter**
 - **francetv**: France 2, 3, 4, 5 and Ô
 - **francetvinfo.fr**
@@ -248,6 +257,7 @@
 - **Funimation**
 - **FunnyOrDie**
 - **Fusion**
 - **FXNetworks**
 - **GameInformer**
 - **GameOne**
 - **gameone:playlist**
@@ -263,9 +273,9 @@
 - **Glide**: Glide mobile video messages (glide.me)
 - **Globo**
 - **GloboArticle**
 - **Go**
 - **GodTube**
 - **GodTV**
 - **GoldenMoustache**
 - **Golem**
 - **GoogleDrive**
 - **Goshgay**
@@ -278,6 +288,8 @@
 - **HellPorno**
 - **Helsinki**: helsinki.fi
 - **HentaiStigma**
 - **HGTV**
 - **hgtv.com:show**
 - **HistoricFilms**
 - **history:topic**: History.com Topic
 - **hitbox**
@@ -397,8 +409,10 @@
 - **MovieClips**
 - **MovieFap**
 - **Moviezine**
 - **MovingImage**
 - **MPORA**
 - **MSN**
 - **mtg**: MTG services
 - **MTV**
 - **mtv.de**
 - **mtvservices:embedded**
@@ -443,6 +457,7 @@
 - **NextMediaActionNews**: 蘋果日報 - 動新聞
 - **nfb**: National Film Board of Canada
 - **nfl.com**
 - **NhkVod**
 - **nhl.com**
 - **nhl.com:news**: NHL news
 - **nhl.com:videocenter**
@@ -451,7 +466,6 @@
 - **nick.de**
 - **niconico**: ニコニコ動画
 - **NiconicoPlaylist**
 - **NineCNineMedia**
 - **Nintendo**
 - **njoy**: N-JOY
 - **njoy:embed**
@@ -509,7 +523,6 @@
 - **Pinkbike**
 - **Pladform**
 - **play.fm**
 - **played.to**
 - **PlaysTV**
 - **Playtvak**: Playtvak.cz, iDNES.cz and Lidovky.cz
 - **Playvid**
@@ -521,6 +534,7 @@
 - **podomatic**
 - **Pokemon**
 - **PolskieRadio**
 - **PornCom**
 - **PornHd**
 - **PornHub**: PornHub and Thumbzilla
 - **PornHubPlaylist**
@@ -564,6 +578,7 @@
 - **RoosterTeeth**
 - **RottenTomatoes**
 - **Roxwel**
 - **Rozhlas**
 - **RTBF**
 - **rte**: Raidió Teilifís Éireann TV
 - **rte:radio**: Raidió Teilifís Éireann radio
@@ -621,6 +636,7 @@
 - **smotri:user**: Smotri.com user videos
 - **Snotr**
 - **Sohu**
 - **SonyLIV**
 - **soundcloud**
 - **soundcloud:playlist**
 - **soundcloud:search**: Soundcloud search
@@ -647,7 +663,6 @@
 - **sr:mediathek**: Saarländischer Rundfunk
 - **SRGSSR**
 - **SRGSSRPlay**: srf.ch, rts.ch, rsi.ch, rtr.ch and swissinfo.ch play sites
 - **SSA**
 - **stanfordoc**: Stanford Open ClassRoom
 - **Steam**
 - **Stitcher**
@@ -663,8 +678,8 @@
 - **SztvHu**
 - **Tagesschau**
 - **tagesschau:player**
 - **Tapely**
 - **Tass**
 - **TBS**
 - **TDSLifeway**
 - **teachertube**: teachertube.com videos
 - **teachertube:user:collection**: teachertube.com user and collection videos
@@ -690,8 +705,6 @@
 - **TheStar**
 - **ThisAmericanLife**
 - **ThisAV**
 - **THVideo**
 - **THVideoPlaylist**
 - **tinypic**: tinypic.com videos
 - **tlc.de**
 - **TMZ**
@@ -706,7 +719,6 @@
 - **TrailerAddict** (Currently broken)
 - **Trilulilu**
 - **trollvids**
 - **TruTube**
 - **Tube8**
 - **TubiTv**
 - **tudou**
@@ -731,7 +743,6 @@
 - **tvp**: Telewizja Polska
 - **tvp:embed**: Telewizja Polska
 - **tvp:series**
 - **TVPlay**: TV3Play and related services
 - **Tweakers**
 - **twitch:chapter**
 - **twitch:clips**
@@ -747,8 +758,12 @@
 - **udemy:course**
 - **UDNEmbed**: 聯合影音
 - **Unistra**
 - **uol.com.br**
 - **uplynk**
 - **uplynk:preplay**
 - **Urort**: NRK P3 Urørt
 - **URPlay**
 - **USANetwork**
 - **USAToday**
 - **ustream**
 - **ustream:channel**
@@ -764,7 +779,9 @@
 - **VevoPlaylist**
 - **VGTV**: VGTV, BTTV, FTV, Aftenposten and Aftonbladet
 - **vh1.com**
 - **Viafree**
 - **Vice**
 - **Viceland**
 - **ViceShow**
 - **Vidbit**
 - **Viddler**
@@ -884,6 +901,4 @@
 - **Zapiks**
 - **ZDF**
 - **ZDFChannel**
- - **zingmp3:album**: mp3.zing.vn albums
+ - **zingmp3**: mp3.zing.vn
 - **zingmp3:song**: mp3.zing.vn songs
 - **ZippCast**
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -39,6 +39,7 @@ from youtube_dl.utils import (
    is_html,
    js_to_json,
    limit_length,
    mimetype2ext,
    ohdave_rsa_encrypt,
    OnDemandPagedList,
    orderedSet,
@@ -625,6 +626,14 @@ class TestUtil(unittest.TestCase):
            limit_length('foo bar baz asd', 12).startswith('foo bar'))
        self.assertTrue('...' in limit_length('foo bar baz asd', 12))
    def test_mimetype2ext(self):
        self.assertEqual(mimetype2ext(None), None)
        self.assertEqual(mimetype2ext('video/x-flv'), 'flv')
        self.assertEqual(mimetype2ext('application/x-mpegURL'), 'm3u8')
        self.assertEqual(mimetype2ext('text/vtt'), 'vtt')
        self.assertEqual(mimetype2ext('text/vtt;charset=utf-8'), 'vtt')
        self.assertEqual(mimetype2ext('text/html; charset=utf-8'), 'html')
    def test_parse_codecs(self):
        self.assertEqual(parse_codecs(''), {})
        self.assertEqual(parse_codecs('avc1.77.30, mp4a.40.2'), {
@@ -712,6 +721,9 @@ class TestUtil(unittest.TestCase):
        inp = '''{"foo":101}'''
        self.assertEqual(js_to_json(inp), '''{"foo":101}''')
        inp = '''{"duration": "00:01:07"}'''
        self.assertEqual(js_to_json(inp), '''{"duration": "00:01:07"}''')
    def test_js_to_json_edgecases(self):
        on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
        self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
@@ -817,7 +829,10 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(parse_filesize('2 MiB'), 2097152)
        self.assertEqual(parse_filesize('5 GB'), 5000000000)
        self.assertEqual(parse_filesize('1.2Tb'), 1200000000000)
        self.assertEqual(parse_filesize('1.2tb'), 1200000000000)
        self.assertEqual(parse_filesize('1,24 KB'), 1240)
        self.assertEqual(parse_filesize('1,24 kb'), 1240)
        self.assertEqual(parse_filesize('8.5 megabytes'), 8500000)
    def test_parse_count(self):
        self.assertEqual(parse_count(None), None)
@@ -968,6 +983,7 @@ The first line
        self.assertEqual(cli_option({'proxy': '127.0.0.1:3128'}, '--proxy', 'proxy'), ['--proxy', '127.0.0.1:3128'])
        self.assertEqual(cli_option({'proxy': None}, '--proxy', 'proxy'), [])
        self.assertEqual(cli_option({}, '--proxy', 'proxy'), [])
        self.assertEqual(cli_option({'retries': 10}, '--retries', 'retries'), ['--retries', '10'])
    def test_cli_valueless_option(self):
        self.assertEqual(cli_valueless_option(
--- a/test/test_verbose_output.py
+++ b/test/test_verbose_output.py
@@ -22,10 +22,10 @@ class TestVerboseOutput(unittest.TestCase):
                '--password', 'secret',
            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        sout, serr = outp.communicate()
-        self.assertTrue('--username' in serr)
+        self.assertTrue(b'--username' in serr)
-        self.assertTrue('johnsmith' not in serr)
+        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue('--password' in serr)
+        self.assertTrue(b'--password' in serr)
-        self.assertTrue('secret' not in serr)
+        self.assertTrue(b'secret' not in serr)
    def test_private_info_shortarg(self):
        outp = subprocess.Popen(
@@ -35,10 +35,10 @@ class TestVerboseOutput(unittest.TestCase):
                '-p', 'secret',
            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        sout, serr = outp.communicate()
-        self.assertTrue('-u' in serr)
+        self.assertTrue(b'-u' in serr)
-        self.assertTrue('johnsmith' not in serr)
+        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue('-p' in serr)
+        self.assertTrue(b'-p' in serr)
-        self.assertTrue('secret' not in serr)
+        self.assertTrue(b'secret' not in serr)
    def test_private_info_eq(self):
        outp = subprocess.Popen(
@@ -48,10 +48,10 @@ class TestVerboseOutput(unittest.TestCase):
                '--password=secret',
            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        sout, serr = outp.communicate()
-        self.assertTrue('--username' in serr)
+        self.assertTrue(b'--username' in serr)
-        self.assertTrue('johnsmith' not in serr)
+        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue('--password' in serr)
+        self.assertTrue(b'--password' in serr)
-        self.assertTrue('secret' not in serr)
+        self.assertTrue(b'secret' not in serr)
    def test_private_info_shortarg_eq(self):
        outp = subprocess.Popen(
@@ -61,10 +61,10 @@ class TestVerboseOutput(unittest.TestCase):
                '-p=secret',
            ], cwd=rootDir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        sout, serr = outp.communicate()
-        self.assertTrue('-u' in serr)
+        self.assertTrue(b'-u' in serr)
-        self.assertTrue('johnsmith' not in serr)
+        self.assertTrue(b'johnsmith' not in serr)
-        self.assertTrue('-p' in serr)
+        self.assertTrue(b'-p' in serr)
-        self.assertTrue('secret' not in serr)
+        self.assertTrue(b'secret' not in serr)
 if __name__ == '__main__':
    unittest.main()
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@@ -249,7 +249,16 @@ class YoutubeDL(object):
    source_address:    (Experimental) Client-side IP address to bind to.
    call_home:         Boolean, true iff we are allowed to contact the
                       youtube-dl servers for debugging.
-    sleep_interval:    Number of seconds to sleep before each download.
+    sleep_interval:    Number of seconds to sleep before each download when
                       used alone or a lower bound of a range for randomized
                       sleep before each download (minimum possible number
                       of seconds to sleep) when used along with
                       max_sleep_interval.
    max_sleep_interval:Upper bound of a range for randomized sleep before each
                       download (maximum possible number of seconds to sleep).
                       Must only be used along with sleep_interval.
                       Actual sleep time will be a random float from range
                       [sleep_interval; max_sleep_interval].
    listformats:       Print an overview of available video formats and exit.
    list_thumbnails:   Print a table of all thumbnails and exit.
    match_filter:      A function that gets called with the info_dict of
@@ -1247,8 +1256,10 @@ class YoutubeDL(object):
                info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
        if thumbnails:
            thumbnails.sort(key=lambda t: (
-                t.get('preference'), t.get('width'), t.get('height'),
+                t.get('preference') if t.get('preference') is not None else -1,
-                t.get('id'), t.get('url')))
+                t.get('width') if t.get('width') is not None else -1,
                t.get('height') if t.get('height') is not None else -1,
                t.get('id') if t.get('id') is not None else '', t.get('url')))
            for i, t in enumerate(thumbnails):
                t['url'] = sanitize_url(t['url'])
                if t.get('width') and t.get('height'):
@@ -1290,7 +1301,7 @@ class YoutubeDL(object):
                for subtitle_format in subtitle:
                    if subtitle_format.get('url'):
                        subtitle_format['url'] = sanitize_url(subtitle_format['url'])
-                    if 'ext' not in subtitle_format:
+                    if subtitle_format.get('ext') is None:
                        subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
        if self.params.get('listsubtitles', False):
@@ -1345,7 +1356,7 @@ class YoutubeDL(object):
                    note=' ({0})'.format(format['format_note']) if format.get('format_note') is not None else '',
                )
            # Automatically determine file extension if missing
-            if 'ext' not in format:
+            if format.get('ext') is None:
                format['ext'] = determine_ext(format['url']).lower()
            # Automatically determine protocol if missing (useful for format
            # selection purposes)
@@ -1594,7 +1605,9 @@ class YoutubeDL(object):
                        self.to_screen('[info] Video subtitle %s.%s is already_present' % (sub_lang, sub_format))
                    else:
                        self.to_screen('[info] Writing video subtitles to: ' + sub_filename)
-                        with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8') as subfile:
+                        # Use newline='' to prevent conversion of newline characters
                        # See https://github.com/rg3/youtube-dl/issues/10268
                        with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8', newline='') as subfile:
                            subfile.write(sub_data)
                except (OSError, IOError):
                    self.report_error('Cannot write subtitles file ' + sub_filename)
--- a/youtube_dl/init.py
+++ b/youtube_dl/init.py
@@ -145,6 +145,16 @@ def _real_main(argv=None):
        if numeric_limit is None:
            parser.error('invalid max_filesize specified')
        opts.max_filesize = numeric_limit
    if opts.sleep_interval is not None:
        if opts.sleep_interval < 0:
            parser.error('sleep interval must be positive or 0')
    if opts.max_sleep_interval is not None:
        if opts.max_sleep_interval < 0:
            parser.error('max sleep interval must be positive or 0')
        if opts.max_sleep_interval < opts.sleep_interval:
            parser.error('max sleep interval must be greater than or equal to min sleep interval')
    else:
        opts.max_sleep_interval = opts.sleep_interval
    def parse_retries(retries):
        if retries in ('inf', 'infinite'):
@@ -370,6 +380,7 @@ def _real_main(argv=None):
        'source_address': opts.source_address,
        'call_home': opts.call_home,
        'sleep_interval': opts.sleep_interval,
        'max_sleep_interval': opts.max_sleep_interval,
        'external_downloader': opts.external_downloader,
        'list_thumbnails': opts.list_thumbnails,
        'playlist_items': opts.playlist_items,
--- a/youtube_dl/downloader/common.py
+++ b/youtube_dl/downloader/common.py
@@ -4,6 +4,7 @@ import os
 import re
 import sys
 import time
 import random
 from ..compat import compat_os_name
 from ..utils import (
@@ -342,8 +343,11 @@ class FileDownloader(object):
            })
            return True
-        sleep_interval = self.params.get('sleep_interval')
+        min_sleep_interval = self.params.get('sleep_interval')
-        if sleep_interval:
+        if min_sleep_interval:
            max_sleep_interval = self.params.get('max_sleep_interval', min_sleep_interval)
            print(min_sleep_interval, max_sleep_interval)
            sleep_interval = random.uniform(min_sleep_interval, max_sleep_interval)
            self.to_screen('[download] Sleeping %s seconds...' % sleep_interval)
            time.sleep(sleep_interval)
--- a/youtube_dl/downloader/external.py
+++ b/youtube_dl/downloader/external.py
@@ -96,6 +96,12 @@ class CurlFD(ExternalFD):
        cmd = [self.exe, '--location', '-o', tmpfilename]
        for key, val in info_dict['http_headers'].items():
            cmd += ['--header', '%s: %s' % (key, val)]
        cmd += self._bool_option('--continue-at', 'continuedl', '-', '0')
        cmd += self._valueless_option('--silent', 'noprogress')
        cmd += self._valueless_option('--verbose', 'verbose')
        cmd += self._option('--limit-rate', 'ratelimit')
        cmd += self._option('--retry', 'retries')
        cmd += self._option('--max-filesize', 'max_filesize')
        cmd += self._option('--interface', 'source_address')
        cmd += self._option('--proxy', 'proxy')
        cmd += self._valueless_option('--insecure', 'nocheckcertificate')
@@ -103,6 +109,16 @@ class CurlFD(ExternalFD):
        cmd += ['--', info_dict['url']]
        return cmd
    def _call_downloader(self, tmpfilename, info_dict):
        cmd = [encodeArgument(a) for a in self._make_cmd(tmpfilename, info_dict)]
        self._debug_cmd(cmd)
        # curl writes the progress to stderr so don't capture it.
        p = subprocess.Popen(cmd)
        p.communicate()
        return p.returncode
 class AxelFD(ExternalFD):
    AVAILABLE_OPT = '-V'
@@ -204,6 +220,12 @@ class FFmpegFD(ExternalFD):
        if proxy:
            if not re.match(r'^[\da-zA-Z]+://', proxy):
                proxy = 'http://%s' % proxy
            if proxy.startswith('socks'):
                self.report_warning(
                    '%s does not support SOCKS proxies. Downloading is likely to fail. '
                    'Consider adding --hls-prefer-native to your command.' % self.get_basename())
            # Since December 2015 ffmpeg supports -http_proxy option (see
            # http://git.videolan.org/?p=ffmpeg.git;a=commit;h=b4eb1f29ebddd60c41a2eb39f5af701e38e0d3fd)
            # We could switch to the following code if we are able to detect version properly
--- a/youtube_dl/downloader/hls.py
+++ b/youtube_dl/downloader/hls.py
@@ -20,6 +20,7 @@ from ..utils import (
    encodeFilename,
    sanitize_open,
    parse_m3u8_attributes,
    update_url_query,
 )
@@ -82,6 +83,10 @@ class HlsFD(FragmentFD):
        self._prepare_and_start_frag_download(ctx)
        extra_query = None
        extra_param_to_segment_url = info_dict.get('extra_param_to_segment_url')
        if extra_param_to_segment_url:
            extra_query = compat_urlparse.parse_qs(extra_param_to_segment_url)
        i = 0
        media_sequence = 0
        decrypt_info = {'METHOD': 'NONE'}
@@ -95,6 +100,8 @@ class HlsFD(FragmentFD):
                        if re.match(r'^https?://', line)
                        else compat_urlparse.urljoin(man_url, line))
                    frag_filename = '%s-Frag%d' % (ctx['tmpfilename'], i)
                    if extra_query:
                        frag_url = update_url_query(frag_url, extra_query)
                    success = ctx['dl'].download(frag_filename, {'url': frag_url})
                    if not success:
                        return False
@@ -116,10 +123,12 @@ class HlsFD(FragmentFD):
                    decrypt_info = parse_m3u8_attributes(line[11:])
                    if decrypt_info['METHOD'] == 'AES-128':
                        if 'IV' in decrypt_info:
-                            decrypt_info['IV'] = binascii.unhexlify(decrypt_info['IV'][2:])
+                            decrypt_info['IV'] = binascii.unhexlify(decrypt_info['IV'][2:].zfill(32))
                        if not re.match(r'^https?://', decrypt_info['URI']):
                            decrypt_info['URI'] = compat_urlparse.urljoin(
                                man_url, decrypt_info['URI'])
                        if extra_query:
                            decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
                        decrypt_info['KEY'] = self.ydl.urlopen(decrypt_info['URI']).read()
                elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
                    media_sequence = int(line[22:])
--- a/youtube_dl/extractor/abc.py
+++ b/youtube_dl/extractor/abc.py
@@ -7,6 +7,7 @@ from ..utils import (
    ExtractorError,
    js_to_json,
    int_or_none,
    parse_iso8601,
 )
@@ -93,3 +94,57 @@ class ABCIE(InfoExtractor):
            'description': self._og_search_description(webpage),
            'thumbnail': self._og_search_thumbnail(webpage),
        }
 class ABCIViewIE(InfoExtractor):
    IE_NAME = 'abc.net.au:iview'
    _VALID_URL = r'https?://iview\.abc\.net\.au/programs/[^/]+/(?P<id>[^/?#]+)'
    _TESTS = [{
        'url': 'http://iview.abc.net.au/programs/gardening-australia/FA1505V024S00',
        'md5': '979d10b2939101f0d27a06b79edad536',
        'info_dict': {
            'id': 'FA1505V024S00',
            'ext': 'mp4',
            'title': 'Series 27 Ep 24',
            'description': 'md5:b28baeae7504d1148e1d2f0e3ed3c15d',
            'upload_date': '20160820',
            'uploader_id': 'abc1',
            'timestamp': 1471719600,
        },
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        video_params = self._parse_json(self._search_regex(
            r'videoParams\s*=\s*({.+?});', webpage, 'video params'), video_id)
        title = video_params['title']
        stream = next(s for s in video_params['playlist'] if s.get('type') == 'program')
        formats = self._extract_akamai_formats(stream['hds-unmetered'], video_id)
        self._sort_formats(formats)
        subtitles = {}
        src_vtt = stream.get('captions', {}).get('src-vtt')
        if src_vtt:
            subtitles['en'] = [{
                'url': src_vtt,
                'ext': 'vtt',
            }]
        return {
            'id': video_id,
            'title': title,
            'description': self._html_search_meta(['og:description', 'twitter:description'], webpage),
            'thumbnail': self._html_search_meta(['og:image', 'twitter:image:src'], webpage),
            'duration': int_or_none(video_params.get('eventDuration')),
            'timestamp': parse_iso8601(video_params.get('pubDate'), ' '),
            'series': video_params.get('seriesTitle'),
            'series_id': video_params.get('seriesHouseNumber') or video_id[:7],
            'episode_number': int_or_none(self._html_search_meta('episodeNumber', webpage)),
            'episode': self._html_search_meta('episode_title', webpage),
            'uploader_id': video_params.get('channel'),
            'formats': formats,
            'subtitles': subtitles,
        }
--- a/youtube_dl/extractor/adobepass.py
+++ b/youtube_dl/extractor/adobepass.py
@@ -0,0 +1,137 @@
 # -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 import re
 import time
 import xml.etree.ElementTree as etree
 from .common import InfoExtractor
 from ..utils import (
    unescapeHTML,
    urlencode_postdata,
    unified_timestamp,
 )
 class AdobePassIE(InfoExtractor):
    _SERVICE_PROVIDER_TEMPLATE = 'https://sp.auth.adobe.com/adobe-services/%s'
    _USER_AGENT = 'Mozilla/5.0 (X11; Linux i686; rv:47.0) Gecko/20100101 Firefox/47.0'
    @staticmethod
    def _get_mvpd_resource(provider_id, title, guid, rating):
        channel = etree.Element('channel')
        channel_title = etree.SubElement(channel, 'title')
        channel_title.text = provider_id
        item = etree.SubElement(channel, 'item')
        resource_title = etree.SubElement(item, 'title')
        resource_title.text = title
        resource_guid = etree.SubElement(item, 'guid')
        resource_guid.text = guid
        resource_rating = etree.SubElement(item, 'media:rating')
        resource_rating.attrib = {'scheme': 'urn:v-chip'}
        resource_rating.text = rating
        return '<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/">' + etree.tostring(channel).decode() + '</rss>'
    def _extract_mvpd_auth(self, url, video_id, requestor_id, resource):
        def xml_text(xml_str, tag):
            return self._search_regex(
                '<%s>(.+?)</%s>' % (tag, tag), xml_str, tag)
        def is_expired(token, date_ele):
            token_expires = unified_timestamp(re.sub(r'[_ ]GMT', '', xml_text(token, date_ele)))
            return token_expires and token_expires <= int(time.time())
        mvpd_headers = {
            'ap_42': 'anonymous',
            'ap_11': 'Linux i686',
            'ap_z': self._USER_AGENT,
            'User-Agent': self._USER_AGENT,
        }
        guid = xml_text(resource, 'guid')
        requestor_info = self._downloader.cache.load('mvpd', requestor_id) or {}
        authn_token = requestor_info.get('authn_token')
        if authn_token and is_expired(authn_token, 'simpleTokenExpires'):
            authn_token = None
        if not authn_token:
            # TODO add support for other TV Providers
            mso_id = 'DTV'
            username, password = self._get_netrc_login_info(mso_id)
            if not username or not password:
                return ''
            def post_form(form_page, note, data={}):
                post_url = self._html_search_regex(r'<form[^>]+action=(["\'])(?P<url>.+?)\1', form_page, 'post url', group='url')
                return self._download_webpage(
                    post_url, video_id, note, data=urlencode_postdata(data or self._hidden_inputs(form_page)), headers={
                        'Content-Type': 'application/x-www-form-urlencoded',
                    })
            provider_redirect_page = self._download_webpage(
                self._SERVICE_PROVIDER_TEMPLATE % 'authenticate/saml', video_id,
                'Downloading Provider Redirect Page', query={
                    'noflash': 'true',
                    'mso_id': mso_id,
                    'requestor_id': requestor_id,
                    'no_iframe': 'false',
                    'domain_name': 'adobe.com',
                    'redirect_url': url,
                })
            provider_login_page = post_form(
                provider_redirect_page, 'Downloading Provider Login Page')
            mvpd_confirm_page = post_form(provider_login_page, 'Logging in', {
                'username': username,
                'password': password,
            })
            post_form(mvpd_confirm_page, 'Confirming Login')
            session = self._download_webpage(
                self._SERVICE_PROVIDER_TEMPLATE % 'session', video_id,
                'Retrieving Session', data=urlencode_postdata({
                    '_method': 'GET',
                    'requestor_id': requestor_id,
                }), headers=mvpd_headers)
            if '<pendingLogout' in session:
                self._downloader.cache.store('mvpd', requestor_id, {})
                return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
            authn_token = unescapeHTML(xml_text(session, 'authnToken'))
            requestor_info['authn_token'] = authn_token
            self._downloader.cache.store('mvpd', requestor_id, requestor_info)
        authz_token = requestor_info.get(guid)
        if authz_token and is_expired(authz_token, 'simpleTokenTTL'):
            authz_token = None
        if not authz_token:
            authorize = self._download_webpage(
                self._SERVICE_PROVIDER_TEMPLATE % 'authorize', video_id,
                'Retrieving Authorization Token', data=urlencode_postdata({
                    'resource_id': resource,
                    'requestor_id': requestor_id,
                    'authentication_token': authn_token,
                    'mso_id': xml_text(authn_token, 'simpleTokenMsoID'),
                    'userMeta': '1',
                }), headers=mvpd_headers)
            if '<pendingLogout' in authorize:
                self._downloader.cache.store('mvpd', requestor_id, {})
                return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
            authz_token = unescapeHTML(xml_text(authorize, 'authzToken'))
            requestor_info[guid] = authz_token
            self._downloader.cache.store('mvpd', requestor_id, requestor_info)
        mvpd_headers.update({
            'ap_19': xml_text(authn_token, 'simpleSamlNameID'),
            'ap_23': xml_text(authn_token, 'simpleSamlSessionIndex'),
        })
        short_authorize = self._download_webpage(
            self._SERVICE_PROVIDER_TEMPLATE % 'shortAuthorize',
            video_id, 'Retrieving Media Token', data=urlencode_postdata({
                'authz_token': authz_token,
                'requestor_id': requestor_id,
                'session_guid': xml_text(authn_token, 'simpleTokenAuthenticationGuid'),
                'hashed_guid': 'false',
            }), headers=mvpd_headers)
        if '<pendingLogout' in short_authorize:
            self._downloader.cache.store('mvpd', requestor_id, {})
            return self._extract_mvpd_auth(url, video_id, requestor_id, resource)
        return short_authorize
--- a/youtube_dl/extractor/adultswim.py
+++ b/youtube_dl/extractor/adultswim.py
@@ -3,16 +3,14 @@ from __future__ import unicode_literals
 import re
-from .common import InfoExtractor
+from .turner import TurnerBaseIE
 from ..utils import (
    determine_ext,
    ExtractorError,
-    float_or_none,
+    int_or_none,
    xpath_text,
 )
-class AdultSwimIE(InfoExtractor):
+class AdultSwimIE(TurnerBaseIE):
    _VALID_URL = r'https?://(?:www\.)?adultswim\.com/videos/(?P<is_playlist>playlists/)?(?P<show_path>[^/]+)/(?P<episode_path>[^/?#]+)/?'
    _TESTS = [{
@@ -96,7 +94,8 @@ class AdultSwimIE(InfoExtractor):
        'params': {
            # m3u8 download
            'skip_download': True,
-        }
+        },
        'expected_warnings': ['Unable to download f4m manifest'],
    }]
    @staticmethod
@@ -148,7 +147,10 @@ class AdultSwimIE(InfoExtractor):
                if bootstrapped_data.get('slugged_video', {}).get('slug') == episode_path:
                    video_info = bootstrapped_data['slugged_video']
            if not video_info:
-                video_info = bootstrapped_data.get('heroMetadata', {}).get('trailer').get('video')
+                video_info = bootstrapped_data.get(
                    'heroMetadata', {}).get('trailer', {}).get('video')
            if not video_info:
                video_info = bootstrapped_data.get('onlineOriginals', [None])[0]
            if not video_info:
                raise ExtractorError('Unable to find video info')
@@ -162,70 +164,38 @@ class AdultSwimIE(InfoExtractor):
            elif video_info.get('videoPlaybackID'):
                segment_ids = [video_info['videoPlaybackID']]
            else:
-                raise ExtractorError(
+                if video_info.get('auth') is True:
-                    'This video is only available via cable service provider subscription that'
+                    raise ExtractorError(
-                    ' is not currently supported. You may want to use --cookies.'
+                        'This video is only available via cable service provider subscription that'
-                    if video_info.get('auth') is True else 'Unable to find stream or clips',
+                        ' is not currently supported. You may want to use --cookies.', expected=True)
-                    expected=True)
+                else:
                    raise ExtractorError('Unable to find stream or clips')
        episode_id = video_info['id']
        episode_title = video_info['title']
-        episode_description = video_info['description']
+        episode_description = video_info.get('description')
-        episode_duration = video_info.get('duration')
+        episode_duration = int_or_none(video_info.get('duration'))
        view_count = int_or_none(video_info.get('views'))
        entries = []
        for part_num, segment_id in enumerate(segment_ids):
-            segment_url = 'http://www.adultswim.com/videos/api/v0/assets?id=%s&platform=desktop' % segment_id
+            segement_info = self._extract_cvp_info(
-
+                'http://www.adultswim.com/videos/api/v0/assets?id=%s&platform=desktop' % segment_id,
                segment_id, {
                    'secure': {
                        'media_src': 'http://androidhls-secure.cdn.turner.com/adultswim/big',
                        'tokenizer_src': 'http://www.adultswim.com/astv/mvpd/processors/services/token_ipadAdobe.do',
                    },
                })
            segment_title = '%s - %s' % (show_title, episode_title)
            if len(segment_ids) > 1:
                segment_title += ' Part %d' % (part_num + 1)
-
+            segement_info.update({
            idoc = self._download_xml(
                segment_url, segment_title,
                'Downloading segment information', 'Unable to download segment information')
            segment_duration = float_or_none(
                xpath_text(idoc, './/trt', 'segment duration').strip())
            formats = []
            file_els = idoc.findall('.//files/file') or idoc.findall('./files/file')
            unique_urls = []
            unique_file_els = []
            for file_el in file_els:
                media_url = file_el.text
                if not media_url or determine_ext(media_url) == 'f4m':
                    continue
                if file_el.text not in unique_urls:
                    unique_urls.append(file_el.text)
                    unique_file_els.append(file_el)
            for file_el in unique_file_els:
                bitrate = file_el.attrib.get('bitrate')
                ftype = file_el.attrib.get('type')
                media_url = file_el.text
                if determine_ext(media_url) == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        media_url, segment_title, 'mp4', preference=0,
                        m3u8_id='hls', fatal=False))
                else:
                    formats.append({
                        'format_id': '%s_%s' % (bitrate, ftype),
                        'url': file_el.text.strip(),
                        # The bitrate may not be a number (for example: 'iphone')
                        'tbr': int(bitrate) if bitrate.isdigit() else None,
                    })
            self._sort_formats(formats)
            entries.append({
                'id': segment_id,
                'title': segment_title,
-                'formats': formats,
+                'description': episode_description,
                'duration': segment_duration,
                'description': episode_description
            })
            entries.append(segement_info)
        return {
            '_type': 'playlist',
@@ -234,5 +204,6 @@ class AdultSwimIE(InfoExtractor):
            'entries': entries,
            'title': '%s - %s' % (show_title, episode_title),
            'description': episode_description,
-            'duration': episode_duration
+            'duration': episode_duration,
            'view_count': view_count,
        }
--- a/youtube_dl/extractor/aenetworks.py
+++ b/youtube_dl/extractor/aenetworks.py
@@ -109,7 +109,10 @@ class AENetworksIE(AENetworksBaseIE):
        info = self._parse_theplatform_metadata(theplatform_metadata)
        if theplatform_metadata.get('AETN$isBehindWall'):
            requestor_id = self._DOMAIN_TO_REQUESTOR_ID[domain]
-            resource = '<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>%s</title><item><title>%s</title><guid>%s</guid><media:rating scheme="urn:v-chip">%s</media:rating></item></channel></rss>' % (requestor_id, theplatform_metadata['title'], theplatform_metadata['AETN$PPL_pplProgramId'], theplatform_metadata['ratings'][0]['rating'])
+            resource = self._get_mvpd_resource(
                requestor_id, theplatform_metadata['title'],
                theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
                theplatform_metadata['ratings'][0]['rating'])
            query['auth'] = self._extract_mvpd_auth(
                url, video_id, requestor_id, resource)
        info.update(self._search_json_ld(webpage, video_id, fatal=False))
--- a/youtube_dl/extractor/amcnetworks.py
+++ b/youtube_dl/extractor/amcnetworks.py
@@ -0,0 +1,91 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .theplatform import ThePlatformIE
 from ..utils import (
    update_url_query,
    parse_age_limit,
    int_or_none,
 )
 class AMCNetworksIE(ThePlatformIE):
    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|wetv)\.com/(?:movies/|shows/[^/]+/(?:full-episodes/)?season-\d+/episode-\d+(?:-(?:[^/]+/)?|/))(?P<id>[^/?#]+)'
    _TESTS = [{
        'url': 'http://www.ifc.com/shows/maron/season-04/episode-01/step-1',
        'md5': '',
        'info_dict': {
            'id': 's3MX01Nl4vPH',
            'ext': 'mp4',
            'title': 'Maron - Season 4 - Step 1',
            'description': 'In denial about his current situation, Marc is reluctantly convinced by his friends to enter rehab. Starring Marc Maron and Constance Zimmer.',
            'age_limit': 17,
            'upload_date': '20160505',
            'timestamp': 1462468831,
            'uploader': 'AMCN',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://www.bbcamerica.com/shows/the-hunt/full-episodes/season-1/episode-01-the-hardest-challenge',
        'only_matching': True,
    }, {
        'url': 'http://www.amc.com/shows/preacher/full-episodes/season-01/episode-00/pilot',
        'only_matching': True,
    }, {
        'url': 'http://www.wetv.com/shows/million-dollar-matchmaker/season-01/episode-06-the-dumped-dj-and-shallow-hal',
        'only_matching': True,
    }, {
        'url': 'http://www.ifc.com/movies/chaos',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        query = {
            'mbr': 'true',
            'manifest': 'm3u',
        }
        media_url = self._search_regex(r'window\.platformLinkURL\s*=\s*[\'"]([^\'"]+)', webpage, 'media url')
        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
            r'https?://link.theplatform.com/s/([^?]+)', media_url, 'theplatform_path'), display_id)
        info = self._parse_theplatform_metadata(theplatform_metadata)
        video_id = theplatform_metadata['pid']
        title = theplatform_metadata['title']
        rating = theplatform_metadata['ratings'][0]['rating']
        auth_required = self._search_regex(r'window\.authRequired\s*=\s*(true|false);', webpage, 'auth required')
        if auth_required == 'true':
            requestor_id = self._search_regex(r'window\.requestor_id\s*=\s*[\'"]([^\'"]+)', webpage, 'requestor id')
            resource = self._get_mvpd_resource(requestor_id, title, video_id, rating)
            query['auth'] = self._extract_mvpd_auth(url, video_id, requestor_id, resource)
        media_url = update_url_query(media_url, query)
        formats, subtitles = self._extract_theplatform_smil(media_url, video_id)
        self._sort_formats(formats)
        info.update({
            'id': video_id,
            'subtitles': subtitles,
            'formats': formats,
            'age_limit': parse_age_limit(parse_age_limit(rating)),
        })
        ns_keys = theplatform_metadata.get('$xmlns', {}).keys()
        if ns_keys:
            ns = list(ns_keys)[0]
            series = theplatform_metadata.get(ns + '$show')
            season_number = int_or_none(theplatform_metadata.get(ns + '$season'))
            episode = theplatform_metadata.get(ns + '$episodeTitle')
            episode_number = int_or_none(theplatform_metadata.get(ns + '$episode'))
            if season_number:
                title = 'Season %d - %s' % (season_number, title)
            if series:
                title = '%s - %s' % (series, title)
            info.update({
                'title': title,
                'series': series,
                'season_number': season_number,
                'episode': episode,
                'episode_number': episode_number,
            })
        return info
--- a/youtube_dl/extractor/aol.py
+++ b/youtube_dl/extractor/aol.py
@@ -123,6 +123,10 @@ class AolFeaturesIE(InfoExtractor):
            'title': 'What To Watch - February 17, 2016',
        },
        'add_ie': ['FiveMin'],
        'params': {
            # encrypted m3u8 download
            'skip_download': True,
        },
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/aparat.py
+++ b/youtube_dl/extractor/aparat.py
@@ -1,8 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
@@ -15,7 +13,7 @@ class AparatIE(InfoExtractor):
    _TEST = {
        'url': 'http://www.aparat.com/v/wP8On',
-        'md5': '6714e0af7e0d875c5a39c4dc4ab46ad1',
+        'md5': '131aca2e14fe7c4dcb3c4877ba300c89',
        'info_dict': {
            'id': 'wP8On',
            'ext': 'mp4',
@@ -31,13 +29,13 @@ class AparatIE(InfoExtractor):
        # Note: There is an easier-to-parse configuration at
        # http://www.aparat.com/video/video/config/videohash/%video_id
        # but the URL in there does not work
-        embed_url = ('http://www.aparat.com/video/video/embed/videohash/' +
+        embed_url = 'http://www.aparat.com/video/video/embed/vt/frame/showvideo/yes/videohash/' + video_id
                     video_id + '/vt/frame')
        webpage = self._download_webpage(embed_url, video_id)
-        video_urls = [video_url.replace('\\/', '/') for video_url in re.findall(
+        file_list = self._parse_json(self._search_regex(
-            r'(?:fileList\[[0-9]+\]\s*=|"file"\s*:)\s*"([^"]+)"', webpage)]
+            r'fileList\s*=\s*JSON\.parse\(\'([^\']+)\'\)', webpage, 'file list'), video_id)
-        for i, video_url in enumerate(video_urls):
+        for i, item in enumerate(file_list[0]):
            video_url = item['file']
            req = HEADRequest(video_url)
            res = self._request_webpage(
                req, video_id, note='Testing video URL %d' % i, errnote=False)
--- a/youtube_dl/extractor/awaan.py
+++ b/youtube_dl/extractor/awaan.py
@@ -12,46 +12,41 @@ from ..compat import (
 from ..utils import (
    int_or_none,
    parse_iso8601,
    sanitized_Request,
    smuggle_url,
    unsmuggle_url,
    urlencode_postdata,
 )
-class DCNIE(InfoExtractor):
+class AWAANIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?show/(?P<show_id>\d+)/[^/]+(?:/(?P<video_id>\d+)/(?P<season_id>\d+))?'
    def _real_extract(self, url):
        show_id, video_id, season_id = re.match(self._VALID_URL, url).groups()
        if video_id and int(video_id) > 0:
            return self.url_result(
-                'http://www.dcndigital.ae/media/%s' % video_id, 'DCNVideo')
+                'http://awaan.ae/media/%s' % video_id, 'AWAANVideo')
        elif season_id and int(season_id) > 0:
            return self.url_result(smuggle_url(
-                'http://www.dcndigital.ae/program/season/%s' % season_id,
+                'http://awaan.ae/program/season/%s' % season_id,
-                {'show_id': show_id}), 'DCNSeason')
+                {'show_id': show_id}), 'AWAANSeason')
        else:
            return self.url_result(
-                'http://www.dcndigital.ae/program/%s' % show_id, 'DCNSeason')
+                'http://awaan.ae/program/%s' % show_id, 'AWAANSeason')
-class DCNBaseIE(InfoExtractor):
+class AWAANBaseIE(InfoExtractor):
-    def _extract_video_info(self, video_data, video_id, is_live):
+    def _parse_video_data(self, video_data, video_id, is_live):
        title = video_data.get('title_en') or video_data['title_ar']
        img = video_data.get('img')
        thumbnail = 'http://admin.mangomolo.com/analytics/%s' % img if img else None
        duration = int_or_none(video_data.get('duration'))
        description = video_data.get('description_en') or video_data.get('description_ar')
        timestamp = parse_iso8601(video_data.get('create_time'), ' ')
        return {
            'id': video_id,
            'title': self._live_title(title) if is_live else title,
-            'description': description,
+            'description': video_data.get('description_en') or video_data.get('description_ar'),
-            'thumbnail': thumbnail,
+            'thumbnail': 'http://admin.mangomolo.com/analytics/%s' % img if img else None,
-            'duration': duration,
+            'duration': int_or_none(video_data.get('duration')),
-            'timestamp': timestamp,
+            'timestamp': parse_iso8601(video_data.get('create_time'), ' '),
            'is_live': is_live,
        }
@@ -75,11 +70,12 @@ class DCNBaseIE(InfoExtractor):
        return formats
-class DCNVideoIE(DCNBaseIE):
+class AWAANVideoIE(AWAANBaseIE):
-    IE_NAME = 'dcn:video'
+    IE_NAME = 'awaan:video'
    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?(?:video(?:/[^/]+)?|media|catchup/[^/]+/[^/]+)/(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.dcndigital.ae/#/video/%D8%B1%D8%AD%D9%84%D8%A9-%D8%A7%D9%84%D8%B9%D9%85%D8%B1-%D8%A7%D9%84%D8%AD%D9%84%D9%82%D8%A9-1/17375',
        'md5': '5f61c33bfc7794315c671a62d43116aa',
        'info_dict':
        {
            'id': '17375',
@@ -90,10 +86,6 @@ class DCNVideoIE(DCNBaseIE):
            'timestamp': 1227504126,
            'upload_date': '20081124',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://awaan.ae/video/26723981/%D8%AF%D8%A7%D8%B1-%D8%A7%D9%84%D8%B3%D9%84%D8%A7%D9%85:-%D8%AE%D9%8A%D8%B1-%D8%AF%D9%88%D8%B1-%D8%A7%D9%84%D8%A3%D9%86%D8%B5%D8%A7%D8%B1',
        'only_matching': True,
@@ -102,11 +94,10 @@ class DCNVideoIE(DCNBaseIE):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        request = sanitized_Request(
+        video_data = self._download_json(
            'http://admin.mangomolo.com/analytics/index.php/plus/video?id=%s' % video_id,
-            headers={'Origin': 'http://www.dcndigital.ae'})
+            video_id, headers={'Origin': 'http://awaan.ae'})
-        video_data = self._download_json(request, video_id)
+        info = self._parse_video_data(video_data, video_id, False)
        info = self._extract_video_info(video_data, video_id, False)
        webpage = self._download_webpage(
            'http://admin.mangomolo.com/analytics/index.php/customers/embed/video?' +
@@ -121,19 +112,31 @@ class DCNVideoIE(DCNBaseIE):
        return info
-class DCNLiveIE(DCNBaseIE):
+class AWAANLiveIE(AWAANBaseIE):
-    IE_NAME = 'dcn:live'
+    IE_NAME = 'awaan:live'
    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?live/(?P<id>\d+)'
    _TEST = {
        'url': 'http://awaan.ae/live/6/dubai-tv',
        'info_dict': {
            'id': '6',
            'ext': 'mp4',
            'title': 're:Dubai Al Oula [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
            'upload_date': '20150107',
            'timestamp': 1420588800,
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }
    def _real_extract(self, url):
        channel_id = self._match_id(url)
-        request = sanitized_Request(
+        channel_data = self._download_json(
            'http://admin.mangomolo.com/analytics/index.php/plus/getchanneldetails?channel_id=%s' % channel_id,
-            headers={'Origin': 'http://www.dcndigital.ae'})
+            channel_id, headers={'Origin': 'http://awaan.ae'})
-
+        info = self._parse_video_data(channel_data, channel_id, True)
        channel_data = self._download_json(request, channel_id)
        info = self._extract_video_info(channel_data, channel_id, True)
        webpage = self._download_webpage(
            'http://admin.mangomolo.com/analytics/index.php/customers/embed/index?' +
@@ -148,8 +151,8 @@ class DCNLiveIE(DCNBaseIE):
        return info
-class DCNSeasonIE(InfoExtractor):
+class AWAANSeasonIE(InfoExtractor):
-    IE_NAME = 'dcn:season'
+    IE_NAME = 'awaan:season'
    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?program/(?:(?P<show_id>\d+)|season/(?P<season_id>\d+))'
    _TEST = {
        'url': 'http://dcndigital.ae/#/program/205024/%D9%85%D8%AD%D8%A7%D8%B6%D8%B1%D8%A7%D8%AA-%D8%A7%D9%84%D8%B4%D9%8A%D8%AE-%D8%A7%D9%84%D8%B4%D8%B9%D8%B1%D8%A7%D9%88%D9%8A',
@@ -170,21 +173,17 @@ class DCNSeasonIE(InfoExtractor):
            data['season'] = season_id
            show_id = smuggled_data.get('show_id')
            if show_id is None:
-                request = sanitized_Request(
+                season = self._download_json(
                    'http://admin.mangomolo.com/analytics/index.php/plus/season_info?id=%s' % season_id,
-                    headers={'Origin': 'http://www.dcndigital.ae'})
+                    season_id, headers={'Origin': 'http://awaan.ae'})
                season = self._download_json(request, season_id)
                show_id = season['id']
        data['show_id'] = show_id
-        request = sanitized_Request(
+        show = self._download_json(
            'http://admin.mangomolo.com/analytics/index.php/plus/show',
-            urlencode_postdata(data),
+            show_id, data=urlencode_postdata(data), headers={
-            {
+                'Origin': 'http://awaan.ae',
                'Origin': 'http://www.dcndigital.ae',
                'Content-Type': 'application/x-www-form-urlencoded'
            })
        show = self._download_json(request, show_id)
        if not season_id:
            season_id = show['default_season']
        for season in show['seasons']:
@@ -195,6 +194,6 @@ class DCNSeasonIE(InfoExtractor):
                for video in show['videos']:
                    video_id = compat_str(video['id'])
                    entries.append(self.url_result(
-                        'http://www.dcndigital.ae/media/%s' % video_id, 'DCNVideo', video_id))
+                        'http://awaan.ae/media/%s' % video_id, 'AWAANVideo', video_id))
                return self.playlist_result(entries, season_id, title)
--- a/youtube_dl/extractor/bandcamp.py
+++ b/youtube_dl/extractor/bandcamp.py
@@ -162,6 +162,15 @@ class BandcampAlbumIE(InfoExtractor):
            'uploader_id': 'dotscale',
        },
        'playlist_mincount': 7,
    }, {
        # with escaped quote in title
        'url': 'https://jstrecords.bandcamp.com/album/entropy-ep',
        'info_dict': {
            'title': '"Entropy" EP',
            'uploader_id': 'jstrecords',
            'id': 'entropy-ep',
        },
        'playlist_mincount': 3,
    }]
    def _real_extract(self, url):
@@ -176,8 +185,11 @@ class BandcampAlbumIE(InfoExtractor):
        entries = [
            self.url_result(compat_urlparse.urljoin(url, t_path), ie=BandcampIE.ie_key())
            for t_path in tracks_paths]
-        title = self._search_regex(
+        title = self._html_search_regex(
-            r'album_title\s*:\s*"(.*?)"', webpage, 'title', fatal=False)
+            r'album_title\s*:\s*"((?:\\.|[^"\\])+?)"',
            webpage, 'title', fatal=False)
        if title:
            title = title.replace(r'\"', '"')
        return {
            '_type': 'playlist',
            'uploader_id': uploader_id,
--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@@ -2,6 +2,7 @@
 from __future__ import unicode_literals
 import re
 import itertools
 from .common import InfoExtractor
 from ..utils import (
@@ -17,6 +18,7 @@ from ..utils import (
 from ..compat import (
    compat_etree_fromstring,
    compat_HTTPError,
    compat_urlparse,
 )
@@ -759,7 +761,7 @@ class BBCIE(BBCCoUkIE):
        webpage = self._download_webpage(url, playlist_id)
-        json_ld_info = self._search_json_ld(webpage, playlist_id, default=None)
+        json_ld_info = self._search_json_ld(webpage, playlist_id, default={})
        timestamp = json_ld_info.get('timestamp')
        playlist_title = json_ld_info.get('title')
@@ -1056,19 +1058,35 @@ class BBCCoUkArticleIE(InfoExtractor):
 class BBCCoUkPlaylistBaseIE(InfoExtractor):
    def _entries(self, webpage, url, playlist_id):
        single_page = 'page' in compat_urlparse.parse_qs(
            compat_urlparse.urlparse(url).query)
        for page_num in itertools.count(2):
            for video_id in re.findall(
                    self._VIDEO_ID_TEMPLATE % BBCCoUkIE._ID_REGEX, webpage):
                yield self.url_result(
                    self._URL_TEMPLATE % video_id, BBCCoUkIE.ie_key())
            if single_page:
                return
            next_page = self._search_regex(
                r'<li[^>]+class=(["\'])pagination_+next\1[^>]*><a[^>]+href=(["\'])(?P<url>(?:(?!\2).)+)\2',
                webpage, 'next page url', default=None, group='url')
            if not next_page:
                break
            webpage = self._download_webpage(
                compat_urlparse.urljoin(url, next_page), playlist_id,
                'Downloading page %d' % page_num, page_num)
    def _real_extract(self, url):
        playlist_id = self._match_id(url)
        webpage = self._download_webpage(url, playlist_id)
        entries = [
            self.url_result(self._URL_TEMPLATE % video_id, BBCCoUkIE.ie_key())
            for video_id in re.findall(
                self._VIDEO_ID_TEMPLATE % BBCCoUkIE._ID_REGEX, webpage)]
        title, description = self._extract_title_and_description(webpage)
-        return self.playlist_result(entries, playlist_id, title, description)
+        return self.playlist_result(
            self._entries(webpage, url, playlist_id),
            playlist_id, title, description)
 class BBCCoUkIPlayerPlaylistIE(BBCCoUkPlaylistBaseIE):
@@ -1117,6 +1135,24 @@ class BBCCoUkPlaylistIE(BBCCoUkPlaylistBaseIE):
            'description': 'French thriller serial about a missing teenager.',
        },
        'playlist_mincount': 7,
    }, {
        # multipage playlist, explicit page
        'url': 'http://www.bbc.co.uk/programmes/b00mfl7n/clips?page=1',
        'info_dict': {
            'id': 'b00mfl7n',
            'title': 'Frozen Planet - Clips - BBC One',
            'description': 'md5:65dcbf591ae628dafe32aa6c4a4a0d8c',
        },
        'playlist_mincount': 24,
    }, {
        # multipage playlist, all pages
        'url': 'http://www.bbc.co.uk/programmes/b00mfl7n/clips',
        'info_dict': {
            'id': 'b00mfl7n',
            'title': 'Frozen Planet - Clips - BBC One',
            'description': 'md5:65dcbf591ae628dafe32aa6c4a4a0d8c',
        },
        'playlist_mincount': 142,
    }, {
        'url': 'http://www.bbc.co.uk/programmes/b05rcz9v/broadcasts/2016/06',
        'only_matching': True,
--- a/youtube_dl/extractor/bet.py
+++ b/youtube_dl/extractor/bet.py
@@ -2,7 +2,6 @@ from __future__ import unicode_literals
 from .mtv import MTVServicesInfoExtractor
 from ..utils import unified_strdate
 from ..compat import compat_urllib_parse_urlencode
 class BetIE(MTVServicesInfoExtractor):
@@ -53,9 +52,9 @@ class BetIE(MTVServicesInfoExtractor):
    _FEED_URL = "http://feeds.mtvnservices.com/od/feed/bet-mrss-player"
    def _get_feed_query(self, uri):
-        return compat_urllib_parse_urlencode({
+        return {
            'uuid': uri,
-        })
+        }
    def _extract_mgid(self, webpage):
        return self._search_regex(r'data-uri="([^"]+)', webpage, 'mgid')
--- a/youtube_dl/extractor/bigflix.py
+++ b/youtube_dl/extractor/bigflix.py
@@ -11,15 +11,6 @@ from ..compat import compat_urllib_parse_unquote
 class BigflixIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?bigflix\.com/.+/(?P<id>[0-9]+)'
    _TESTS = [{
        'url': 'http://www.bigflix.com/Hindi-movies/Action-movies/Singham-Returns/16537',
        'md5': 'dc1b4aebb46e3a7077ecc0d9f43f61e3',
        'info_dict': {
            'id': '16537',
            'ext': 'mp4',
            'title': 'Singham Returns',
            'description': 'md5:3d2ba5815f14911d5cc6a501ae0cf65d',
        }
    }, {
        # 2 formats
        'url': 'http://www.bigflix.com/Tamil-movies/Drama-movies/Madarasapatinam/16070',
        'info_dict': {
--- a/youtube_dl/extractor/bilibili.py
+++ b/youtube_dl/extractor/bilibili.py
@@ -1,22 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals
-import calendar
+import hashlib
 import datetime
 import re
 from .common import InfoExtractor
-from ..compat import (
+from ..compat import compat_parse_qs
    compat_etree_fromstring,
    compat_str,
    compat_parse_qs,
    compat_xml_parse_error,
 )
 from ..utils import (
    ExtractorError,
    int_or_none,
    float_or_none,
-    xpath_text,
+    unified_timestamp,
 )
@@ -25,13 +18,13 @@ class BiliBiliIE(InfoExtractor):
    _TESTS = [{
        'url': 'http://www.bilibili.tv/video/av1074402/',
-        'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
+        'md5': '9fa226fe2b8a9a4d5a69b4c6a183417e',
        'info_dict': {
-            'id': '1554319',
+            'id': '1074402',
-            'ext': 'flv',
+            'ext': 'mp4',
            'title': '【金坷垃】金泡沫',
            'description': 'md5:ce18c2a2d2193f0df2917d270f2e5923',
-            'duration': 308.067,
+            'duration': 308.315,
            'timestamp': 1398012660,
            'upload_date': '20140420',
            'thumbnail': 're:^https?://.+\.jpg',
@@ -42,76 +35,42 @@ class BiliBiliIE(InfoExtractor):
        'url': 'http://www.bilibili.com/video/av1041170/',
        'info_dict': {
            'id': '1041170',
            'ext': 'mp4',
            'title': '【BD1080P】刀语【诸神&异域】',
            'description': '这是个神奇的故事~每个人不留弹幕不给走哦~切利哦！~',
            'duration': 3382.259,
            'timestamp': 1396530060,
            'upload_date': '20140403',
            'thumbnail': 're:^https?://.+\.jpg',
            'uploader': '枫叶逝去',
            'uploader_id': '520116',
        },
        'playlist_count': 9,
    }, {
        'url': 'http://www.bilibili.com/video/av4808130/',
        'info_dict': {
            'id': '4808130',
            'ext': 'mp4',
            'title': '【长篇】哆啦A梦443【钉铛】',
            'description': '(2016.05.27)来组合客人的脸吧&amp;amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;amp;illust_id=56912929',
            'duration': 1493.995,
            'timestamp': 1464564180,
            'upload_date': '20160529',
            'thumbnail': 're:^https?://.+\.jpg',
            'uploader': '喜欢拉面',
            'uploader_id': '151066',
        },
        'playlist': [{
            'md5': '55cdadedf3254caaa0d5d27cf20a8f9c',
            'info_dict': {
                'id': '4808130_part1',
                'ext': 'flv',
                'title': '【长篇】哆啦A梦443【钉铛】',
                'description': '(2016.05.27)来组合客人的脸吧&amp;amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;amp;illust_id=56912929',
                'timestamp': 1464564180,
                'upload_date': '20160529',
                'uploader': '喜欢拉面',
                'uploader_id': '151066',
            },
        }, {
            'md5': '926f9f67d0c482091872fbd8eca7ea3d',
            'info_dict': {
                'id': '4808130_part2',
                'ext': 'flv',
                'title': '【长篇】哆啦A梦443【钉铛】',
                'description': '(2016.05.27)来组合客人的脸吧&amp;amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;amp;illust_id=56912929',
                'timestamp': 1464564180,
                'upload_date': '20160529',
                'uploader': '喜欢拉面',
                'uploader_id': '151066',
            },
        }, {
            'md5': '4b7b225b968402d7c32348c646f1fd83',
            'info_dict': {
                'id': '4808130_part3',
                'ext': 'flv',
                'title': '【长篇】哆啦A梦443【钉铛】',
                'description': '(2016.05.27)来组合客人的脸吧&amp;amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;amp;illust_id=56912929',
                'timestamp': 1464564180,
                'upload_date': '20160529',
                'uploader': '喜欢拉面',
                'uploader_id': '151066',
            },
        }, {
            'md5': '7b795e214166501e9141139eea236e91',
            'info_dict': {
                'id': '4808130_part4',
                'ext': 'flv',
                'title': '【长篇】哆啦A梦443【钉铛】',
                'description': '(2016.05.27)来组合客人的脸吧&amp;amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;amp;illust_id=56912929',
                'timestamp': 1464564180,
                'upload_date': '20160529',
                'uploader': '喜欢拉面',
                'uploader_id': '151066',
            },
        }],
    }, {
        # Missing upload time
        'url': 'http://www.bilibili.com/video/av1867637/',
        'info_dict': {
-            'id': '2880301',
+            'id': '1867637',
-            'ext': 'flv',
+            'ext': 'mp4',
            'title': '【HDTV】【喜剧】岳父岳母真难当 （2014）【法国票房冠军】',
            'description': '一个信奉天主教的法国旧式传统资产阶级家庭中有四个女儿。三个女儿却分别找了阿拉伯、犹太、中国丈夫，老夫老妻唯独期盼剩下未嫁的小女儿能找一个信奉天主教的法国白人，结果没想到小女儿找了一位非裔黑人……【这次应该不会跳帧了】',
            'duration': 5760.0,
            'uploader': '黑夜为猫',
            'uploader_id': '610729',
            'thumbnail': 're:^https?://.+\.jpg',
        },
        'params': {
            # Just to test metadata extraction
@@ -120,86 +79,61 @@ class BiliBiliIE(InfoExtractor):
        'expected_warnings': ['upload time'],
    }]
-    # BiliBili blocks keys from time to time. The current key is extracted from
+    _APP_KEY = '6f90a59ac58a4123'
-    # the Android client
+    _BILIBILI_KEY = '0bfd84cc3940035173f35e6777508326'
    # TODO: find the sign algorithm used in the flash player
    _APP_KEY = '86385cdc024c0f6c'
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id = self._match_id(url)
        video_id = mobj.group('id')
        webpage = self._download_webpage(url, video_id)
-        params = compat_parse_qs(self._search_regex(
+        cid = compat_parse_qs(self._search_regex(
            [r'EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
             r'<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
-            webpage, 'player parameters'))
+            webpage, 'player parameters'))['cid'][0]
        cid = params['cid'][0]
-        info_xml_str = self._download_webpage(
+        payload = 'appkey=%s&cid=%s&otype=json&quality=2&type=mp4' % (self._APP_KEY, cid)
-            'http://interface.bilibili.com/v_cdn_play',
+        sign = hashlib.md5((payload + self._BILIBILI_KEY).encode('utf-8')).hexdigest()
            cid, query={'appkey': self._APP_KEY, 'cid': cid},
            note='Downloading video info page')
-        err_msg = None
+        video_info = self._download_json(
-        durls = None
+            'http://interface.bilibili.com/playurl?%s&sign=%s' % (payload, sign),
-        info_xml = None
+            video_id, note='Downloading video info page')
        try:
            info_xml = compat_etree_fromstring(info_xml_str.encode('utf-8'))
        except compat_xml_parse_error:
            info_json = self._parse_json(info_xml_str, video_id, fatal=False)
            err_msg = (info_json or {}).get('error_text')
        else:
            err_msg = xpath_text(info_xml, './message')
        if info_xml is not None:
            durls = info_xml.findall('./durl')
        if not durls:
            if err_msg:
                raise ExtractorError('%s said: %s' % (self.IE_NAME, err_msg), expected=True)
            else:
                raise ExtractorError('No videos found!')
        entries = []
-        for durl in durls:
+        for idx, durl in enumerate(video_info['durl']):
            size = xpath_text(durl, ['./filesize', './size'])
            formats = [{
-                'url': durl.find('./url').text,
+                'url': durl['url'],
-                'filesize': int_or_none(size),
+                'filesize': int_or_none(durl['size']),
            }]
-            for backup_url in durl.findall('./backup_url/url'):
+            for backup_url in durl['backup_url']:
                formats.append({
-                    'url': backup_url.text,
+                    'url': backup_url,
                    # backup URLs have lower priorities
-                    'preference': -2 if 'hd.mp4' in backup_url.text else -3,
+                    'preference': -2 if 'hd.mp4' in backup_url else -3,
                })
            self._sort_formats(formats)
            entries.append({
-                'id': '%s_part%s' % (cid, xpath_text(durl, './order')),
+                'id': '%s_part%s' % (video_id, idx),
-                'duration': int_or_none(xpath_text(durl, './length'), 1000),
+                'duration': float_or_none(durl.get('length'), 1000),
                'formats': formats,
            })
        title = self._html_search_regex('<h1[^>]+title="([^"]+)">', webpage, 'title')
        description = self._html_search_meta('description', webpage)
-        datetime_str = self._html_search_regex(
+        timestamp = unified_timestamp(self._html_search_regex(
-            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False)
+            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False))
        timestamp = None
        if datetime_str:
            timestamp = calendar.timegm(datetime.datetime.strptime(datetime_str, '%Y-%m-%dT%H:%M').timetuple())
        # TODO 'view_count' requires deobfuscating Javascript
        info = {
-            'id': compat_str(cid),
+            'id': video_id,
            'title': title,
            'description': description,
            'timestamp': timestamp,
            'thumbnail': self._html_search_meta('thumbnailUrl', webpage),
-            'duration': float_or_none(xpath_text(info_xml, './timelength'), scale=1000),
+            'duration': float_or_none(video_info.get('timelength'), scale=1000),
        }
        uploader_mobj = re.search(
--- a/youtube_dl/extractor/biqle.py
+++ b/youtube_dl/extractor/biqle.py
@@ -24,7 +24,8 @@ class BIQLEIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Ребенок в шоке от автоматической мойки',
            'uploader': 'Dmitry Kotov',
-        }
+        },
        'skip': ' This video was marked as adult.  Embedding adult videos on external sites is prohibited.',
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/bravotv.py
+++ b/youtube_dl/extractor/bravotv.py
@@ -1,31 +1,74 @@
 # coding: utf-8
 from __future__ import unicode_literals
-from .common import InfoExtractor
+from .adobepass import AdobePassIE
-from ..utils import smuggle_url
+from ..utils import (
    smuggle_url,
    update_url_query,
    int_or_none,
 )
-class BravoTVIE(InfoExtractor):
+class BravoTVIE(AdobePassIE):
-    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+videos/(?P<id>[^/?]+)'
+    _VALID_URL = r'https?://(?:www\.)?bravotv\.com/(?:[^/]+/)+(?P<id>[^/?#]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://www.bravotv.com/last-chance-kitchen/season-5/videos/lck-ep-12-fishy-finale',
-        'md5': 'd60cdf68904e854fac669bd26cccf801',
+        'md5': '9086d0b7ef0ea2aabc4781d75f4e5863',
        'info_dict': {
-            'id': 'LitrBdX64qLn',
+            'id': 'zHyk1_HU_mPy',
            'ext': 'mp4',
-            'title': 'Last Chance Kitchen Returns',
+            'title': 'LCK Ep 12: Fishy Finale',
-            'description': 'S13: Last Chance Kitchen Returns for Top Chef Season 13',
+            'description': 'S13/E12: Two eliminated chefs have just 12 minutes to cook up a delicious fish dish.',
            'timestamp': 1448926740,
            'upload_date': '20151130',
            'uploader': 'NBCU-BRAV',
            'upload_date': '20160302',
            'timestamp': 1456945320,
        }
-    }
+    }, {
        'url': 'http://www.bravotv.com/below-deck/season-3/ep-14-reunion-part-1',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        webpage = self._download_webpage(url, display_id)
-        account_pid = self._search_regex(r'"account_pid"\s*:\s*"([^"]+)"', webpage, 'account pid')
+        settings = self._parse_json(self._search_regex(
-        release_pid = self._search_regex(r'"release_pid"\s*:\s*"([^"]+)"', webpage, 'release pid')
+            r'jQuery\.extend\(Drupal\.settings\s*,\s*({.+?})\);', webpage, 'drupal settings'),
-        return self.url_result(smuggle_url(
+            display_id)
-            'http://link.theplatform.com/s/%s/%s?mbr=true&switch=progressive' % (account_pid, release_pid),
+        info = {}
-            {'force_smil_url': True}), 'ThePlatform', release_pid)
+        query = {
            'mbr': 'true',
        }
        account_pid, release_pid = [None] * 2
        tve = settings.get('sharedTVE')
        if tve:
            query['manifest'] = 'm3u'
            account_pid = 'HNK2IC'
            release_pid = tve['release_pid']
            if tve.get('entitlement') == 'auth':
                adobe_pass = settings.get('adobePass', {})
                resource = self._get_mvpd_resource(
                    adobe_pass.get('adobePassResourceId', 'bravo'),
                    tve['title'], release_pid, tve.get('rating'))
                query['auth'] = self._extract_mvpd_auth(
                    url, release_pid, adobe_pass.get('adobePassRequestorId', 'bravo'), resource)
        else:
            shared_playlist = settings['shared_playlist']
            account_pid = shared_playlist['account_pid']
            metadata = shared_playlist['video_metadata'][shared_playlist['default_clip']]
            release_pid = metadata['release_pid']
            info.update({
                'title': metadata['title'],
                'description': metadata.get('description'),
                'season_number': int_or_none(metadata.get('season_num')),
                'episode_number': int_or_none(metadata.get('episode_num')),
            })
            query['switch'] = 'progressive'
        info.update({
            '_type': 'url_transparent',
            'id': release_pid,
            'url': smuggle_url(update_url_query(
                'http://link.theplatform.com/s/%s/%s' % (account_pid, release_pid),
                query), {'force_smil_url': True}),
            'ie_key': 'ThePlatform',
        })
        return info
--- a/youtube_dl/extractor/cartoonnetwork.py
+++ b/youtube_dl/extractor/cartoonnetwork.py
@@ -0,0 +1,36 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .turner import TurnerBaseIE
 class CartoonNetworkIE(TurnerBaseIE):
    _VALID_URL = r'https?://(?:www\.)?cartoonnetwork\.com/video/(?:[^/]+/)+(?P<id>[^/?#]+)-(?:clip|episode)\.html'
    _TEST = {
        'url': 'http://www.cartoonnetwork.com/video/teen-titans-go/starfire-the-cat-lady-clip.html',
        'info_dict': {
            'id': '8a250ab04ed07e6c014ef3f1e2f9016c',
            'ext': 'mp4',
            'title': 'Starfire the Cat Lady',
            'description': 'Robin decides to become a cat so that Starfire will finally love him.',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        id_type, video_id = re.search(r"_cnglobal\.cvp(Video|Title)Id\s*=\s*'([^']+)';", webpage).groups()
        query = ('id' if id_type == 'Video' else 'titleId') + '=' + video_id
        return self._extract_cvp_info(
            'http://www.cartoonnetwork.com/video-seo-svc/episodeservices/getCvpPlaylist?networkName=CN2&' + query, video_id, {
                'secure': {
                    'media_src': 'http://apple-secure.cdn.turner.com/toon/big',
                    'tokenizer_src': 'http://www.cartoonnetwork.com/cntv/mvpd/processors/services/token_ipadAdobe.do',
                },
            })
--- a/youtube_dl/extractor/cbc.py
+++ b/youtube_dl/extractor/cbc.py
@@ -9,10 +9,19 @@ from ..utils import (
    js_to_json,
    smuggle_url,
    try_get,
    xpath_text,
    xpath_element,
    xpath_with_ns,
    find_xpath_attr,
    parse_iso8601,
    parse_age_limit,
    int_or_none,
    ExtractorError,
 )
 class CBCIE(InfoExtractor):
    IE_NAME = 'cbc.ca'
    _VALID_URL = r'https?://(?:www\.)?cbc\.ca/(?!player/)(?:[^/]+/)+(?P<id>[^/?#]+)'
    _TESTS = [{
        # with mediaId
@@ -114,6 +123,7 @@ class CBCIE(InfoExtractor):
 class CBCPlayerIE(InfoExtractor):
    IE_NAME = 'cbc.ca:player'
    _VALID_URL = r'(?:cbcplayer:|https?://(?:www\.)?cbc\.ca/(?:player/play/|i/caffeine/syndicate/\?mediaId=))(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.cbc.ca/player/play/2683190193',
@@ -167,3 +177,165 @@ class CBCPlayerIE(InfoExtractor):
                }),
            'id': video_id,
        }
 class CBCWatchBaseIE(InfoExtractor):
    _device_id = None
    _device_token = None
    _API_BASE_URL = 'https://api-cbc.cloud.clearleap.com/cloffice/client/'
    _NS_MAP = {
        'media': 'http://search.yahoo.com/mrss/',
        'clearleap': 'http://www.clearleap.com/namespace/clearleap/1.0/',
    }
    def _call_api(self, path, video_id):
        url = path if path.startswith('http') else self._API_BASE_URL + path
        result = self._download_xml(url, video_id, headers={
            'X-Clearleap-DeviceId': self._device_id,
            'X-Clearleap-DeviceToken': self._device_token,
        })
        error_message = xpath_text(result, 'userMessage') or xpath_text(result, 'systemMessage')
        if error_message:
            raise ExtractorError('%s said: %s' % (self.IE_NAME, error_message))
        return result
    def _real_initialize(self):
        if not self._device_id or not self._device_token:
            device = self._downloader.cache.load('cbcwatch', 'device') or {}
            self._device_id, self._device_token = device.get('id'), device.get('token')
            if not self._device_id or not self._device_token:
                result = self._download_xml(
                    self._API_BASE_URL + 'device/register',
                    None, data=b'<device><type>web</type></device>')
                self._device_id = xpath_text(result, 'deviceId', fatal=True)
                self._device_token = xpath_text(result, 'deviceToken', fatal=True)
                self._downloader.cache.store(
                    'cbcwatch', 'device', {
                        'id': self._device_id,
                        'token': self._device_token,
                    })
    def _parse_rss_feed(self, rss):
        channel = xpath_element(rss, 'channel', fatal=True)
        def _add_ns(path):
            return xpath_with_ns(path, self._NS_MAP)
        entries = []
        for item in channel.findall('item'):
            guid = xpath_text(item, 'guid', fatal=True)
            title = xpath_text(item, 'title', fatal=True)
            media_group = xpath_element(item, _add_ns('media:group'), fatal=True)
            content = xpath_element(media_group, _add_ns('media:content'), fatal=True)
            content_url = content.attrib['url']
            thumbnails = []
            for thumbnail in media_group.findall(_add_ns('media:thumbnail')):
                thumbnail_url = thumbnail.get('url')
                if not thumbnail_url:
                    continue
                thumbnails.append({
                    'id': thumbnail.get('profile'),
                    'url': thumbnail_url,
                    'width': int_or_none(thumbnail.get('width')),
                    'height': int_or_none(thumbnail.get('height')),
                })
            timestamp = None
            release_date = find_xpath_attr(
                item, _add_ns('media:credit'), 'role', 'releaseDate')
            if release_date is not None:
                timestamp = parse_iso8601(release_date.text)
            entries.append({
                '_type': 'url_transparent',
                'url': content_url,
                'id': guid,
                'title': title,
                'description': xpath_text(item, 'description'),
                'timestamp': timestamp,
                'duration': int_or_none(content.get('duration')),
                'age_limit': parse_age_limit(xpath_text(item, _add_ns('media:rating'))),
                'episode': xpath_text(item, _add_ns('clearleap:episode')),
                'episode_number': int_or_none(xpath_text(item, _add_ns('clearleap:episodeInSeason'))),
                'series': xpath_text(item, _add_ns('clearleap:series')),
                'season_number': int_or_none(xpath_text(item, _add_ns('clearleap:season'))),
                'thumbnails': thumbnails,
                'ie_key': 'CBCWatchVideo',
            })
        return self.playlist_result(
            entries, xpath_text(channel, 'guid'),
            xpath_text(channel, 'title'),
            xpath_text(channel, 'description'))
 class CBCWatchVideoIE(CBCWatchBaseIE):
    IE_NAME = 'cbc.ca:watch:video'
    _VALID_URL = r'https?://api-cbc\.cloud\.clearleap\.com/cloffice/client/web/play/?\?.*?\bcontentId=(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
    def _real_extract(self, url):
        video_id = self._match_id(url)
        result = self._call_api(url, video_id)
        m3u8_url = xpath_text(result, 'url', fatal=True)
        formats = self._extract_m3u8_formats(re.sub(r'/([^/]+)/[^/?]+\.m3u8', r'/\1/\1.m3u8', m3u8_url), video_id, 'mp4', fatal=False)
        if len(formats) < 2:
            formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4')
        # Despite metadata in m3u8 all video+audio formats are
        # actually video-only (no audio)
        for f in formats:
            if f.get('acodec') != 'none' and f.get('vcodec') != 'none':
                f['acodec'] = 'none'
        self._sort_formats(formats)
        info = {
            'id': video_id,
            'title': video_id,
            'formats': formats,
        }
        rss = xpath_element(result, 'rss')
        if rss:
            info.update(self._parse_rss_feed(rss)['entries'][0])
            del info['url']
            del info['_type']
            del info['ie_key']
        return info
 class CBCWatchIE(CBCWatchBaseIE):
    IE_NAME = 'cbc.ca:watch'
    _VALID_URL = r'https?://watch\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
    _TESTS = [{
        'url': 'http://watch.cbc.ca/doc-zone/season-6/customer-disservice/38e815a-009e3ab12e4',
        'info_dict': {
            'id': '38e815a-009e3ab12e4',
            'ext': 'mp4',
            'title': 'Customer (Dis)Service',
            'description': 'md5:8bdd6913a0fe03d4b2a17ebe169c7c87',
            'upload_date': '20160219',
            'timestamp': 1455840000,
        },
        'params': {
            # m3u8 download
            'skip_download': True,
            'format': 'bestvideo',
        },
        'skip': 'Geo-restricted to Canada',
    }, {
        'url': 'http://watch.cbc.ca/arthur/all/1ed4b385-cd84-49cf-95f0-80f004680057',
        'info_dict': {
            'id': '1ed4b385-cd84-49cf-95f0-80f004680057',
            'title': 'Arthur',
            'description': 'Arthur, the sweetest 8-year-old aardvark, and his pals solve all kinds of problems with humour, kindness and teamwork.',
        },
        'playlist_mincount': 30,
        'skip': 'Geo-restricted to Canada',
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        rss = self._call_api('web/browse/' + video_id, video_id)
        return self._parse_rss_feed(rss)
--- a/youtube_dl/extractor/cbs.py
+++ b/youtube_dl/extractor/cbs.py
@@ -4,6 +4,7 @@ from .theplatform import ThePlatformFeedIE
 from ..utils import (
    int_or_none,
    find_xpath_attr,
    ExtractorError,
 )
@@ -17,19 +18,6 @@ class CBSBaseIE(ThePlatformFeedIE):
            }]
        } if closed_caption_e is not None and closed_caption_e.attrib.get('value') else []
    def _extract_video_info(self, filter_query, video_id):
        return self._extract_feed_info(
            'dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id, lambda entry: {
                'series': entry.get('cbs$SeriesTitle'),
                'season_number': int_or_none(entry.get('cbs$SeasonNumber')),
                'episode': entry.get('cbs$EpisodeTitle'),
                'episode_number': int_or_none(entry.get('cbs$EpisodeNumber')),
            }, {
                'StreamPack': {
                    'manifest': 'm3u',
                }
            })
 class CBSIE(CBSBaseIE):
    _VALID_URL = r'(?:cbs:|https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/video|colbertlateshow\.com/(?:video|podcasts))/)(?P<id>[\w-]+)'
@@ -38,7 +26,6 @@ class CBSIE(CBSBaseIE):
        'url': 'http://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
        'info_dict': {
            'id': '_u7W953k6la293J7EPTd9oHkSPs6Xn6_',
            'display_id': 'connect-chat-feat-garth-brooks',
            'ext': 'mp4',
            'title': 'Connect Chat feat. Garth Brooks',
            'description': 'Connect with country music singer Garth Brooks, as he chats with fans on Wednesday November 27, 2013. Be sure to tune in to Garth Brooks: Live from Las Vegas, Friday November 29, at 9/8c on CBS!',
@@ -47,7 +34,10 @@ class CBSIE(CBSBaseIE):
            'upload_date': '20131127',
            'uploader': 'CBSI-NEW',
        },
-        'expected_warnings': ['Failed to download m3u8 information'],
+        'params': {
            # m3u8 download
            'skip_download': True,
        },
        '_skip': 'Blocked outside the US',
    }, {
        'url': 'http://colbertlateshow.com/video/8GmB0oY0McANFvp2aEffk9jZZZ2YyXxy/the-colbeard/',
@@ -56,8 +46,31 @@ class CBSIE(CBSBaseIE):
        'url': 'http://www.colbertlateshow.com/podcasts/dYSwjqPs_X1tvbV_P2FcPWRa_qT6akTC/in-the-bad-room-with-stephen/',
        'only_matching': True,
    }]
-    TP_RELEASE_URL_TEMPLATE = 'http://link.theplatform.com/s/dJ5BDC/%s?mbr=true'
+
    def _extract_video_info(self, guid):
        path = 'dJ5BDC/media/guid/2198311517/' + guid
        smil_url = 'http://link.theplatform.com/s/%s?mbr=true' % path
        formats, subtitles = self._extract_theplatform_smil(smil_url + '&manifest=m3u', guid)
        for r in ('OnceURL&formats=M3U', 'HLS&formats=M3U', 'RTMP', 'WIFI', '3G'):
            try:
                tp_formats, _ = self._extract_theplatform_smil(smil_url + '&assetTypes=' + r, guid, 'Downloading %s SMIL data' % r.split('&')[0])
                formats.extend(tp_formats)
            except ExtractorError:
                continue
        self._sort_formats(formats)
        metadata = self._download_theplatform_metadata(path, guid)
        info = self._parse_theplatform_metadata(metadata)
        info.update({
            'id': guid,
            'formats': formats,
            'subtitles': subtitles,
            'series': metadata.get('cbs$SeriesTitle'),
            'season_number': int_or_none(metadata.get('cbs$SeasonNumber')),
            'episode': metadata.get('cbs$EpisodeTitle'),
            'episode_number': int_or_none(metadata.get('cbs$EpisodeNumber')),
        })
        return info
    def _real_extract(self, url):
        content_id = self._match_id(url)
-        return self._extract_video_info('byGuid=%s' % content_id, content_id)
+        return self._extract_video_info(content_id)
--- a/youtube_dl/extractor/cbslocal.py
+++ b/youtube_dl/extractor/cbslocal.py
@@ -41,13 +41,8 @@ class CBSLocalIE(AnvatoIE):
        'url': 'http://cleveland.cbslocal.com/2016/05/16/indians-score-season-high-15-runs-in-blowout-win-over-reds-rapid-reaction/',
        'info_dict': {
            'id': 'GxfCe0Zo7D-175909-5588',
            'ext': 'mp4',
            'title': 'Recap: CLE 15, CIN 6',
            'description': '5/16/16: Indians\' bats explode for 15 runs in a win',
            'upload_date': '20160516',
            'timestamp': 1463433840,
            'duration': 49,
        },
        'playlist_count': 9,
        'params': {
            # m3u8 download
            'skip_download': True,
@@ -60,12 +55,11 @@ class CBSLocalIE(AnvatoIE):
        sendtonews_url = SendtoNewsIE._extract_url(webpage)
        if sendtonews_url:
-            info_dict = {
+            return self.url_result(
-                '_type': 'url_transparent',
+                compat_urlparse.urljoin(url, sendtonews_url),
-                'url': compat_urlparse.urljoin(url, sendtonews_url),
+                ie=SendtoNewsIE.ie_key())
-            }
+
-        else:
+        info_dict = self._extract_anvato_videos(webpage, display_id)
            info_dict = self._extract_anvato_videos(webpage, display_id)
        time_str = self._html_search_regex(
            r'class="entry-date">([^<]+)<', webpage, 'released date', fatal=False)
--- a/youtube_dl/extractor/cbsnews.py
+++ b/youtube_dl/extractor/cbsnews.py
@@ -2,13 +2,13 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
-from .cbs import CBSBaseIE
+from .cbs import CBSIE
 from ..utils import (
    parse_duration,
 )
-class CBSNewsIE(CBSBaseIE):
+class CBSNewsIE(CBSIE):
    IE_DESC = 'CBS News'
    _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/(?:news|videos)/(?P<id>[\da-z_-]+)'
@@ -35,7 +35,8 @@ class CBSNewsIE(CBSBaseIE):
                'ext': 'mp4',
                'title': 'Fort Hood shooting: Army downplays mental illness as cause of attack',
                'description': 'md5:4a6983e480542d8b333a947bfc64ddc7',
-                'upload_date': '19700101',
+                'upload_date': '20140404',
                'timestamp': 1396650660,
                'uploader': 'CBSI-NEW',
                'thumbnail': 're:^https?://.*\.jpg$',
                'duration': 205,
@@ -63,14 +64,15 @@ class CBSNewsIE(CBSBaseIE):
        item = video_info['item'] if 'item' in video_info else video_info
        guid = item['mpxRefId']
-        return self._extract_video_info('byGuid=%s' % guid, guid)
+        return self._extract_video_info(guid)
 class CBSNewsLiveVideoIE(InfoExtractor):
    IE_DESC = 'CBS News Live Videos'
    _VALID_URL = r'https?://(?:www\.)?cbsnews\.com/live/video/(?P<id>[\da-z_-]+)'
-    _TESTS = [{
+    # Live videos get deleted soon. See http://www.cbsnews.com/live/ for the latest examples
    _TEST = {
        'url': 'http://www.cbsnews.com/live/video/clinton-sanders-prepare-to-face-off-in-nh/',
        'info_dict': {
            'id': 'clinton-sanders-prepare-to-face-off-in-nh',
@@ -78,15 +80,8 @@ class CBSNewsLiveVideoIE(InfoExtractor):
            'title': 'Clinton, Sanders Prepare To Face Off In NH',
            'duration': 334,
        },
-        'skip': 'Video gone, redirected to http://www.cbsnews.com/live/',
+        'skip': 'Video gone',
-    }, {
+    }
        'url': 'http://www.cbsnews.com/live/video/video-shows-intense-paragliding-accident/',
        'info_dict': {
            'id': 'video-shows-intense-paragliding-accident',
            'ext': 'flv',
            'title': 'Video Shows Intense Paragliding Accident',
        },
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
--- a/youtube_dl/extractor/cbssports.py
+++ b/youtube_dl/extractor/cbssports.py
@@ -23,6 +23,9 @@ class CBSSportsIE(CBSBaseIE):
        }
    }]
    def _extract_video_info(self, filter_query, video_id):
        return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
    def _real_extract(self, url):
        video_id = self._match_id(url)
        return self._extract_video_info('byId=%s' % video_id, video_id)
--- a/youtube_dl/extractor/charlierose.py
+++ b/youtube_dl/extractor/charlierose.py
@@ -0,0 +1,51 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import remove_end
 class CharlieRoseIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?charlierose\.com/video(?:s|/player)/(?P<id>\d+)'
    _TESTS = [{
        'url': 'https://charlierose.com/videos/27996',
        'md5': 'fda41d49e67d4ce7c2411fd2c4702e09',
        'info_dict': {
            'id': '27996',
            'ext': 'mp4',
            'title': 'Remembering Zaha Hadid',
            'thumbnail': 're:^https?://.*\.jpg\?\d+',
            'description': 'We revisit past conversations with Zaha Hadid, in memory of the world renowned Iraqi architect.',
            'subtitles': {
                'en': [{
                    'ext': 'vtt',
                }],
            },
        },
    }, {
        'url': 'https://charlierose.com/videos/27996',
        'only_matching': True,
    }]
    _PLAYER_BASE = 'https://charlierose.com/video/player/%s'
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(self._PLAYER_BASE % video_id, video_id)
        title = remove_end(self._og_search_title(webpage), ' - Charlie Rose')
        info_dict = self._parse_html5_media_entries(
            self._PLAYER_BASE % video_id, webpage, video_id,
            m3u8_entry_protocol='m3u8_native')[0]
        self._sort_formats(info_dict['formats'])
        self._remove_duplicate_formats(info_dict['formats'])
        info_dict.update({
            'id': video_id,
            'title': title,
            'thumbnail': self._og_search_thumbnail(webpage),
            'description': self._og_search_description(webpage),
        })
        return info_dict
--- a/youtube_dl/extractor/chaturbate.py
+++ b/youtube_dl/extractor/chaturbate.py
@@ -17,7 +17,8 @@ class ChaturbateIE(InfoExtractor):
        },
        'params': {
            'skip_download': True,
-        }
+        },
        'skip': 'Room is offline',
    }, {
        'url': 'https://en.chaturbate.com/siswet19/',
        'only_matching': True,
--- a/youtube_dl/extractor/chirbit.py
+++ b/youtube_dl/extractor/chirbit.py
@@ -1,30 +1,33 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import base64
 from .common import InfoExtractor
-from ..utils import (
+from ..utils import parse_duration
    parse_duration,
    int_or_none,
 )
 class ChirbitIE(InfoExtractor):
    IE_NAME = 'chirbit'
    _VALID_URL = r'https?://(?:www\.)?chirb\.it/(?:(?:wp|pl)/|fb_chirbit_player\.swf\?key=)?(?P<id>[\da-zA-Z]+)'
    _TESTS = [{
-        'url': 'http://chirb.it/PrIPv5',
+        'url': 'http://chirb.it/be2abG',
        'md5': '9847b0dad6ac3e074568bf2cfb197de8',
        'info_dict': {
-            'id': 'PrIPv5',
+            'id': 'be2abG',
            'ext': 'mp3',
-            'title': 'Фасадстрой',
+            'title': 'md5:f542ea253f5255240be4da375c6a5d7e',
-            'duration': 52,
+            'description': 'md5:f24a4e22a71763e32da5fed59e47c770',
-            'view_count': int,
+            'duration': 306,
-            'comment_count': int,
+        },
        'params': {
            'skip_download': True,
        }
    }, {
        'url': 'https://chirb.it/fb_chirbit_player.swf?key=PrIPv5',
        'only_matching': True,
    }, {
        'url': 'https://chirb.it/wp/MN58c2',
        'only_matching': True,
    }]
    def _real_extract(self, url):
@@ -33,27 +36,30 @@ class ChirbitIE(InfoExtractor):
        webpage = self._download_webpage(
            'http://chirb.it/%s' % audio_id, audio_id)
-        audio_url = self._search_regex(
+        data_fd = self._search_regex(
-            r'"setFile"\s*,\s*"([^"]+)"', webpage, 'audio url')
+            r'data-fd=(["\'])(?P<url>(?:(?!\1).)+)\1',
            webpage, 'data fd', group='url')
        # Reverse engineered from https://chirb.it/js/chirbit.player.js (look
        # for soundURL)
        audio_url = base64.b64decode(
            data_fd[::-1].encode('ascii')).decode('utf-8')
        title = self._search_regex(
-            r'itemprop="name">([^<]+)', webpage, 'title')
+            r'class=["\']chirbit-title["\'][^>]*>([^<]+)', webpage, 'title')
-        duration = parse_duration(self._html_search_meta(
+        description = self._search_regex(
-            'duration', webpage, 'duration', fatal=False))
+            r'<h3>Description</h3>\s*<pre[^>]*>([^<]+)</pre>',
-        view_count = int_or_none(self._search_regex(
+            webpage, 'description', default=None)
-            r'itemprop="playCount"\s*>(\d+)', webpage,
+        duration = parse_duration(self._search_regex(
-            'listen count', fatal=False))
+            r'class=["\']c-length["\'][^>]*>([^<]+)',
-        comment_count = int_or_none(self._search_regex(
+            webpage, 'duration', fatal=False))
            r'>(\d+) Comments?:', webpage,
            'comment count', fatal=False))
        return {
            'id': audio_id,
            'url': audio_url,
            'title': title,
            'description': description,
            'duration': duration,
            'view_count': view_count,
            'comment_count': comment_count,
        }
--- a/youtube_dl/extractor/cnn.py
+++ b/youtube_dl/extractor/cnn.py
@@ -3,15 +3,12 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..utils import (
+from .turner import TurnerBaseIE
-    int_or_none,
+from ..utils import url_basename
    parse_duration,
    url_basename,
 )
-class CNNIE(InfoExtractor):
+class CNNIE(TurnerBaseIE):
-    _VALID_URL = r'''(?x)https?://(?:(?:edition|www)\.)?cnn\.com/video/(?:data/.+?|\?)/
+    _VALID_URL = r'''(?x)https?://(?:(?P<sub_domain>edition|www|money)\.)?cnn\.com/(?:video/(?:data/.+?|\?)/)?videos?/
        (?P<path>.+?/(?P<title>[^/]+?)(?:\.(?:[a-z\-]+)|(?=&)))'''
    _TESTS = [{
@@ -25,6 +22,7 @@ class CNNIE(InfoExtractor):
            'duration': 135,
            'upload_date': '20130609',
        },
        'expected_warnings': ['Failed to download m3u8 information'],
    }, {
        'url': 'http://edition.cnn.com/video/?/video/us/2013/08/21/sot-student-gives-epic-speech.georgia-institute-of-technology&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+rss%2Fcnn_topstories+%28RSS%3A+Top+Stories%29',
        'md5': 'b5cc60c60a3477d185af8f19a2a26f4e',
@@ -34,7 +32,8 @@ class CNNIE(InfoExtractor):
            'title': "Student's epic speech stuns new freshmen",
            'description': "A Georgia Tech student welcomes the incoming freshmen with an epic speech backed by music from \"2001: A Space Odyssey.\"",
            'upload_date': '20130821',
-        }
+        },
        'expected_warnings': ['Failed to download m3u8 information'],
    }, {
        'url': 'http://www.cnn.com/video/data/2.0/video/living/2014/12/22/growing-america-nashville-salemtown-board-episode-1.hln.html',
        'md5': 'f14d02ebd264df951feb2400e2c25a1b',
@@ -44,80 +43,61 @@ class CNNIE(InfoExtractor):
            'title': 'Nashville Ep. 1: Hand crafted skateboards',
            'description': 'md5:e7223a503315c9f150acac52e76de086',
            'upload_date': '20141222',
-        }
+        },
        'expected_warnings': ['Failed to download m3u8 information'],
    }, {
        'url': 'http://money.cnn.com/video/news/2016/08/19/netflix-stunning-stats.cnnmoney/index.html',
        'md5': '52a515dc1b0f001cd82e4ceda32be9d1',
        'info_dict': {
            'id': '/video/news/2016/08/19/netflix-stunning-stats.cnnmoney',
            'ext': 'mp4',
            'title': '5 stunning stats about Netflix',
            'description': 'Did you know that Netflix has more than 80 million members? Here are five facts about the online video distributor that you probably didn\'t know.',
            'upload_date': '20160819',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://cnn.com/video/?/video/politics/2015/03/27/pkg-arizona-senator-church-attendance-mandatory.ktvk',
        'only_matching': True,
    }, {
        'url': 'http://cnn.com/video/?/video/us/2015/04/06/dnt-baker-refuses-anti-gay-order.wkmg',
        'only_matching': True,
    }, {
        'url': 'http://edition.cnn.com/videos/arts/2016/04/21/olympic-games-cultural-a-z-brazil.cnn',
        'only_matching': True,
    }]
    _CONFIG = {
        # http://edition.cnn.com/.element/apps/cvp/3.0/cfg/spider/cnn/expansion/config.xml
        'edition': {
            'data_src': 'http://edition.cnn.com/video/data/3.0/video/%s/index.xml',
            'media_src': 'http://pmd.cdn.turner.com/cnn/big',
        },
        # http://money.cnn.com/.element/apps/cvp2/cfg/config.xml
        'money': {
            'data_src': 'http://money.cnn.com/video/data/4.0/video/%s.xml',
            'media_src': 'http://ht3.cdn.turner.com/money/big',
        },
    }
    def _extract_timestamp(self, video_data):
        # TODO: fix timestamp extraction
        return None
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        sub_domain, path, page_title = re.match(self._VALID_URL, url).groups()
-        path = mobj.group('path')
+        if sub_domain not in ('money', 'edition'):
-        page_title = mobj.group('title')
+            sub_domain = 'edition'
-        info_url = 'http://edition.cnn.com/video/data/3.0/%s/index.xml' % path
+        config = self._CONFIG[sub_domain]
-        info = self._download_xml(info_url, page_title)
+        return self._extract_cvp_info(
-
+            config['data_src'] % path, page_title, {
-        formats = []
+                'default': {
-        rex = re.compile(r'''(?x)
+                    'media_src': config['media_src'],
-            (?P<width>[0-9]+)x(?P<height>[0-9]+)
+                }
-            (?:_(?P<bitrate>[0-9]+)k)?
+            })
        ''')
        for f in info.findall('files/file'):
            video_url = 'http://ht.cdn.turner.com/cnn/big%s' % (f.text.strip())
            fdct = {
                'format_id': f.attrib['bitrate'],
                'url': video_url,
            }
            mf = rex.match(f.attrib['bitrate'])
            if mf:
                fdct['width'] = int(mf.group('width'))
                fdct['height'] = int(mf.group('height'))
                fdct['tbr'] = int_or_none(mf.group('bitrate'))
            else:
                mf = rex.search(f.text)
                if mf:
                    fdct['width'] = int(mf.group('width'))
                    fdct['height'] = int(mf.group('height'))
                    fdct['tbr'] = int_or_none(mf.group('bitrate'))
                else:
                    mi = re.match(r'ios_(audio|[0-9]+)$', f.attrib['bitrate'])
                    if mi:
                        if mi.group(1) == 'audio':
                            fdct['vcodec'] = 'none'
                            fdct['ext'] = 'm4a'
                        else:
                            fdct['tbr'] = int(mi.group(1))
            formats.append(fdct)
        self._sort_formats(formats)
        thumbnails = [{
            'height': int(t.attrib['height']),
            'width': int(t.attrib['width']),
            'url': t.text,
        } for t in info.findall('images/image')]
        metas_el = info.find('metas')
        upload_date = (
            metas_el.attrib.get('version') if metas_el is not None else None)
        duration_el = info.find('length')
        duration = parse_duration(duration_el.text)
        return {
            'id': info.attrib['id'],
            'title': info.find('headline').text,
            'formats': formats,
            'thumbnails': thumbnails,
            'description': info.find('description').text,
            'duration': duration,
            'upload_date': upload_date,
        }
 class CNNBlogsIE(InfoExtractor):
@@ -132,6 +112,7 @@ class CNNBlogsIE(InfoExtractor):
            'description': 'Glenn Greenwald responds to comments made this week on Capitol Hill that journalists could be criminal accessories.',
            'upload_date': '20140209',
        },
        'expected_warnings': ['Failed to download m3u8 information'],
        'add_ie': ['CNN'],
    }
@@ -146,7 +127,7 @@ class CNNBlogsIE(InfoExtractor):
 class CNNArticleIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:edition|www)\.)?cnn\.com/(?!video/)'
+    _VALID_URL = r'https?://(?:(?:edition|www)\.)?cnn\.com/(?!videos?/)'
    _TEST = {
        'url': 'http://www.cnn.com/2014/12/21/politics/obama-north-koreas-hack-not-war-but-cyber-vandalism/',
        'md5': '689034c2a3d9c6dc4aa72d65a81efd01',
@@ -154,9 +135,10 @@ class CNNArticleIE(InfoExtractor):
            'id': 'bestoftv/2014/12/21/ip-north-korea-obama.cnn',
            'ext': 'mp4',
            'title': 'Obama: Cyberattack not an act of war',
-            'description': 'md5:51ce6750450603795cad0cdfbd7d05c5',
+            'description': 'md5:0a802a40d2376f60e6b04c8d5bcebc4b',
            'upload_date': '20141221',
        },
        'expected_warnings': ['Failed to download m3u8 information'],
        'add_ie': ['CNN'],
    }
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -662,6 +662,24 @@ class InfoExtractor(object):
        else:
            return res
    def _get_netrc_login_info(self, netrc_machine=None):
        username = None
        password = None
        netrc_machine = netrc_machine or self._NETRC_MACHINE
        if self._downloader.params.get('usenetrc', False):
            try:
                info = netrc.netrc().authenticators(netrc_machine)
                if info is not None:
                    username = info[0]
                    password = info[2]
                else:
                    raise netrc.NetrcParseError('No authenticators for %s' % netrc_machine)
            except (IOError, netrc.NetrcParseError) as err:
                self._downloader.report_warning('parsing .netrc: %s' % error_to_compat_str(err))
        return (username, password)
    def _get_login_info(self):
        """
        Get the login info as (username, password)
@@ -679,16 +697,8 @@ class InfoExtractor(object):
        if downloader_params.get('username') is not None:
            username = downloader_params['username']
            password = downloader_params['password']
-        elif downloader_params.get('usenetrc', False):
+        else:
-            try:
+            username, password = self._get_netrc_login_info()
                info = netrc.netrc().authenticators(self._NETRC_MACHINE)
                if info is not None:
                    username = info[0]
                    password = info[2]
                else:
                    raise netrc.NetrcParseError('No authenticators for %s' % self._NETRC_MACHINE)
            except (IOError, netrc.NetrcParseError) as err:
                self._downloader.report_warning('parsing .netrc: %s' % error_to_compat_str(err))
        return (username, password)
@@ -816,11 +826,14 @@ class InfoExtractor(object):
        json_ld = self._search_regex(
            r'(?s)<script[^>]+type=(["\'])application/ld\+json\1[^>]*>(?P<json_ld>.+?)</script>',
            html, 'JSON-LD', group='json_ld', **kwargs)
        default = kwargs.get('default', NO_DEFAULT)
        if not json_ld:
-            return {}
+            return default if default is not NO_DEFAULT else {}
-        return self._json_ld(
+        # JSON-LD may be malformed and thus `fatal` should be respected.
-            json_ld, video_id, fatal=kwargs.get('fatal', True),
+        # At the same time `default` may be passed that assumes `fatal=False`
-            expected_type=expected_type)
+        # for _search_regex. Let's simulate the same behavior here as well.
        fatal = kwargs.get('fatal', True) if default == NO_DEFAULT else False
        return self._json_ld(json_ld, video_id, fatal=fatal, expected_type=expected_type)
    def _json_ld(self, json_ld, video_id, fatal=True, expected_type=None):
        if isinstance(json_ld, compat_str):
@@ -1188,27 +1201,44 @@ class InfoExtractor(object):
                'protocol': entry_protocol,
                'preference': preference,
            }]
-        last_info = None
+        last_info = {}
-        last_media = None
+        last_media = {}
        for line in m3u8_doc.splitlines():
            if line.startswith('#EXT-X-STREAM-INF:'):
                last_info = parse_m3u8_attributes(line)
            elif line.startswith('#EXT-X-MEDIA:'):
-                last_media = parse_m3u8_attributes(line)
+                media = parse_m3u8_attributes(line)
                media_type = media.get('TYPE')
                if media_type in ('VIDEO', 'AUDIO'):
                    media_url = media.get('URI')
                    if media_url:
                        format_id = []
                        for v in (media.get('GROUP-ID'), media.get('NAME')):
                            if v:
                                format_id.append(v)
                        formats.append({
                            'format_id': '-'.join(format_id),
                            'url': format_url(media_url),
                            'language': media.get('LANGUAGE'),
                            'vcodec': 'none' if media_type == 'AUDIO' else None,
                            'ext': ext,
                            'protocol': entry_protocol,
                            'preference': preference,
                        })
                    else:
                        # When there is no URI in EXT-X-MEDIA let this tag's
                        # data be used by regular URI lines below
                        last_media = media
            elif line.startswith('#') or not line.strip():
                continue
            else:
-                if last_info is None:
+                tbr = int_or_none(last_info.get('AVERAGE-BANDWIDTH') or last_info.get('BANDWIDTH'), scale=1000)
                    formats.append({'url': format_url(line)})
                    continue
                tbr = int_or_none(last_info.get('BANDWIDTH'), scale=1000)
                format_id = []
                if m3u8_id:
                    format_id.append(m3u8_id)
                last_media_name = last_media.get('NAME') if last_media and last_media.get('TYPE') not in ('SUBTITLES', 'CLOSED-CAPTIONS') else None
                # Despite specification does not mention NAME attribute for
                # EXT-X-STREAM-INF it still sometimes may be present
-                stream_name = last_info.get('NAME') or last_media_name
+                stream_name = last_info.get('NAME') or last_media.get('NAME')
                # Bandwidth of live streams may differ over time thus making
                # format_id unpredictable. So it's better to keep provided
                # format_id intact.
@@ -1239,11 +1269,9 @@ class InfoExtractor(object):
                        'abr': abr,
                    })
                f.update(parse_codecs(last_info.get('CODECS')))
                if last_media is not None:
                    f['m3u8_media'] = last_media
                    last_media = None
                formats.append(f)
                last_info = {}
                last_media = {}
        return formats
    @staticmethod
@@ -1682,7 +1710,7 @@ class InfoExtractor(object):
                        self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
        return formats
-    def _parse_html5_media_entries(self, base_url, webpage):
+    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8'):
        def absolute_url(video_url):
            return compat_urlparse.urljoin(base_url, video_url)
@@ -1697,6 +1725,21 @@ class InfoExtractor(object):
                return f
            return {}
        def _media_formats(src, cur_media_type):
            full_url = absolute_url(src)
            if determine_ext(full_url) == 'm3u8':
                is_plain_url = False
                formats = self._extract_m3u8_formats(
                    full_url, video_id, ext='mp4',
                    entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id)
            else:
                is_plain_url = True
                formats = [{
                    'url': full_url,
                    'vcodec': 'none' if cur_media_type == 'audio' else None,
                }]
            return is_plain_url, formats
        entries = []
        for media_tag, media_type, media_content in re.findall(r'(?s)(<(?P<tag>video|audio)[^>]*>)(.*?)</(?P=tag)>', webpage):
            media_info = {
@@ -1706,10 +1749,8 @@ class InfoExtractor(object):
            media_attributes = extract_attributes(media_tag)
            src = media_attributes.get('src')
            if src:
-                media_info['formats'].append({
+                _, formats = _media_formats(src)
-                    'url': absolute_url(src),
+                media_info['formats'].extend(formats)
                    'vcodec': 'none' if media_type == 'audio' else None,
                })
            media_info['thumbnail'] = media_attributes.get('poster')
            if media_content:
                for source_tag in re.findall(r'<source[^>]+>', media_content):
@@ -1717,12 +1758,13 @@ class InfoExtractor(object):
                    src = source_attributes.get('src')
                    if not src:
                        continue
-                    f = parse_content_type(source_attributes.get('type'))
+                    is_plain_url, formats = _media_formats(src, media_type)
-                    f.update({
+                    if is_plain_url:
-                        'url': absolute_url(src),
+                        f = parse_content_type(source_attributes.get('type'))
-                        'vcodec': 'none' if media_type == 'audio' else None,
+                        f.update(formats[0])
-                    })
+                        media_info['formats'].append(f)
-                    media_info['formats'].append(f)
+                    else:
                        media_info['formats'].extend(formats)
                for track_tag in re.findall(r'<track[^>]+>', media_content):
                    track_attributes = extract_attributes(track_tag)
                    kind = track_attributes.get('kind')
@@ -1738,6 +1780,18 @@ class InfoExtractor(object):
                entries.append(media_info)
        return entries
    def _extract_akamai_formats(self, manifest_url, video_id):
        formats = []
        f4m_url = re.sub(r'(https?://.+?)/i/', r'\1/z/', manifest_url).replace('/master.m3u8', '/manifest.f4m')
        formats.extend(self._extract_f4m_formats(
            update_url_query(f4m_url, {'hdcore': '3.7.0'}),
            video_id, f4m_id='hds', fatal=False))
        m3u8_url = re.sub(r'(https?://.+?)/z/', r'\1/i/', manifest_url).replace('/manifest.f4m', '/master.m3u8')
        formats.extend(self._extract_m3u8_formats(
            m3u8_url, video_id, 'mp4', 'm3u8_native',
            m3u8_id='hls', fatal=False))
        return formats
    def _live_title(self, name):
        """ Generate the title for a live video """
        now = datetime.datetime.now()
--- a/youtube_dl/extractor/condenast.py
+++ b/youtube_dl/extractor/condenast.py
@@ -143,7 +143,8 @@ class CondeNastIE(InfoExtractor):
            })
        self._sort_formats(formats)
-        info = self._search_json_ld(webpage, video_id) if url_type != 'embed' else {}
+        info = self._search_json_ld(
            webpage, video_id, fatal=False) if url_type != 'embed' else {}
        info.update({
            'id': video_id,
            'formats': formats,
--- a/youtube_dl/extractor/crackle.py
+++ b/youtube_dl/extractor/crackle.py
@@ -1,5 +1,5 @@
 # coding: utf-8
-from __future__ import unicode_literals
+from __future__ import unicode_literals, division
 from .common import InfoExtractor
 from ..utils import int_or_none
@@ -8,12 +8,22 @@ from ..utils import int_or_none
 class CrackleIE(InfoExtractor):
    _VALID_URL = r'(?:crackle:|https?://(?:www\.)?crackle\.com/(?:playlist/\d+/|(?:[^/]+/)+))(?P<id>\d+)'
    _TEST = {
-        'url': 'http://www.crackle.com/the-art-of-more/2496419',
+        'url': 'http://www.crackle.com/comedians-in-cars-getting-coffee/2498934',
        'info_dict': {
-            'id': '2496419',
+            'id': '2498934',
            'ext': 'mp4',
-            'title': 'Heavy Lies the Head',
+            'title': 'Everybody Respects A Bloody Nose',
-            'description': 'md5:bb56aa0708fe7b9a4861535f15c3abca',
+            'description': 'Jerry is kaffeeklatsching in L.A. with funnyman J.B. Smoove (Saturday Night Live, Real Husbands of Hollywood). They’re headed for brew at 10 Speed Coffee in a 1964 Studebaker Avanti.',
            'thumbnail': 're:^https?://.*\.jpg',
            'duration': 906,
            'series': 'Comedians In Cars Getting Coffee',
            'season_number': 8,
            'episode_number': 4,
            'subtitles': {
                'en-US': [{
                    'ext': 'ttml',
                }]
            },
        },
        'params': {
            # m3u8 download
@@ -21,12 +31,8 @@ class CrackleIE(InfoExtractor):
        }
    }
    # extracted from http://legacyweb-us.crackle.com/flash/QueryReferrer.ashx
    _SUBTITLE_SERVER = 'http://web-us-az.crackle.com'
    _UPLYNK_OWNER_ID = 'e8773f7770a44dbd886eee4fca16a66b'
    _THUMBNAIL_TEMPLATE = 'http://images-us-am.crackle.com/%stnl_1920x1080.jpg?ts=20140107233116?c=635333335057637614'
    # extracted from http://legacyweb-us.crackle.com/flash/ReferrerRedirect.ashx
    _THUMBNAIL_TEMPLATE = 'http://images-us-am.crackle.com/%stnl_1920x1080.jpg?ts=20140107233116?c=635333335057637614'
    _MEDIA_FILE_SLOTS = {
        'c544.flv': {
            'width': 544,
@@ -48,16 +54,21 @@ class CrackleIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
        config_doc = self._download_xml(
            'http://legacyweb-us.crackle.com/flash/QueryReferrer.ashx?site=16',
            video_id, 'Downloading config')
        item = self._download_xml(
            'http://legacyweb-us.crackle.com/app/revamp/vidwallcache.aspx?flags=-1&fm=%s' % video_id,
            video_id).find('i')
        title = item.attrib['t']
        thumbnail = None
        subtitles = {}
        formats = self._extract_m3u8_formats(
-            'http://content.uplynk.com/ext/%s/%s.m3u8' % (self._UPLYNK_OWNER_ID, video_id),
+            'http://content.uplynk.com/ext/%s/%s.m3u8' % (config_doc.attrib['strUplynkOwnerId'], video_id),
            video_id, 'mp4', m3u8_id='hls', fatal=None)
        thumbnail = None
        path = item.attrib.get('p')
        if path:
            thumbnail = self._THUMBNAIL_TEMPLATE % path
@@ -76,7 +87,7 @@ class CrackleIE(InfoExtractor):
                    if locale not in subtitles:
                        subtitles[locale] = []
                    subtitles[locale] = [{
-                        'url': '%s/%s%s_%s.xml' % (self._SUBTITLE_SERVER, path, locale, v),
+                        'url': '%s/%s%s_%s.xml' % (config_doc.attrib['strSubtitleServer'], path, locale, v),
                        'ext': 'ttml',
                    }]
        self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
@@ -85,7 +96,7 @@ class CrackleIE(InfoExtractor):
            'id': video_id,
            'title': title,
            'description': item.attrib.get('d'),
-            'duration': int(item.attrib.get('r'), 16) if item.attrib.get('r') else None,
+            'duration': int(item.attrib.get('r'), 16) / 1000 if item.attrib.get('r') else None,
            'series': item.attrib.get('sn'),
            'season_number': int_or_none(item.attrib.get('se')),
            'episode_number': int_or_none(item.attrib.get('ep')),
--- a/youtube_dl/extractor/crunchyroll.py
+++ b/youtube_dl/extractor/crunchyroll.py
@@ -114,6 +114,21 @@ class CrunchyrollIE(CrunchyrollBaseIE):
            # rtmp
            'skip_download': True,
        },
    }, {
        'url': 'http://www.crunchyroll.com/rezero-starting-life-in-another-world-/episode-5-the-morning-of-our-promise-is-still-distant-702409',
        'info_dict': {
            'id': '702409',
            'ext': 'mp4',
            'title': 'Re:ZERO -Starting Life in Another World- Episode 5 – The Morning of Our Promise Is Still Distant',
            'description': 'md5:97664de1ab24bbf77a9c01918cb7dca9',
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': 'TV TOKYO',
            'upload_date': '20160508',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://www.crunchyroll.fr/girl-friend-beta/episode-11-goodbye-la-mode-661697',
        'only_matching': True,
@@ -336,9 +351,18 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            if video_encode_id in video_encode_ids:
                continue
            video_encode_ids.append(video_encode_id)
            video_file = xpath_text(stream_info, './file')
            if not video_file:
                continue
            if video_file.startswith('http'):
                formats.extend(self._extract_m3u8_formats(
                    video_file, video_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
                continue
            video_url = xpath_text(stream_info, './host')
-            video_play_path = xpath_text(stream_info, './file')
+            if not video_url:
            if not video_url or not video_play_path:
                continue
            metadata = stream_info.find('./metadata')
            format_info = {
@@ -353,7 +377,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                parsed_video_url = compat_urlparse.urlparse(video_url)
                direct_video_url = compat_urlparse.urlunparse(parsed_video_url._replace(
                    netloc='v.lvlt.crcdn.net',
-                    path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_play_path.split(':')[-1])))
+                    path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_file.split(':')[-1])))
                if self._is_valid_url(direct_video_url, video_id, video_format):
                    format_info.update({
                        'url': direct_video_url,
@@ -363,7 +387,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            format_info.update({
                'url': video_url,
-                'play_path': video_play_path,
+                'play_path': video_file,
                'ext': 'flv',
            })
            formats.append(format_info)
--- a/youtube_dl/extractor/ctsnews.py
+++ b/youtube_dl/extractor/ctsnews.py
@@ -1,13 +1,12 @@
-# -*- coding: utf-8 -*-
+# coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
-from ..utils import parse_iso8601, ExtractorError
+from ..utils import unified_timestamp
 class CtsNewsIE(InfoExtractor):
    IE_DESC = '華視新聞'
    # https connection failed (Connection reset)
    _VALID_URL = r'https?://news\.cts\.com\.tw/[a-z]+/[a-z]+/\d+/(?P<id>\d+)\.html'
    _TESTS = [{
        'url': 'http://news.cts.com.tw/cts/international/201501/201501291578109.html',
@@ -16,7 +15,7 @@ class CtsNewsIE(InfoExtractor):
            'id': '201501291578109',
            'ext': 'mp4',
            'title': '以色列.真主黨交火 3人死亡',
-            'description': 'md5:95e9b295c898b7ff294f09d450178d7d',
+            'description': '以色列和黎巴嫩真主黨，爆發五年最嚴重衝突，雙方砲轟交火，兩名以軍死亡，還有一名西班牙籍的聯合國維和人...',
            'timestamp': 1422528540,
            'upload_date': '20150129',
        }
@@ -28,7 +27,7 @@ class CtsNewsIE(InfoExtractor):
            'id': '201309031304098',
            'ext': 'mp4',
            'title': '韓國31歲童顏男 貌如十多歲小孩',
-            'description': 'md5:f183feeba3752b683827aab71adad584',
+            'description': '越有年紀的人，越希望看起來年輕一點，而南韓卻有一位31歲的男子，看起來像是11、12歲的小孩，身...',
            'thumbnail': 're:^https?://.*\.jpg$',
            'timestamp': 1378205880,
            'upload_date': '20130903',
@@ -36,8 +35,7 @@ class CtsNewsIE(InfoExtractor):
    }, {
        # With Youtube embedded video
        'url': 'http://news.cts.com.tw/cts/money/201501/201501291578003.html',
-        'md5': '1d842c771dc94c8c3bca5af2cc1db9c5',
+        'md5': 'e4726b2ccd70ba2c319865e28f0a91d1',
        'add_ie': ['Youtube'],
        'info_dict': {
            'id': 'OVbfO7d0_hQ',
            'ext': 'mp4',
@@ -47,42 +45,37 @@ class CtsNewsIE(InfoExtractor):
            'upload_date': '20150128',
            'uploader_id': 'TBSCTS',
            'uploader': '中華電視公司',
-        }
+        },
        'add_ie': ['Youtube'],
    }]
    def _real_extract(self, url):
        news_id = self._match_id(url)
        page = self._download_webpage(url, news_id)
-        if self._search_regex(r'(CTSPlayer2)', page, 'CTSPlayer2 identifier', default=None):
+        news_id = self._hidden_inputs(page).get('get_id')
-            feed_url = self._html_search_regex(
+
-                r'(http://news\.cts\.com\.tw/action/mp4feed\.php\?news_id=\d+)',
+        if news_id:
-                page, 'feed url')
+            mp4_feed = self._download_json(
-            video_url = self._download_webpage(
+                'http://news.cts.com.tw/action/test_mp4feed.php',
-                feed_url, news_id, note='Fetching feed')
+                news_id, note='Fetching feed', query={'news_id': news_id})
            video_url = mp4_feed['source_url']
        else:
            self.to_screen('Not CTSPlayer video, trying Youtube...')
            youtube_url = self._search_regex(
-                r'src="(//www\.youtube\.com/embed/[^"]+)"', page, 'youtube url',
+                r'src="(//www\.youtube\.com/embed/[^"]+)"', page, 'youtube url')
                default=None)
            if not youtube_url:
                raise ExtractorError('The news includes no videos!', expected=True)
-            return {
+            return self.url_result(youtube_url, ie='Youtube')
                '_type': 'url',
                'url': youtube_url,
                'ie_key': 'Youtube',
            }
        description = self._html_search_meta('description', page)
-        title = self._html_search_meta('title', page)
+        title = self._html_search_meta('title', page, fatal=True)
        thumbnail = self._html_search_meta('image', page)
        datetime_str = self._html_search_regex(
-            r'(\d{4}/\d{2}/\d{2} \d{2}:\d{2})', page, 'date and time')
+            r'(\d{4}/\d{2}/\d{2} \d{2}:\d{2})', page, 'date and time', fatal=False)
-        # Transform into ISO 8601 format with timezone info
+        timestamp = None
-        datetime_str = datetime_str.replace('/', '-') + ':00+0800'
+        if datetime_str:
-        timestamp = parse_iso8601(datetime_str, delimiter=' ')
+            timestamp = unified_timestamp(datetime_str) - 8 * 3600
        return {
            'id': news_id,
--- a/youtube_dl/extractor/ctv.py
+++ b/youtube_dl/extractor/ctv.py
@@ -1,11 +1,13 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 class CTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?ctv\.ca/video/player\?vid=(?P<id>[0-9.]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?P<domain>ctv|tsn|bnn|thecomedynetwork)\.ca/.*?(?:\bvid=|-vid|~|%7E)(?P<id>[0-9.]+)'
    _TESTS = [{
        'url': 'http://www.ctv.ca/video/player?vid=706966',
        'md5': 'ff2ebbeae0aa2dcc32a830c3fd69b7b0',
@@ -18,13 +20,27 @@ class CTVIE(InfoExtractor):
            'timestamp': 1442624700,
        },
        'expected_warnings': ['HTTP Error 404'],
    }, {
        'url': 'http://www.thecomedynetwork.ca/video/player?vid=923582',
        'only_matching': True,
    }, {
        'url': 'http://www.tsn.ca/video/expectations-high-for-milos-raonic-at-us-open~939549',
        'only_matching': True,
    }, {
        'url': 'http://www.bnn.ca/video/berman-s-call-part-two-viewer-questions~939654',
        'only_matching': True,
    }, {
        'url': 'http://www.ctv.ca/YourMorning/Video/S1E6-Monday-August-29-2016-vid938009',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        domain, video_id = re.match(self._VALID_URL, url).groups()
        if domain == 'thecomedynetwork':
            domain = 'comedy'
        return {
            '_type': 'url_transparent',
            'id': video_id,
-            'url': '9c9media:ctv_web:%s' % video_id,
+            'url': '9c9media:%s_web:%s' % (domain, video_id),
            'ie_key': 'NineCNineMedia',
        }
--- a/youtube_dl/extractor/cultureunplugged.py
+++ b/youtube_dl/extractor/cultureunplugged.py
@@ -1,9 +1,13 @@
 from __future__ import unicode_literals
 import re
 import time
 from .common import InfoExtractor
-from ..utils import int_or_none
+from ..utils import (
    int_or_none,
    HEADRequest,
 )
 class CultureUnpluggedIE(InfoExtractor):
@@ -32,6 +36,9 @@ class CultureUnpluggedIE(InfoExtractor):
        video_id = mobj.group('id')
        display_id = mobj.group('display_id') or video_id
        # request setClientTimezone.php to get PHPSESSID cookie which is need to get valid json data in the next request
        self._request_webpage(HEADRequest(
            'http://www.cultureunplugged.com/setClientTimezone.php?timeOffset=%d' % -(time.timezone / 3600)), display_id)
        movie_data = self._download_json(
            'http://www.cultureunplugged.com/movie-data/cu-%s.json' % video_id, display_id)
--- a/youtube_dl/extractor/curiositystream.py
+++ b/youtube_dl/extractor/curiositystream.py
@@ -0,0 +1,120 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    urlencode_postdata,
    compat_str,
    ExtractorError,
 )
 class CuriosityStreamBaseIE(InfoExtractor):
    _NETRC_MACHINE = 'curiositystream'
    _auth_token = None
    _API_BASE_URL = 'https://api.curiositystream.com/v1/'
    def _handle_errors(self, result):
        error = result.get('error', {}).get('message')
        if error:
            if isinstance(error, dict):
                error = ', '.join(error.values())
            raise ExtractorError(
                '%s said: %s' % (self.IE_NAME, error), expected=True)
    def _call_api(self, path, video_id):
        headers = {}
        if self._auth_token:
            headers['X-Auth-Token'] = self._auth_token
        result = self._download_json(
            self._API_BASE_URL + path, video_id, headers=headers)
        self._handle_errors(result)
        return result['data']
    def _real_initialize(self):
        (email, password) = self._get_login_info()
        if email is None:
            return
        result = self._download_json(
            self._API_BASE_URL + 'login', None, data=urlencode_postdata({
                'email': email,
                'password': password,
            }))
        self._handle_errors(result)
        self._auth_token = result['message']['auth_token']
    def _extract_media_info(self, media):
        video_id = compat_str(media['id'])
        limelight_media_id = media['limelight_media_id']
        title = media['title']
        subtitles = {}
        for closed_caption in media.get('closed_captions', []):
            sub_url = closed_caption.get('file')
            if not sub_url:
                continue
            lang = closed_caption.get('code') or closed_caption.get('language') or 'en'
            subtitles.setdefault(lang, []).append({
                'url': sub_url,
            })
        return {
            '_type': 'url_transparent',
            'id': video_id,
            'url': 'limelight:media:' + limelight_media_id,
            'title': title,
            'description': media.get('description'),
            'thumbnail': media.get('image_large') or media.get('image_medium') or media.get('image_small'),
            'duration': int_or_none(media.get('duration')),
            'tags': media.get('tags'),
            'subtitles': subtitles,
            'ie_key': 'LimelightMedia',
        }
 class CuriosityStreamIE(CuriosityStreamBaseIE):
    IE_NAME = 'curiositystream'
    _VALID_URL = r'https?://app\.curiositystream\.com/video/(?P<id>\d+)'
    _TEST = {
        'url': 'https://app.curiositystream.com/video/2',
        'md5': 'a0074c190e6cddaf86900b28d3e9ee7a',
        'info_dict': {
            'id': '2',
            'ext': 'mp4',
            'title': 'How Did You Develop The Internet?',
            'description': 'Vint Cerf, Google\'s Chief Internet Evangelist, describes how he and Bob Kahn created the internet.',
            'timestamp': 1448388615,
            'upload_date': '20151124',
        }
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        media = self._call_api('media/' + video_id, video_id)
        return self._extract_media_info(media)
 class CuriosityStreamCollectionIE(CuriosityStreamBaseIE):
    IE_NAME = 'curiositystream:collection'
    _VALID_URL = r'https?://app\.curiositystream\.com/collection/(?P<id>\d+)'
    _TEST = {
        'url': 'https://app.curiositystream.com/collection/2',
        'info_dict': {
            'id': '2',
            'title': 'Curious Minds: The Internet',
            'description': 'How is the internet shaping our lives in the 21st Century?',
        },
        'playlist_mincount': 17,
    }
    def _real_extract(self, url):
        collection_id = self._match_id(url)
        collection = self._call_api(
            'collections/' + collection_id, collection_id)
        entries = []
        for media in collection.get('media', []):
            entries.append(self._extract_media_info(media))
        return self.playlist_result(
            entries, collection_id,
            collection.get('title'), collection.get('description'))
--- a/youtube_dl/extractor/cwtv.py
+++ b/youtube_dl/extractor/cwtv.py
@@ -28,7 +28,8 @@ class CWTVIE(InfoExtractor):
        'params': {
            # m3u8 download
            'skip_download': True,
-        }
+        },
        'skip': 'redirect to http://cwtv.com/shows/arrow/',
    }, {
        'url': 'http://www.cwseed.com/shows/whose-line-is-it-anyway/jeff-davis-4/?play=24282b12-ead2-42f2-95ad-26770c2c6088',
        'info_dict': {
@@ -44,10 +45,6 @@ class CWTVIE(InfoExtractor):
            'upload_date': '20151006',
            'timestamp': 1444107300,
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        }
    }, {
        'url': 'http://cwtv.com/thecw/chroniclesofcisco/?play=8adebe35-f447-465f-ab52-e863506ff6d6',
        'only_matching': True,
@@ -61,11 +58,30 @@ class CWTVIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        video_data = self._download_json(
+        video_data = None
-            'http://metaframe.digitalsmiths.tv/v2/CWtv/assets/%s/partner/132?format=json' % video_id, video_id)
+        formats = []
-
+        for partner in (154, 213):
-        formats = self._extract_m3u8_formats(
+            vdata = self._download_json(
-            video_data['videos']['variantplaylist']['uri'], video_id, 'mp4')
+                'http://metaframe.digitalsmiths.tv/v2/CWtv/assets/%s/partner/%d?format=json' % (video_id, partner), video_id, fatal=False)
            if not vdata:
                continue
            video_data = vdata
            for quality, quality_data in vdata.get('videos', {}).items():
                quality_url = quality_data.get('uri')
                if not quality_url:
                    continue
                if quality == 'variantplaylist':
                    formats.extend(self._extract_m3u8_formats(
                        quality_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
                else:
                    tbr = int_or_none(quality_data.get('bitrate'))
                    format_id = 'http' + ('-%d' % tbr if tbr else '')
                    if self._is_valid_url(quality_url, video_id, format_id):
                        formats.append({
                            'format_id': format_id,
                            'url': quality_url,
                            'tbr': tbr,
                        })
        self._sort_formats(formats)
        thumbnails = [{
--- a/youtube_dl/extractor/dbtv.py
+++ b/youtube_dl/extractor/dbtv.py
@@ -38,6 +38,12 @@ class DBTVIE(InfoExtractor):
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [url for _, url in re.findall(
            r'<iframe[^>]+src=(["\'])((?:https?:)?//(?:www\.)?dbtv\.no/(?:lazy)?player/\d+.*?)\1',
            webpage)]
    def _real_extract(self, url):
        video_id, display_id = re.match(self._VALID_URL, url).groups()
--- a/youtube_dl/extractor/discoverygo.py
+++ b/youtube_dl/extractor/discoverygo.py
@@ -7,11 +7,22 @@ from ..utils import (
    int_or_none,
    parse_age_limit,
    unescapeHTML,
    ExtractorError,
 )
 class DiscoveryGoIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?discoverygo\.com/(?:[^/]+/)*(?P<id>[^/?#&]+)'
+    _VALID_URL = r'''(?x)https?://(?:www\.)?(?:
            discovery|
            investigationdiscovery|
            discoverylife|
            animalplanet|
            ahctv|
            destinationamerica|
            sciencechannel|
            tlc|
            velocitychannel
        )go\.com/(?:[^/]+/)*(?P<id>[^/?#&]+)'''
    _TEST = {
        'url': 'https://www.discoverygo.com/love-at-first-kiss/kiss-first-ask-questions-later/',
        'info_dict': {
@@ -43,7 +54,14 @@ class DiscoveryGoIE(InfoExtractor):
        title = video['name']
-        stream = video['stream']
+        stream = video.get('stream')
        if not stream:
            if video.get('authenticated') is True:
                raise ExtractorError(
                    'This video is only available via cable service provider subscription that'
                    ' is not currently supported. You may want to use --cookies.', expected=True)
            else:
                raise ExtractorError('Unable to find stream')
        STREAM_URL_SUFFIX = 'streamUrl'
        formats = []
        for stream_kind in ('', 'hds'):
--- a/youtube_dl/extractor/dotsub.py
+++ b/youtube_dl/extractor/dotsub.py
@@ -10,18 +10,18 @@ from ..utils import (
 class DotsubIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?dotsub\.com/view/(?P<id>[^/]+)'
    _TEST = {
-        'url': 'http://dotsub.com/view/aed3b8b2-1889-4df5-ae63-ad85f5572f27',
+        'url': 'https://dotsub.com/view/9c63db2a-fa95-4838-8e6e-13deafe47f09',
-        'md5': '0914d4d69605090f623b7ac329fea66e',
+        'md5': '21c7ff600f545358134fea762a6d42b6',
        'info_dict': {
-            'id': 'aed3b8b2-1889-4df5-ae63-ad85f5572f27',
+            'id': '9c63db2a-fa95-4838-8e6e-13deafe47f09',
            'ext': 'flv',
-            'title': 'Pyramids of Waste (2010), AKA The Lightbulb Conspiracy - Planned obsolescence documentary',
+            'title': 'MOTIVATION - "It\'s Possible" Best Inspirational Video Ever',
-            'description': 'md5:699a0f7f50aeec6042cb3b1db2d0d074',
+            'description': 'md5:41af1e273edbbdfe4e216a78b9d34ac6',
-            'thumbnail': 're:^https?://dotsub.com/media/aed3b8b2-1889-4df5-ae63-ad85f5572f27/p',
+            'thumbnail': 're:^https?://dotsub.com/media/9c63db2a-fa95-4838-8e6e-13deafe47f09/p',
-            'duration': 3169,
+            'duration': 198,
-            'uploader': '4v4l0n42',
+            'uploader': 'liuxt',
-            'timestamp': 1292248482.625,
+            'timestamp': 1385778501.104,
-            'upload_date': '20101213',
+            'upload_date': '20131130',
            'view_count': int,
        }
    }
--- a/youtube_dl/extractor/douyutv.py
+++ b/youtube_dl/extractor/douyutv.py
@@ -3,9 +3,17 @@ from __future__ import unicode_literals
 import hashlib
 import time
 import uuid
 from .common import InfoExtractor
-from ..utils import (ExtractorError, unescapeHTML)
+from ..compat import (
-from ..compat import (compat_str, compat_basestring)
+    compat_str,
    compat_urllib_parse_urlencode,
 )
 from ..utils import (
    ExtractorError,
    unescapeHTML,
 )
 class DouyuTVIE(InfoExtractor):
@@ -21,7 +29,6 @@ class DouyuTVIE(InfoExtractor):
            'description': 're:.*m7show@163\.com.*',
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': '7师傅',
            'uploader_id': '431925',
            'is_live': True,
        },
        'params': {
@@ -37,7 +44,6 @@ class DouyuTVIE(InfoExtractor):
            'description': 'md5:746a2f7a253966a06755a912f0acc0d2',
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': 'douyu小漠',
            'uploader_id': '3769985',
            'is_live': True,
        },
        'params': {
@@ -54,7 +60,6 @@ class DouyuTVIE(InfoExtractor):
            'description': 're:.*m7show@163\.com.*',
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': '7师傅',
            'uploader_id': '431925',
            'is_live': True,
        },
        'params': {
@@ -65,6 +70,10 @@ class DouyuTVIE(InfoExtractor):
        'only_matching': True,
    }]
    # Decompile core.swf in webpage by ffdec "Search SWFs in memory". core.swf
    # is encrypted originally, but ffdec can dump memory to get the decrypted one.
    _API_KEY = 'A12Svb&%1UUmf@hC'
    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -75,74 +84,56 @@ class DouyuTVIE(InfoExtractor):
            room_id = self._html_search_regex(
                r'"room_id"\s*:\s*(\d+),', page, 'room id')
-        config = None
+        room = self._download_json(
-        # Douyu API sometimes returns error "Unable to load the requested class: eticket_redis_cache"
+            'http://m.douyu.com/html5/live?roomId=%s' % room_id, video_id,
-        # Retry with different parameters - same parameters cause same errors
+            note='Downloading room info')['data']
        for i in range(5):
            prefix = 'room/%s?aid=android&client_sys=android&time=%d' % (
                room_id, int(time.time()))
            auth = hashlib.md5((prefix + '1231').encode('ascii')).hexdigest()
            config_page = self._download_webpage(
                'http://www.douyutv.com/api/v1/%s&auth=%s' % (prefix, auth),
                video_id)
            try:
                config = self._parse_json(config_page, video_id, fatal=False)
            except ExtractorError:
                # Wait some time before retrying to get a different time() value
                self._sleep(1, video_id, msg_template='%(video_id)s: Error occurs. '
                                                      'Waiting for %(timeout)s seconds before retrying')
                continue
            else:
                break
        if config is None:
            raise ExtractorError('Unable to fetch API result')
        data = config['data']
        error_code = config.get('error', 0)
        if error_code is not 0:
            error_desc = 'Server reported error %i' % error_code
            if isinstance(data, (compat_str, compat_basestring)):
                error_desc += ': ' + data
            raise ExtractorError(error_desc, expected=True)
        show_status = data.get('show_status')
        # 1 = live, 2 = offline
-        if show_status == '2':
+        if room.get('show_status') == '2':
            raise ExtractorError('Live stream is offline', expected=True)
        tt = compat_str(int(time.time() / 60))
        did = uuid.uuid4().hex.upper()
        sign_content = ''.join((room_id, did, self._API_KEY, tt))
        sign = hashlib.md5((sign_content).encode('utf-8')).hexdigest()
        flv_data = compat_urllib_parse_urlencode({
            'cdn': 'ws',
            'rate': '0',
            'tt': tt,
            'did': did,
            'sign': sign,
        })
        video_info = self._download_json(
            'http://www.douyu.com/lapi/live/getPlay/%s' % room_id, video_id,
            data=flv_data, note='Downloading video info',
            headers={'Content-Type': 'application/x-www-form-urlencoded'})
        error_code = video_info.get('error', 0)
        if error_code is not 0:
            raise ExtractorError(
-                'Live stream is offline', expected=True)
+                '%s reported error %i' % (self.IE_NAME, error_code),
                expected=True)
-        base_url = data['rtmp_url']
+        base_url = video_info['data']['rtmp_url']
-        live_path = data['rtmp_live']
+        live_path = video_info['data']['rtmp_live']
-        title = self._live_title(unescapeHTML(data['room_name']))
+        video_url = '%s/%s' % (base_url, live_path)
        description = data.get('show_details')
        thumbnail = data.get('room_src')
-        uploader = data.get('nickname')
+        title = self._live_title(unescapeHTML(room['room_name']))
-        uploader_id = data.get('owner_uid')
+        description = room.get('notice')
-
+        thumbnail = room.get('room_src')
-        multi_formats = data.get('rtmp_multi_bitrate')
+        uploader = room.get('nickname')
        if not isinstance(multi_formats, dict):
            multi_formats = {}
        multi_formats['live'] = live_path
        formats = [{
            'url': '%s/%s' % (base_url, format_path),
            'format_id': format_id,
            'preference': 1 if format_id == 'live' else 0,
        } for format_id, format_path in multi_formats.items()]
        self._sort_formats(formats)
        return {
            'id': room_id,
            'display_id': video_id,
            'url': video_url,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'uploader': uploader,
            'uploader_id': uploader_id,
            'formats': formats,
            'is_live': True,
        }
--- a/youtube_dl/extractor/drtuber.py
+++ b/youtube_dl/extractor/drtuber.py
@@ -3,7 +3,10 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..utils import str_to_int
+from ..utils import (
    NO_DEFAULT,
    str_to_int,
 )
 class DrTuberIE(InfoExtractor):
@@ -17,7 +20,6 @@ class DrTuberIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'hot perky blonde naked golf',
            'like_count': int,
            'dislike_count': int,
            'comment_count': int,
            'categories': ['Babe', 'Blonde', 'Erotic', 'Outdoor', 'Softcore', 'Solo'],
            'thumbnail': 're:https?://.*\.jpg$',
@@ -36,25 +38,29 @@ class DrTuberIE(InfoExtractor):
            r'<source src="([^"]+)"', webpage, 'video URL')
        title = self._html_search_regex(
-            [r'<p[^>]+class="title_substrate">([^<]+)</p>', r'<title>([^<]+) - \d+'],
+            (r'class="title_watch"[^>]*><p>([^<]+)<',
             r'<p[^>]+class="title_substrate">([^<]+)</p>',
             r'<title>([^<]+) - \d+'),
            webpage, 'title')
        thumbnail = self._html_search_regex(
            r'poster="([^"]+)"',
            webpage, 'thumbnail', fatal=False)
-        def extract_count(id_, name):
+        def extract_count(id_, name, default=NO_DEFAULT):
            return str_to_int(self._html_search_regex(
                r'<span[^>]+(?:class|id)="%s"[^>]*>([\d,\.]+)</span>' % id_,
-                webpage, '%s count' % name, fatal=False))
+                webpage, '%s count' % name, default=default, fatal=False))
        like_count = extract_count('rate_likes', 'like')
-        dislike_count = extract_count('rate_dislikes', 'dislike')
+        dislike_count = extract_count('rate_dislikes', 'dislike', default=None)
        comment_count = extract_count('comments_count', 'comment')
        cats_str = self._search_regex(
-            r'<div[^>]+class="categories_list">(.+?)</div>', webpage, 'categories', fatal=False)
+            r'<div[^>]+class="categories_list">(.+?)</div>',
-        categories = [] if not cats_str else re.findall(r'<a title="([^"]+)"', cats_str)
+            webpage, 'categories', fatal=False)
        categories = [] if not cats_str else re.findall(
            r'<a title="([^"]+)"', cats_str)
        return {
            'id': video_id,
--- a/youtube_dl/extractor/drtv.py
+++ b/youtube_dl/extractor/drtv.py
@@ -4,26 +4,45 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
    float_or_none,
    mimetype2ext,
    parse_iso8601,
    remove_end,
 )
 class DRTVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?dr\.dk/tv/se/(?:[^/]+/)*(?P<id>[\da-z-]+)(?:[/#?]|$)'
+    _VALID_URL = r'https?://(?:www\.)?dr\.dk/(?:tv/se|nyheder)/(?:[^/]+/)*(?P<id>[\da-z-]+)(?:[/#?]|$)'
-    _TEST = {
+    _TESTS = [{
-        'url': 'https://www.dr.dk/tv/se/boern/ultra/panisk-paske/panisk-paske-5',
+        'url': 'https://www.dr.dk/tv/se/boern/ultra/klassen-ultra/klassen-darlig-taber-10',
-        'md5': 'dc515a9ab50577fa14cc4e4b0265168f',
+        'md5': '25e659cccc9a2ed956110a299fdf5983',
        'info_dict': {
-            'id': 'panisk-paske-5',
+            'id': 'klassen-darlig-taber-10',
            'ext': 'mp4',
-            'title': 'Panisk Påske (5)',
+            'title': 'Klassen - Dårlig taber (10)',
-            'description': 'md5:ca14173c5ab24cd26b0fcc074dff391c',
+            'description': 'md5:815fe1b7fa656ed80580f31e8b3c79aa',
-            'timestamp': 1426984612,
+            'timestamp': 1471991907,
-            'upload_date': '20150322',
+            'upload_date': '20160823',
-            'duration': 1455,
+            'duration': 606.84,
        },
-    }
+        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.dr.dk/nyheder/indland/live-christianias-rydning-af-pusher-street-er-i-gang',
        'md5': '2c37175c718155930f939ef59952474a',
        'info_dict': {
            'id': 'christiania-pusher-street-ryddes-drdkrjpo',
            'ext': 'mp4',
            'title': 'LIVE Christianias rydning af Pusher Street er i gang',
            'description': '- Det er det fedeste, der er sket i 20 år, fortæller christianit til DR Nyheder.',
            'timestamp': 1472800279,
            'upload_date': '20160902',
            'duration': 131.4,
        },
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -35,7 +54,8 @@ class DRTVIE(InfoExtractor):
                'Video %s is not available' % video_id, expected=True)
        video_id = self._search_regex(
-            r'data-(?:material-identifier|episode-slug)="([^"]+)"',
+            (r'data-(?:material-identifier|episode-slug)="([^"]+)"',
                r'data-resource="[^>"]+mu/programcard/expanded/([^"]+)"'),
            webpage, 'video id')
        programcard = self._download_json(
@@ -43,9 +63,12 @@ class DRTVIE(InfoExtractor):
            video_id, 'Downloading video JSON')
        data = programcard['Data'][0]
-        title = data['Title']
+        title = remove_end(self._og_search_title(
-        description = data['Description']
+            webpage, default=None), ' | TV | DR') or data['Title']
-        timestamp = parse_iso8601(data['CreatedTime'])
+        description = self._og_search_description(
            webpage, default=None) or data.get('Description')
        timestamp = parse_iso8601(data.get('CreatedTime'))
        thumbnail = None
        duration = None
@@ -56,16 +79,18 @@ class DRTVIE(InfoExtractor):
        subtitles = {}
        for asset in data['Assets']:
-            if asset['Kind'] == 'Image':
+            if asset.get('Kind') == 'Image':
-                thumbnail = asset['Uri']
+                thumbnail = asset.get('Uri')
-            elif asset['Kind'] == 'VideoResource':
+            elif asset.get('Kind') == 'VideoResource':
-                duration = asset['DurationInMilliseconds'] / 1000.0
+                duration = float_or_none(asset.get('DurationInMilliseconds'), 1000)
-                restricted_to_denmark = asset['RestrictedToDenmark']
+                restricted_to_denmark = asset.get('RestrictedToDenmark')
-                spoken_subtitles = asset['Target'] == 'SpokenSubtitles'
+                spoken_subtitles = asset.get('Target') == 'SpokenSubtitles'
-                for link in asset['Links']:
+                for link in asset.get('Links', []):
-                    uri = link['Uri']
+                    uri = link.get('Uri')
-                    target = link['Target']
+                    if not uri:
-                    format_id = target
+                        continue
                    target = link.get('Target')
                    format_id = target or ''
                    preference = None
                    if spoken_subtitles:
                        preference = -1
@@ -76,8 +101,8 @@ class DRTVIE(InfoExtractor):
                            video_id, preference, f4m_id=format_id))
                    elif target == 'HLS':
                        formats.extend(self._extract_m3u8_formats(
-                            uri, video_id, 'mp4', preference=preference,
+                            uri, video_id, 'mp4', entry_protocol='m3u8_native',
-                            m3u8_id=format_id))
+                            preference=preference, m3u8_id=format_id))
                    else:
                        bitrate = link.get('Bitrate')
                        if bitrate:
@@ -85,7 +110,7 @@ class DRTVIE(InfoExtractor):
                        formats.append({
                            'url': uri,
                            'format_id': format_id,
-                            'tbr': bitrate,
+                            'tbr': int_or_none(bitrate),
                            'ext': link.get('FileFormat'),
                        })
                subtitles_list = asset.get('SubtitlesList')
@@ -94,12 +119,18 @@ class DRTVIE(InfoExtractor):
                        'Danish': 'da',
                    }
                    for subs in subtitles_list:
-                        lang = subs['Language']
+                        if not subs.get('Uri'):
-                        subtitles[LANGS.get(lang, lang)] = [{'url': subs['Uri'], 'ext': 'vtt'}]
+                            continue
                        lang = subs.get('Language') or 'da'
                        subtitles.setdefault(LANGS.get(lang, lang), []).append({
                            'url': subs['Uri'],
                            'ext': mimetype2ext(subs.get('MimeType')) or 'vtt'
                        })
        if not formats and restricted_to_denmark:
-            raise ExtractorError(
+            self.raise_geo_restricted(
-                'Unfortunately, DR is not allowed to show this program outside Denmark.', expected=True)
+                'Unfortunately, DR is not allowed to show this program outside Denmark.',
                expected=True)
        self._sort_formats(formats)
--- a/youtube_dl/extractor/eagleplatform.py
+++ b/youtube_dl/extractor/eagleplatform.py
@@ -52,11 +52,24 @@ class EaglePlatformIE(InfoExtractor):
    @staticmethod
    def _extract_url(webpage):
        # Regular iframe embedding
        mobj = re.search(
            r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//.+?\.media\.eagleplatform\.com/index/player\?.+?)\1',
            webpage)
        if mobj is not None:
            return mobj.group('url')
        # Basic usage embedding (see http://dultonmedia.github.io/eplayer/)
        mobj = re.search(
            r'''(?xs)
                    <script[^>]+
                        src=(?P<q1>["\'])(?:https?:)?//(?P<host>.+?\.media\.eagleplatform\.com)/player/player\.js(?P=q1)
                    .+?
                    <div[^>]+
                        class=(?P<q2>["\'])eagleplayer(?P=q2)[^>]+
                        data-id=["\'](?P<id>\d+)
            ''', webpage)
        if mobj is not None:
            return 'eagleplatform:%(host)s:%(id)s' % mobj.groupdict()
    @staticmethod
    def _handle_error(response):
--- a/youtube_dl/extractor/exfm.py
+++ b/youtube_dl/extractor/exfm.py
@@ -1,58 +0,0 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 class ExfmIE(InfoExtractor):
    IE_NAME = 'exfm'
    IE_DESC = 'ex.fm'
    _VALID_URL = r'https?://(?:www\.)?ex\.fm/song/(?P<id>[^/]+)'
    _SOUNDCLOUD_URL = r'http://(?:www\.)?api\.soundcloud\.com/tracks/([^/]+)/stream'
    _TESTS = [
        {
            'url': 'http://ex.fm/song/eh359',
            'md5': 'e45513df5631e6d760970b14cc0c11e7',
            'info_dict': {
                'id': '44216187',
                'ext': 'mp3',
                'title': 'Test House "Love Is Not Enough" (Extended Mix) DeadJournalist Exclusive',
                'uploader': 'deadjournalist',
                'upload_date': '20120424',
                'description': 'Test House \"Love Is Not Enough\" (Extended Mix) DeadJournalist Exclusive',
            },
            'note': 'Soundcloud song',
            'skip': 'The site is down too often',
        },
        {
            'url': 'http://ex.fm/song/wddt8',
            'md5': '966bd70741ac5b8570d8e45bfaed3643',
            'info_dict': {
                'id': 'wddt8',
                'ext': 'mp3',
                'title': 'Safe and Sound',
                'uploader': 'Capital Cities',
            },
            'skip': 'The site is down too often',
        },
    ]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        song_id = mobj.group('id')
        info_url = 'http://ex.fm/api/v3/song/%s' % song_id
        info = self._download_json(info_url, song_id)['song']
        song_url = info['url']
        if re.match(self._SOUNDCLOUD_URL, song_url) is not None:
            self.to_screen('Soundcloud song detected')
            return self.url_result(song_url.replace('/stream', ''), 'Soundcloud')
        return {
            'id': song_id,
            'url': song_url,
            'ext': 'mp3',
            'title': info['title'],
            'thumbnail': info['image']['large'],
            'uploader': info['artist'],
            'view_count': info['loved_count'],
        }
--- a/youtube_dl/extractor/expotv.py
+++ b/youtube_dl/extractor/expotv.py
@@ -1,7 +1,5 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
@@ -12,23 +10,22 @@ from ..utils import (
 class ExpoTVIE(InfoExtractor):
    _VALID_URL = r'https?://www\.expotv\.com/videos/[^?#]*/(?P<id>[0-9]+)($|[?#])'
    _TEST = {
-        'url': 'http://www.expotv.com/videos/reviews/1/24/LinneCardscom/17561',
+        'url': 'http://www.expotv.com/videos/reviews/3/40/NYX-Butter-lipstick/667916',
-        'md5': '2985e6d7a392b2f7a05e0ca350fe41d0',
+        'md5': 'fe1d728c3a813ff78f595bc8b7a707a8',
        'info_dict': {
-            'id': '17561',
+            'id': '667916',
            'ext': 'mp4',
-            'upload_date': '20060212',
+            'title': 'NYX Butter Lipstick Little Susie',
-            'title': 'My Favorite Online Scrapbook Store',
+            'description': 'Goes on like butter, but looks better!',
            'view_count': int,
            'description': 'You\'ll find most everything you need at this virtual store front.',
            'uploader': 'Anna T.',
            'thumbnail': 're:^https?://.*\.jpg$',
            'uploader': 'Stephanie S.',
            'upload_date': '20150520',
            'view_count': int,
        }
    }
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id = self._match_id(url)
        video_id = mobj.group('id')
        webpage = self._download_webpage(url, video_id)
        player_key = self._search_regex(
@@ -66,7 +63,7 @@ class ExpoTVIE(InfoExtractor):
            fatal=False)
        upload_date = unified_strdate(self._search_regex(
            r'<h5>Reviewed on ([0-9/.]+)</h5>', webpage, 'upload date',
-            fatal=False))
+            fatal=False), day_first=False)
        return {
            'id': video_id,
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@@ -1,7 +1,10 @@
 # flake8: noqa
 from __future__ import unicode_literals
-from .abc import ABCIE
+from .abc import (
    ABCIE,
    ABCIViewIE,
 )
 from .abc7news import Abc7NewsIE
 from .abcnews import (
    AbcNewsIE,
@@ -29,6 +32,7 @@ from .aftonbladet import AftonbladetIE
 from .airmozilla import AirMozillaIE
 from .aljazeera import AlJazeeraIE
 from .alphaporno import AlphaPornoIE
 from .amcnetworks import AMCNetworksIE
 from .animeondemand import AnimeOnDemandIE
 from .anitube import AnitubeIE
 from .anysex import AnySexIE
@@ -67,6 +71,12 @@ from .atttechchannel import ATTTechChannelIE
 from .audimedia import AudiMediaIE
 from .audioboom import AudioBoomIE
 from .audiomack import AudiomackIE, AudiomackAlbumIE
 from .awaan import (
    AWAANIE,
    AWAANVideoIE,
    AWAANLiveIE,
    AWAANSeasonIE,
 )
 from .azubu import AzubuIE, AzubuLiveIE
 from .baidu import BaiduVideoIE
 from .bambuser import BambuserIE, BambuserChannelIE
@@ -117,9 +127,12 @@ from .carambatv import (
    CarambaTVIE,
    CarambaTVPageIE,
 )
 from .cartoonnetwork import CartoonNetworkIE
 from .cbc import (
    CBCIE,
    CBCPlayerIE,
    CBCWatchVideoIE,
    CBCWatchIE,
 )
 from .cbs import CBSIE
 from .cbslocal import CBSLocalIE
@@ -133,6 +146,7 @@ from .ccc import CCCIE
 from .cda import CDAIE
 from .ceskatelevize import CeskaTelevizeIE
 from .channel9 import Channel9IE
 from .charlierose import CharlieRoseIE
 from .chaturbate import ChaturbateIE
 from .chilloutzone import ChilloutzoneIE
 from .chirbit import (
@@ -180,6 +194,10 @@ from .ctsnews import CtsNewsIE
 from .ctv import CTVIE
 from .ctvnews import CTVNewsIE
 from .cultureunplugged import CultureUnpluggedIE
 from .curiositystream import (
    CuriosityStreamIE,
    CuriosityStreamCollectionIE,
 )
 from .cwtv import CWTVIE
 from .dailymail import DailyMailIE
 from .dailymotion import (
@@ -195,12 +213,6 @@ from .daum import (
    DaumUserIE,
 )
 from .dbtv import DBTVIE
 from .dcn import (
    DCNIE,
    DCNVideoIE,
    DCNLiveIE,
    DCNSeasonIE,
 )
 from .dctp import DctpTvIE
 from .deezer import DeezerPlaylistIE
 from .democracynow import DemocracynowIE
@@ -249,11 +261,13 @@ from .espn import ESPNIE
 from .esri import EsriVideoIE
 from .europa import EuropaIE
 from .everyonesmixtape import EveryonesMixtapeIE
 from .exfm import ExfmIE
 from .expotv import ExpoTVIE
 from .extremetube import ExtremeTubeIE
 from .eyedotv import EyedoTVIE
-from .facebook import FacebookIE
+from .facebook import (
    FacebookIE,
    FacebookPluginsVideoIE,
 )
 from .faz import FazIE
 from .fc2 import FC2IE
 from .fczenit import FczenitIE
@@ -272,10 +286,7 @@ from .fox import FOXIE
 from .foxgay import FoxgayIE
 from .foxnews import FoxNewsIE
 from .foxsports import FoxSportsIE
-from .franceculture import (
+from .franceculture import FranceCultureIE
    FranceCultureIE,
    FranceCultureEmissionIE,
 )
 from .franceinter import FranceInterIE
 from .francetv import (
    PluzzIE,
@@ -290,6 +301,7 @@ from .freevideo import FreeVideoIE
 from .funimation import FunimationIE
 from .funnyordie import FunnyOrDieIE
 from .fusion import FusionIE
 from .fxnetworks import FXNetworksIE
 from .gameinformer import GameInformerIE
 from .gameone import (
    GameOneIE,
@@ -309,9 +321,9 @@ from .globo import (
    GloboIE,
    GloboArticleIE,
 )
 from .go import GoIE
 from .godtube import GodTubeIE
 from .godtv import GodTVIE
 from .goldenmoustache import GoldenMoustacheIE
 from .golem import GolemIE
 from .googledrive import GoogleDriveIE
 from .googleplus import GooglePlusIE
@@ -326,6 +338,10 @@ from .heise import HeiseIE
 from .hellporno import HellPornoIE
 from .helsinki import HelsinkiIE
 from .hentaistigma import HentaiStigmaIE
 from .hgtv import (
    HGTVIE,
    HGTVComShowIE,
 )
 from .historicfilms import HistoricFilmsIE
 from .hitbox import HitboxIE, HitboxLiveIE
 from .hornbunny import HornBunnyIE
@@ -477,6 +493,7 @@ from .motherless import MotherlessIE
 from .motorsport import MotorsportIE
 from .movieclips import MovieClipsIE
 from .moviezine import MoviezineIE
 from .movingimage import MovingImageIE
 from .msn import MSNIE
 from .mtv import (
    MTVIE,
@@ -533,6 +550,7 @@ from .nextmedia import (
 )
 from .nfb import NFBIE
 from .nfl import NFLIE
 from .nhk import NhkVodIE
 from .nhl import (
    NHLVideocenterIE,
    NHLNewsIE,
@@ -544,7 +562,10 @@ from .nick import (
    NickDeIE,
 )
 from .niconico import NiconicoIE, NiconicoPlaylistIE
-from .ninecninemedia import NineCNineMediaIE
+from .ninecninemedia import (
    NineCNineMediaStackIE,
    NineCNineMediaIE,
 )
 from .ninegag import NineGagIE
 from .ninenow import NineNowIE
 from .nintendo import NintendoIE
@@ -626,7 +647,6 @@ from .phoenix import PhoenixIE
 from .photobucket import PhotobucketIE
 from .pinkbike import PinkbikeIE
 from .pladform import PladformIE
 from .played import PlayedIE
 from .playfm import PlayFMIE
 from .plays import PlaysTVIE
 from .playtvak import PlaytvakIE
@@ -640,6 +660,7 @@ from .podomatic import PodomaticIE
 from .pokemon import PokemonIE
 from .polskieradio import PolskieRadioIE
 from .porn91 import Porn91IE
 from .porncom import PornComIE
 from .pornhd import PornHdIE
 from .pornhub import (
    PornHubIE,
@@ -696,6 +717,7 @@ from .rockstargames import RockstarGamesIE
 from .roosterteeth import RoosterTeethIE
 from .rottentomatoes import RottenTomatoesIE
 from .roxwel import RoxwelIE
 from .rozhlas import RozhlasIE
 from .rtbf import RTBFIE
 from .rte import RteIE, RteRadioIE
 from .rtlnl import RtlNlIE
@@ -755,6 +777,7 @@ from .smotri import (
 )
 from .snotr import SnotrIE
 from .sohu import SohuIE
 from .sonyliv import SonyLIVIE
 from .soundcloud import (
    SoundcloudIE,
    SoundcloudSetIE,
@@ -791,7 +814,6 @@ from .srgssr import (
    SRGSSRPlayIE,
 )
 from .srmediathek import SRMediathekIE
 from .ssa import SSAIE
 from .stanfordoc import StanfordOpenClassroomIE
 from .steam import SteamIE
 from .streamable import StreamableIE
@@ -810,8 +832,8 @@ from .tagesschau import (
    TagesschauPlayerIE,
    TagesschauIE,
 )
 from .tapely import TapelyIE
 from .tass import TassIE
 from .tbs import TBSIE
 from .tdslifeway import TDSLifewayIE
 from .teachertube import (
    TeacherTubeIE,
@@ -854,16 +876,11 @@ from .tnaflix import (
    MovieFapIE,
 )
 from .toggle import ToggleIE
 from .thvideo import (
    THVideoIE,
    THVideoPlaylistIE
 )
 from .toutv import TouTvIE
 from .toypics import ToypicsUserIE, ToypicsIE
 from .traileraddict import TrailerAddictIE
 from .trilulilu import TriluliluIE
 from .trollvids import TrollvidsIE
 from .trutube import TruTubeIE
 from .tube8 import Tube8IE
 from .tubitv import TubiTvIE
 from .tudou import (
@@ -898,7 +915,10 @@ from .tvp import (
    TVPIE,
    TVPSeriesIE,
 )
-from .tvplay import TVPlayIE
+from .tvplay import (
    TVPlayIE,
    ViafreeIE,
 )
 from .tweakers import TweakersIE
 from .twentyfourvideo import TwentyFourVideoIE
 from .twentymin import TwentyMinutenIE
@@ -927,8 +947,14 @@ from .udemy import (
 from .udn import UDNEmbedIE
 from .digiteka import DigitekaIE
 from .unistra import UnistraIE
 from .uol import UOLIE
 from .uplynk import (
    UplynkIE,
    UplynkPreplayIE,
 )
 from .urort import UrortIE
 from .urplay import URPlayIE
 from .usanetwork import USANetworkIE
 from .usatoday import USATodayIE
 from .ustream import UstreamIE, UstreamChannelIE
 from .ustudio import (
@@ -955,6 +981,7 @@ from .vice import (
    ViceIE,
    ViceShowIE,
 )
 from .viceland import VicelandIE
 from .vidbit import VidbitIE
 from .viddler import ViddlerIE
 from .videodetective import VideoDetectiveIE
@@ -1101,8 +1128,4 @@ from .youtube import (
 )
 from .zapiks import ZapiksIE
 from .zdf import ZDFIE, ZDFChannelIE
-from .zingmp3 import (
+from .zingmp3 import ZingMp3IE
    ZingMp3SongIE,
    ZingMp3AlbumIE,
 )
 from .zippcast import ZippCastIE
--- a/youtube_dl/extractor/extremetube.py
+++ b/youtube_dl/extractor/extremetube.py
@@ -1,20 +1,14 @@
 from __future__ import unicode_literals
-import re
+from ..utils import str_to_int
-
+from .keezmovies import KeezMoviesIE
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    sanitized_Request,
    str_to_int,
 )
-class ExtremeTubeIE(InfoExtractor):
+class ExtremeTubeIE(KeezMoviesIE):
    _VALID_URL = r'https?://(?:www\.)?extremetube\.com/(?:[^/]+/)?video/(?P<id>[^/#?&]+)'
    _TESTS = [{
        'url': 'http://www.extremetube.com/video/music-video-14-british-euro-brit-european-cumshots-swallow-652431',
-        'md5': '344d0c6d50e2f16b06e49ca011d8ac69',
+        'md5': '1fb9228f5e3332ec8c057d6ac36f33e0',
        'info_dict': {
            'id': 'music-video-14-british-euro-brit-european-cumshots-swallow-652431',
            'ext': 'mp4',
@@ -35,58 +29,22 @@ class ExtremeTubeIE(InfoExtractor):
    }]
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        webpage, info = self._extract_info(url)
-        req = sanitized_Request(url)
+        if not info['title']:
-        req.add_header('Cookie', 'age_verified=1')
+            info['title'] = self._search_regex(
-        webpage = self._download_webpage(req, video_id)
+                r'<h1[^>]+title="([^"]+)"[^>]*>', webpage, 'title')
        video_title = self._html_search_regex(
            r'<h1 [^>]*?title="([^"]+)"[^>]*>', webpage, 'title')
        uploader = self._html_search_regex(
            r'Uploaded by:\s*</strong>\s*(.+?)\s*</div>',
            webpage, 'uploader', fatal=False)
-        view_count = str_to_int(self._html_search_regex(
+        view_count = str_to_int(self._search_regex(
            r'Views:\s*</strong>\s*<span>([\d,\.]+)</span>',
            webpage, 'view count', fatal=False))
-        flash_vars = self._parse_json(
+        info.update({
            self._search_regex(
                r'var\s+flashvars\s*=\s*({.+?});', webpage, 'flash vars'),
            video_id)
        formats = []
        for quality_key, video_url in flash_vars.items():
            height = int_or_none(self._search_regex(
                r'quality_(\d+)[pP]$', quality_key, 'height', default=None))
            if not height:
                continue
            f = {
                'url': video_url,
            }
            mobj = re.search(
                r'/(?P<height>\d{3,4})[pP]_(?P<bitrate>\d+)[kK]_\d+', video_url)
            if mobj:
                height = int(mobj.group('height'))
                bitrate = int(mobj.group('bitrate'))
                f.update({
                    'format_id': '%dp-%dk' % (height, bitrate),
                    'height': height,
                    'tbr': bitrate,
                })
            else:
                f.update({
                    'format_id': '%dp' % height,
                    'height': height,
                })
            formats.append(f)
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': video_title,
            'formats': formats,
            'uploader': uploader,
            'view_count': view_count,
-            'age_limit': 18,
+        })
-        }
+
        return info
--- a/youtube_dl/extractor/facebook.py
+++ b/youtube_dl/extractor/facebook.py
@@ -15,6 +15,7 @@ from ..compat import (
 from ..utils import (
    error_to_compat_str,
    ExtractorError,
    int_or_none,
    limit_length,
    sanitized_Request,
    urlencode_postdata,
@@ -62,6 +63,8 @@ class FacebookIE(InfoExtractor):
            'ext': 'mp4',
            'title': 're:Did you know Kei Nishikori is the first Asian man to ever reach a Grand Slam',
            'uploader': 'Tennis on Facebook',
            'upload_date': '20140908',
            'timestamp': 1410199200,
        }
    }, {
        'note': 'Video without discernible title',
@@ -71,6 +74,8 @@ class FacebookIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Facebook video #274175099429670',
            'uploader': 'Asif Nawab Butt',
            'upload_date': '20140506',
            'timestamp': 1399398998,
        },
        'expected_warnings': [
            'title'
@@ -78,12 +83,14 @@ class FacebookIE(InfoExtractor):
    }, {
        'note': 'Video with DASH manifest',
        'url': 'https://www.facebook.com/video.php?v=957955867617029',
-        'md5': '54706e4db4f5ad58fbad82dde1f1213f',
+        'md5': 'b2c28d528273b323abe5c6ab59f0f030',
        'info_dict': {
            'id': '957955867617029',
            'ext': 'mp4',
            'title': 'When you post epic content on instagram.com/433 8 million followers, this is ...',
            'uploader': 'Demy de Zeeuw',
            'upload_date': '20160110',
            'timestamp': 1452431627,
        },
    }, {
        'url': 'https://www.facebook.com/maxlayn/posts/10153807558977570',
@@ -306,12 +313,16 @@ class FacebookIE(InfoExtractor):
        if not video_title:
            video_title = 'Facebook video #%s' % video_id
        uploader = clean_html(get_element_by_id('fbPhotoPageAuthorName', webpage))
        timestamp = int_or_none(self._search_regex(
            r'<abbr[^>]+data-utime=["\'](\d+)', webpage,
            'timestamp', default=None))
        info_dict = {
            'id': video_id,
            'title': video_title,
            'formats': formats,
            'uploader': uploader,
            'timestamp': timestamp,
        }
        return webpage, info_dict
@@ -340,3 +351,32 @@ class FacebookIE(InfoExtractor):
                self._VIDEO_PAGE_TEMPLATE % video_id,
                video_id, fatal_if_no_video=True)
            return info_dict
 class FacebookPluginsVideoIE(InfoExtractor):
    _VALID_URL = r'https?://(?:[\w-]+\.)?facebook\.com/plugins/video\.php\?.*?\bhref=(?P<id>https.+)'
    _TESTS = [{
        'url': 'https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fgov.sg%2Fvideos%2F10154383743583686%2F&show_text=0&width=560',
        'md5': '5954e92cdfe51fe5782ae9bda7058a07',
        'info_dict': {
            'id': '10154383743583686',
            'ext': 'mp4',
            'title': 'What to do during the haze?',
            'uploader': 'Gov.sg',
            'upload_date': '20160826',
            'timestamp': 1472184808,
        },
        'add_ie': [FacebookIE.ie_key()],
    }, {
        'url': 'https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fvideo.php%3Fv%3D10204634152394104',
        'only_matching': True,
    }, {
        'url': 'https://www.facebook.com/plugins/video.php?href=https://www.facebook.com/gov.sg/videos/10154383743583686/&show_text=0&width=560',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        return self.url_result(
            compat_urllib_parse_unquote(self._match_id(url)),
            FacebookIE.ie_key())
--- a/youtube_dl/extractor/firsttv.py
+++ b/youtube_dl/extractor/firsttv.py
@@ -2,44 +2,40 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
-from ..compat import compat_xpath
+from ..compat import compat_urlparse
 from ..utils import (
    int_or_none,
    qualities,
    unified_strdate,
    xpath_attr,
    xpath_element,
    xpath_text,
    xpath_with_ns,
 )
 class FirstTVIE(InfoExtractor):
    IE_NAME = '1tv'
    IE_DESC = 'Первый канал'
-    _VALID_URL = r'https?://(?:www\.)?1tv\.ru/(?:[^/]+/)+p?(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?1tv\.ru/(?:[^/]+/)+(?P<id>[^/?#]+)'
    _TESTS = [{
-        # single format via video_materials.json API
+        # single format
-        'url': 'http://www.1tv.ru/prj/inprivate/vypusk/35930',
+        'url': 'http://www.1tv.ru/shows/naedine-so-vsemi/vypuski/gost-lyudmila-senchina-naedine-so-vsemi-vypusk-ot-12-02-2015',
-        'md5': '82a2777648acae812d58b3f5bd42882b',
+        'md5': 'a1b6b60d530ebcf8daacf4565762bbaf',
        'info_dict': {
-            'id': '35930',
+            'id': '40049',
            'ext': 'mp4',
            'title': 'Гость Людмила Сенчина. Наедине со всеми. Выпуск от 12.02.2015',
-            'description': 'md5:357933adeede13b202c7c21f91b871b2',
+            'description': 'md5:36a39c1d19618fec57d12efe212a8370',
            'thumbnail': 're:^https?://.*\.(?:jpg|JPG)$',
            'upload_date': '20150212',
            'duration': 2694,
        },
    }, {
-        # multiple formats via video_materials.json API
+        # multiple formats
-        'url': 'http://www.1tv.ru/video_archive/projects/dobroeutro/p113641',
+        'url': 'http://www.1tv.ru/shows/dobroe-utro/pro-zdorove/vesennyaya-allergiya-dobroe-utro-fragment-vypuska-ot-07042016',
        'info_dict': {
-            'id': '113641',
+            'id': '364746',
            'ext': 'mp4',
            'title': 'Весенняя аллергия. Доброе утро. Фрагмент выпуска от 07.04.2016',
-            'description': 'md5:8dcebb3dded0ff20fade39087fd1fee2',
+            'description': 'md5:a242eea0031fd180a4497d52640a9572',
            'thumbnail': 're:^https?://.*\.(?:jpg|JPG)$',
            'upload_date': '20160407',
            'duration': 179,
@@ -48,84 +44,47 @@ class FirstTVIE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
    }, {
        # single format only available via ONE_ONLINE_VIDEOS.archive_single_xml API
        'url': 'http://www.1tv.ru/video_archive/series/f7552/p47038',
        'md5': '519d306c5b5669761fd8906c39dbee23',
        'info_dict': {
            'id': '47038',
            'ext': 'mp4',
            'title': '"Побег". Второй сезон. 3 серия',
            'description': 'md5:3abf8f6b9bce88201c33e9a3d794a00b',
            'thumbnail': 're:^https?://.*\.(?:jpg|JPG)$',
            'upload_date': '20120516',
            'duration': 3080,
        },
    }, {
        'url': 'http://www.1tv.ru/videoarchive/9967',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        display_id = self._match_id(url)
-        # Videos with multiple formats only available via this API
+        webpage = self._download_webpage(url, display_id)
-        video = self._download_json(
+        playlist_url = compat_urlparse.urljoin(url, self._search_regex(
-            'http://www.1tv.ru/video_materials.json?legacy_id=%s' % video_id,
+            r'data-playlist-url="([^"]+)', webpage, 'playlist url'))
            video_id, fatal=False)
        description, thumbnail, upload_date, duration = [None] * 4
        if video:
            item = video[0]
            title = item['title']
            quality = qualities(('ld', 'sd', 'hd', ))
            formats = [{
                'url': f['src'],
                'format_id': f.get('name'),
                'quality': quality(f.get('name')),
            } for f in item['mbr'] if f.get('src')]
            thumbnail = item.get('poster')
        else:
            # Some videos are not available via video_materials.json
            video = self._download_xml(
                'http://www.1tv.ru/owa/win/ONE_ONLINE_VIDEOS.archive_single_xml?pid=%s' % video_id,
                video_id)
            NS_MAP = {
                'media': 'http://search.yahoo.com/mrss/',
            }
            item = xpath_element(video, './channel/item', fatal=True)
            title = xpath_text(item, './title', fatal=True)
            formats = [{
                'url': content.attrib['url'],
            } for content in item.findall(
                compat_xpath(xpath_with_ns('./media:content', NS_MAP))) if content.attrib.get('url')]
            thumbnail = xpath_attr(
                item, xpath_with_ns('./media:thumbnail', NS_MAP), 'url')
        item = self._download_json(playlist_url, display_id)[0]
        video_id = item['id']
        quality = qualities(('ld', 'sd', 'hd', ))
        formats = []
        for f in item.get('mbr', []):
            src = f.get('src')
            if not src:
                continue
            fname = f.get('name')
            formats.append({
                'url': src,
                'format_id': fname,
                'quality': quality(fname),
            })
        self._sort_formats(formats)
-        webpage = self._download_webpage(url, video_id, 'Downloading page', fatal=False)
+        title = self._html_search_regex(
-        if webpage:
+            (r'<div class="tv_translation">\s*<h1><a href="[^"]+">([^<]*)</a>',
-            title = self._html_search_regex(
+             r"'title'\s*:\s*'([^']+)'"),
-                (r'<div class="tv_translation">\s*<h1><a href="[^"]+">([^<]*)</a>',
+            webpage, 'title', default=None) or item['title']
-                 r"'title'\s*:\s*'([^']+)'"),
+        description = self._html_search_regex(
-                webpage, 'title', default=None) or title
+            r'<div class="descr">\s*<div>&nbsp;</div>\s*<p>([^<]*)</p></div>',
-            description = self._html_search_regex(
+            webpage, 'description', default=None) or self._html_search_meta(
-                r'<div class="descr">\s*<div>&nbsp;</div>\s*<p>([^<]*)</p></div>',
+            'description', webpage, 'description')
-                webpage, 'description', default=None) or self._html_search_meta(
+        duration = int_or_none(self._html_search_meta(
-                'description', webpage, 'description')
+            'video:duration', webpage, 'video duration', fatal=False))
-            thumbnail = thumbnail or self._og_search_thumbnail(webpage)
+        upload_date = unified_strdate(self._html_search_meta(
-            duration = int_or_none(self._html_search_meta(
+            'ya:ovs:upload_date', webpage, 'upload date', fatal=False))
                'video:duration', webpage, 'video duration', fatal=False))
            upload_date = unified_strdate(self._html_search_meta(
                'ya:ovs:upload_date', webpage, 'upload date', fatal=False))
        return {
            'id': video_id,
-            'thumbnail': thumbnail,
+            'thumbnail': item.get('poster') or self._og_search_thumbnail(webpage),
            'title': title,
            'description': description,
            'upload_date': upload_date,
--- a/youtube_dl/extractor/flipagram.py
+++ b/youtube_dl/extractor/flipagram.py
@@ -48,7 +48,7 @@ class FlipagramIE(InfoExtractor):
        flipagram = video_data['flipagram']
        video = flipagram['video']
-        json_ld = self._search_json_ld(webpage, video_id, fatal=False)
+        json_ld = self._search_json_ld(webpage, video_id, default={})
        title = json_ld.get('title') or flipagram['captionText']
        description = json_ld.get('description') or flipagram.get('captionText')
--- a/youtube_dl/extractor/formula1.py
+++ b/youtube_dl/extractor/formula1.py
@@ -5,8 +5,8 @@ from .common import InfoExtractor
 class Formula1IE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?formula1\.com/content/fom-website/en/video/\d{4}/\d{1,2}/(?P<id>.+?)\.html'
+    _VALID_URL = r'https?://(?:www\.)?formula1\.com/(?:content/fom-website/)?en/video/\d{4}/\d{1,2}/(?P<id>.+?)\.html'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://www.formula1.com/content/fom-website/en/video/2016/5/Race_highlights_-_Spain_2016.html',
        'md5': '8c79e54be72078b26b89e0e111c0502b',
        'info_dict': {
@@ -15,7 +15,10 @@ class Formula1IE(InfoExtractor):
            'title': 'Race highlights - Spain 2016',
        },
        'add_ie': ['Ooyala'],
-    }
+    }, {
        'url': 'http://www.formula1.com/en/video/2016/5/Race_highlights_-_Spain_2016.html',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
--- a/youtube_dl/extractor/fourtube.py
+++ b/youtube_dl/extractor/fourtube.py
@@ -43,14 +43,14 @@ class FourTubeIE(InfoExtractor):
            'uploadDate', webpage))
        thumbnail = self._html_search_meta('thumbnailUrl', webpage)
        uploader_id = self._html_search_regex(
-            r'<a class="img-avatar" href="[^"]+/channels/([^/"]+)" title="Go to [^"]+ page">',
+            r'<a class="item-to-subscribe" href="[^"]+/channels/([^/"]+)" title="Go to [^"]+ page">',
            webpage, 'uploader id', fatal=False)
        uploader = self._html_search_regex(
-            r'<a class="img-avatar" href="[^"]+/channels/[^/"]+" title="Go to ([^"]+) page">',
+            r'<a class="item-to-subscribe" href="[^"]+/channels/[^/"]+" title="Go to ([^"]+) page">',
            webpage, 'uploader', fatal=False)
        categories_html = self._search_regex(
-            r'(?s)><i class="icon icon-tag"></i>\s*Categories / Tags\s*.*?<ul class="list">(.*?)</ul>',
+            r'(?s)><i class="icon icon-tag"></i>\s*Categories / Tags\s*.*?<ul class="[^"]*?list[^"]*?">(.*?)</ul>',
            webpage, 'categories', fatal=False)
        categories = None
        if categories_html:
@@ -59,10 +59,10 @@ class FourTubeIE(InfoExtractor):
                    r'(?s)<li><a.*?>(.*?)</a>', categories_html)]
        view_count = str_to_int(self._search_regex(
-            r'<meta itemprop="interactionCount" content="UserPlays:([0-9,]+)">',
+            r'<meta[^>]+itemprop="interactionCount"[^>]+content="UserPlays:([0-9,]+)">',
            webpage, 'view count', fatal=False))
        like_count = str_to_int(self._search_regex(
-            r'<meta itemprop="interactionCount" content="UserLikes:([0-9,]+)">',
+            r'<meta[^>]+itemprop="interactionCount"[^>]+content="UserLikes:([0-9,]+)">',
            webpage, 'like count', fatal=False))
        duration = parse_duration(self._html_search_meta('duration', webpage))
--- a/youtube_dl/extractor/franceculture.py
+++ b/youtube_dl/extractor/franceculture.py
@@ -2,104 +2,56 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..compat import (
    compat_urlparse,
 )
 from ..utils import (
    determine_ext,
-    int_or_none,
+    unified_strdate,
    ExtractorError,
 )
 class FranceCultureIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/player/reecouter\?play=(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/emissions/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TEST = {
-        'url': 'http://www.franceculture.fr/player/reecouter?play=4795174',
+        'url': 'http://www.franceculture.fr/emissions/carnet-nomade/rendez-vous-au-pays-des-geeks',
        'info_dict': {
-            'id': '4795174',
+            'id': 'rendez-vous-au-pays-des-geeks',
            'display_id': 'rendez-vous-au-pays-des-geeks',
            'ext': 'mp3',
            'title': 'Rendez-vous au pays des geeks',
-            'alt_title': 'Carnet nomade | 13-14',
+            'thumbnail': 're:^https?://.*\\.jpg$',
            'vcodec': 'none',
            'upload_date': '20140301',
-            'thumbnail': r're:^http://static\.franceculture\.fr/.*/images/player/Carnet-nomade\.jpg$',
+            'vcodec': 'none',
            'description': 'startswith:Avec :Jean-Baptiste Péretié pour son documentaire sur Arte "La revanche',
            'timestamp': 1393700400,
        }
    }
-    def _extract_from_player(self, url, video_id):
+    def _real_extract(self, url):
-        webpage = self._download_webpage(url, video_id)
+        display_id = self._match_id(url)
-        video_path = self._search_regex(
+        webpage = self._download_webpage(url, display_id)
-            r'<a id="player".*?href="([^"]+)"', webpage, 'video path')
+
-        video_url = compat_urlparse.urljoin(url, video_path)
+        video_url = self._search_regex(
-        timestamp = int_or_none(self._search_regex(
+            r'(?s)<div[^>]+class="[^"]*?title-zone-diffusion[^"]*?"[^>]*>.*?<a[^>]+href="([^"]+)"',
-            r'<a id="player".*?data-date="([0-9]+)"',
+            webpage, 'video path')
        title = self._og_search_title(webpage)
        upload_date = unified_strdate(self._search_regex(
            '(?s)<div[^>]+class="date"[^>]*>.*?<span[^>]+class="inner"[^>]*>([^<]+)<',
            webpage, 'upload date', fatal=False))
        thumbnail = self._search_regex(
-            r'<a id="player".*?>\s+<img src="([^"]+)"',
+            r'(?s)<figure[^>]+itemtype="https://schema.org/ImageObject"[^>]*>.*?<img[^>]+data-pagespeed-(?:lazy|high-res)-src="([^"]+)"',
            webpage, 'thumbnail', fatal=False)
        display_id = self._search_regex(
            r'<span class="path-diffusion">emission-(.*?)</span>', webpage, 'display_id')
        title = self._html_search_regex(
            r'<span class="title-diffusion">(.*?)</span>', webpage, 'title')
        alt_title = self._html_search_regex(
            r'<span class="title">(.*?)</span>',
            webpage, 'alt_title', fatal=False)
        description = self._html_search_regex(
            r'<span class="description">(.*?)</span>',
            webpage, 'description', fatal=False)
        uploader = self._html_search_regex(
            r'(?s)<div id="emission".*?<span class="author">(.*?)</span>',
            webpage, 'uploader', default=None)
        vcodec = 'none' if determine_ext(video_url.lower()) == 'mp3' else None
        return {
-            'id': video_id,
+            'id': display_id,
            'display_id': display_id,
            'url': video_url,
            'title': title,
            'thumbnail': thumbnail,
            'vcodec': vcodec,
            'uploader': uploader,
-            'timestamp': timestamp,
+            'upload_date': upload_date,
            'title': title,
            'alt_title': alt_title,
            'thumbnail': thumbnail,
            'description': description,
            'display_id': display_id,
        }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        return self._extract_from_player(url, video_id)
 class FranceCultureEmissionIE(FranceCultureIE):
    _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/emission-(?P<id>[^?#]+)'
    _TEST = {
        'url': 'http://www.franceculture.fr/emission-les-carnets-de-la-creation-jean-gabriel-periot-cineaste-2015-10-13',
        'info_dict': {
            'title': 'Jean-Gabriel Périot, cinéaste',
            'alt_title': 'Les Carnets de la création',
            'id': '5093239',
            'display_id': 'les-carnets-de-la-creation-jean-gabriel-periot-cineaste-2015-10-13',
            'ext': 'mp3',
            'timestamp': 1444762500,
            'upload_date': '20151013',
            'description': 'startswith:Aujourd\'hui dans "Les carnets de la création", le cinéaste',
        },
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        video_path = self._html_search_regex(
            r'<a class="rf-player-open".*?href="([^"]+)"', webpage, 'video path', 'no_path_player')
        if video_path == 'no_path_player':
            raise ExtractorError('no player : no sound in this page.', expected=True)
        new_id = self._search_regex('play=(?P<id>[0-9]+)', video_path, 'new_id', group='id')
        video_url = compat_urlparse.urljoin(url, video_path)
        return self._extract_from_player(video_url, new_id)
--- a/youtube_dl/extractor/francetv.py
+++ b/youtube_dl/extractor/francetv.py
@@ -131,7 +131,7 @@ class PluzzIE(FranceTVBaseInfoExtractor):
 class FranceTvInfoIE(FranceTVBaseInfoExtractor):
    IE_NAME = 'francetvinfo.fr'
-    _VALID_URL = r'https?://(?:www|mobile|france3-regions)\.francetvinfo\.fr/.*/(?P<title>.+)\.html'
+    _VALID_URL = r'https?://(?:www|mobile|france3-regions)\.francetvinfo\.fr/(?:[^/]+/)*(?P<title>[^/?#&.]+)'
    _TESTS = [{
        'url': 'http://www.francetvinfo.fr/replay-jt/france-3/soir-3/jt-grand-soir-3-lundi-26-aout-2013_393427.html',
@@ -206,6 +206,9 @@ class FranceTvInfoIE(FranceTVBaseInfoExtractor):
            'uploader_id': 'x2q2ez',
        },
        'add_ie': ['Dailymotion'],
    }, {
        'url': 'http://france3-regions.francetvinfo.fr/limousin/emissions/jt-1213-limousin',
        'only_matching': True,
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/fxnetworks.py
+++ b/youtube_dl/extractor/fxnetworks.py
@@ -0,0 +1,70 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .adobepass import AdobePassIE
 from ..utils import (
    update_url_query,
    extract_attributes,
    parse_age_limit,
    smuggle_url,
 )
 class FXNetworksIE(AdobePassIE):
    _VALID_URL = r'https?://(?:www\.)?(?:fxnetworks|simpsonsworld)\.com/video/(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.fxnetworks.com/video/719841347694',
        'md5': '1447d4722e42ebca19e5232ab93abb22',
        'info_dict': {
            'id': '719841347694',
            'ext': 'mp4',
            'title': 'Vanpage',
            'description': 'F*ck settling down. You\'re the Worst returns for an all new season August 31st on FXX.',
            'age_limit': 14,
            'uploader': 'NEWA-FNG-FX',
            'upload_date': '20160706',
            'timestamp': 1467844741,
        },
        'add_ie': ['ThePlatform'],
    }, {
        'url': 'http://www.simpsonsworld.com/video/716094019682',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        if 'The content you are trying to access is not available in your region.' in webpage:
            self.raise_geo_restricted()
        video_data = extract_attributes(self._search_regex(
            r'(<a.+?rel="http://link\.theplatform\.com/s/.+?</a>)', webpage, 'video data'))
        player_type = self._search_regex(r'playerType\s*=\s*[\'"]([^\'"]+)', webpage, 'player type', default=None)
        release_url = video_data['rel']
        title = video_data['data-title']
        rating = video_data.get('data-rating')
        query = {
            'mbr': 'true',
        }
        if player_type == 'movies':
            query.update({
                'manifest': 'm3u',
            })
        else:
            query.update({
                'switch': 'http',
            })
        if video_data.get('data-req-auth') == '1':
            resource = self._get_mvpd_resource(
                video_data['data-channel'], title,
                video_data.get('data-guid'), rating)
            query['auth'] = self._extract_mvpd_auth(url, video_id, 'fx', resource)
        return {
            '_type': 'url_transparent',
            'id': video_id,
            'title': title,
            'url': smuggle_url(update_url_query(release_url, query), {'force_smil_url': True}),
            'thumbnail': video_data.get('data-large-thumb'),
            'age_limit': parse_age_limit(rating),
            'ie_key': 'ThePlatform',
        }
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@@ -72,6 +72,8 @@ from .kaltura import KalturaIE
 from .eagleplatform import EaglePlatformIE
 from .facebook import FacebookIE
 from .soundcloud import SoundcloudIE
 from .vbox7 import Vbox7IE
 from .dbtv import DBTVIE
 class GenericIE(InfoExtractor):
@@ -102,7 +104,8 @@ class GenericIE(InfoExtractor):
            },
            'expected_warnings': [
                'URL could be a direct video link, returning it as such.'
-            ]
+            ],
            'skip': 'URL invalid',
        },
        # Direct download with broken HEAD
        {
@@ -266,7 +269,8 @@ class GenericIE(InfoExtractor):
            'params': {
                # m3u8 downloads
                'skip_download': True,
-            }
+            },
            'skip': 'video gone',
        },
        # m3u8 served with Content-Type: text/plain
        {
@@ -281,7 +285,8 @@ class GenericIE(InfoExtractor):
            'params': {
                # m3u8 downloads
                'skip_download': True,
-            }
+            },
            'skip': 'video gone',
        },
        # google redirect
        {
@@ -366,6 +371,7 @@ class GenericIE(InfoExtractor):
                'description': 'Mississauga resident David Farmer is still out of power as a result of the ice storm a month ago. To keep the house warm, Farmer cuts wood from his property for a wood burning stove downstairs.',
            },
            'add_ie': ['BrightcoveLegacy'],
            'skip': 'video gone',
        },
        {
            'url': 'http://www.championat.com/video/football/v/87/87499.html',
@@ -419,6 +425,7 @@ class GenericIE(InfoExtractor):
            'params': {
                'skip_download': True,
            },
            'skip': 'movie expired',
        },
        # embed.ly video
        {
@@ -446,6 +453,8 @@ class GenericIE(InfoExtractor):
                'title': 'Between Two Ferns with Zach Galifianakis: President Barack Obama',
                'description': 'Episode 18: President Barack Obama sits down with Zach Galifianakis for his most memorable interview yet.',
            },
            # HEAD requests lead to endless 301, while GET is OK
            'expected_warnings': ['301'],
        },
        # RUTV embed
        {
@@ -520,6 +529,9 @@ class GenericIE(InfoExtractor):
                'title': '[NSFL] [FM15] which pumiscer was this ( vid ) ( alfa as fuck srx )',
            },
            'playlist_mincount': 7,
            # This forum does not allow <iframe> syntaxes anymore
            # Now HTML tags are displayed as-is
            'skip': 'No videos on this page',
        },
        # Embedded TED video
        {
@@ -568,7 +580,8 @@ class GenericIE(InfoExtractor):
            },
            'params': {
                'skip_download': 'Requires rtmpdump'
-            }
+            },
            'skip': 'video gone',
        },
        # francetv embed
        {
@@ -1373,6 +1386,27 @@ class GenericIE(InfoExtractor):
            },
            'add_ie': [ArkenaIE.ie_key()],
        },
        {
            'url': 'http://nova.bg/news/view/2016/08/16/156543/%D0%BD%D0%B0-%D0%BA%D0%BE%D1%81%D1%8A%D0%BC-%D0%BE%D1%82-%D0%B2%D0%B7%D1%80%D0%B8%D0%B2-%D0%BE%D1%82%D1%86%D0%B5%D0%BF%D0%B8%D1%85%D0%B0-%D1%86%D1%8F%D0%BB-%D0%BA%D0%B2%D0%B0%D1%80%D1%82%D0%B0%D0%BB-%D0%B7%D0%B0%D1%80%D0%B0%D0%B4%D0%B8-%D0%B8%D0%B7%D1%82%D0%B8%D1%87%D0%B0%D0%BD%D0%B5-%D0%BD%D0%B0-%D0%B3%D0%B0%D0%B7-%D0%B2-%D0%BF%D0%BB%D0%BE%D0%B2%D0%B4%D0%B8%D0%B2/',
            'info_dict': {
                'id': '1c7141f46c',
                'ext': 'mp4',
                'title': 'НА КОСЪМ ОТ ВЗРИВ: Изтичане на газ на бензиностанция в Пловдив',
            },
            'params': {
                'skip_download': True,
            },
            'add_ie': [Vbox7IE.ie_key()],
        },
        {
            # DBTV embeds
            'url': 'http://www.dagbladet.no/2016/02/23/nyheter/nordlys/ski/troms/ver/43254897/',
            'info_dict': {
                'id': '43254897',
                'title': 'Etter ett års planlegging, klaffet endelig alt: - Jeg måtte ta en liten dans',
            },
            'playlist_mincount': 3,
        },
        # {
        #     # TODO: find another test
        #     # http://schema.org/VideoObject
@@ -2209,11 +2243,11 @@ class GenericIE(InfoExtractor):
        # Look for VODPlatform embeds
        mobj = re.search(
-            r'<iframe[^>]+src=[\'"]((?:https?:)?//(?:www\.)?vod-platform\.net/embed/[^/?#]+)',
+            r'<iframe[^>]+src=(["\'])(?P<url>(?:https?:)?//(?:www\.)?vod-platform\.net/[eE]mbed/.+?)\1',
            webpage)
        if mobj is not None:
            return self.url_result(
-                self._proto_relative_url(unescapeHTML(mobj.group(1))), 'VODPlatform')
+                self._proto_relative_url(unescapeHTML(mobj.group('url'))), 'VODPlatform')
        # Look for Instagram embeds
        instagram_embed_url = InstagramIE._extract_embed_url(webpage)
@@ -2239,10 +2273,20 @@ class GenericIE(InfoExtractor):
                'uploader': video_uploader,
            }
        # Look for VBOX7 embeds
        vbox7_url = Vbox7IE._extract_url(webpage)
        if vbox7_url:
            return self.url_result(vbox7_url, Vbox7IE.ie_key())
        # Look for DBTV embeds
        dbtv_urls = DBTVIE._extract_urls(webpage)
        if dbtv_urls:
            return _playlist_from_matches(dbtv_urls, ie=DBTVIE.ie_key())
        # Looking for http://schema.org/VideoObject
        json_ld = self._search_json_ld(
-            webpage, video_id, fatal=False, expected_type='VideoObject')
+            webpage, video_id, default={}, expected_type='VideoObject')
-        if json_ld and json_ld.get('url'):
+        if json_ld.get('url'):
            info_dict.update({
                'title': video_title or info_dict['title'],
                'description': video_description,
--- a/youtube_dl/extractor/glide.py
+++ b/youtube_dl/extractor/glide.py
@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import unified_strdate
 class GlideIE(InfoExtractor):
@@ -14,10 +13,8 @@ class GlideIE(InfoExtractor):
        'info_dict': {
            'id': 'UZF8zlmuQbe4mr+7dCiQ0w==',
            'ext': 'mp4',
-            'title': 'Damon Timm\'s Glide message',
+            'title': "Damon's Glide message",
            'thumbnail': 're:^https?://.*?\.cloudfront\.net/.*\.jpg$',
            'uploader': 'Damon Timm',
            'upload_date': '20140919',
        }
    }
@@ -27,7 +24,8 @@ class GlideIE(InfoExtractor):
        webpage = self._download_webpage(url, video_id)
        title = self._html_search_regex(
-            r'<title>(.+?)</title>', webpage, 'title')
+            r'<title>(.+?)</title>', webpage,
            'title', default=None) or self._og_search_title(webpage)
        video_url = self._proto_relative_url(self._search_regex(
            r'<source[^>]+src=(["\'])(?P<url>.+?)\1',
            webpage, 'video URL', default=None,
@@ -36,18 +34,10 @@ class GlideIE(InfoExtractor):
            r'<img[^>]+id=["\']video-thumbnail["\'][^>]+src=(["\'])(?P<url>.+?)\1',
            webpage, 'thumbnail url', default=None,
            group='url')) or self._og_search_thumbnail(webpage)
        uploader = self._search_regex(
            r'<div[^>]+class=["\']info-name["\'][^>]*>([^<]+)',
            webpage, 'uploader', fatal=False)
        upload_date = unified_strdate(self._search_regex(
            r'<div[^>]+class="info-date"[^>]*>([^<]+)',
            webpage, 'upload date', fatal=False))
        return {
            'id': video_id,
            'title': title,
            'url': video_url,
            'thumbnail': thumbnail,
            'uploader': uploader,
            'upload_date': upload_date,
        }
--- a/youtube_dl/extractor/globo.py
+++ b/youtube_dl/extractor/globo.py
@@ -396,12 +396,12 @@ class GloboIE(InfoExtractor):
 class GloboArticleIE(InfoExtractor):
-    _VALID_URL = 'https?://.+?\.globo\.com/(?:[^/]+/)*(?P<id>[^/]+)\.html'
+    _VALID_URL = 'https?://.+?\.globo\.com/(?:[^/]+/)*(?P<id>[^/]+)(?:\.html)?'
    _VIDEOID_REGEXES = [
        r'\bdata-video-id=["\'](\d{7,})',
        r'\bdata-player-videosids=["\'](\d{7,})',
-        r'\bvideosIDs\s*:\s*["\'](\d{7,})',
+        r'\bvideosIDs\s*:\s*["\']?(\d{7,})',
        r'\bdata-id=["\'](\d{7,})',
        r'<div[^>]+\bid=["\'](\d{7,})',
    ]
@@ -423,6 +423,9 @@ class GloboArticleIE(InfoExtractor):
    }, {
        'url': 'http://gshow.globo.com/programas/tv-xuxa/O-Programa/noticia/2014/01/xuxa-e-junno-namoram-muuuito-em-luau-de-zeze-di-camargo-e-luciano.html',
        'only_matching': True,
    }, {
        'url': 'http://oglobo.globo.com/rio/a-amizade-entre-um-entregador-de-farmacia-um-piano-19946271',
        'only_matching': True,
    }]
    @classmethod
--- a/youtube_dl/extractor/go.py
+++ b/youtube_dl/extractor/go.py
@@ -0,0 +1,101 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    determine_ext,
    parse_age_limit,
 )
 class GoIE(InfoExtractor):
    _BRANDS = {
        'abc': '001',
        'freeform': '002',
        'watchdisneychannel': '004',
        'watchdisneyjunior': '008',
        'watchdisneyxd': '009',
    }
    _VALID_URL = r'https?://(?:(?P<sub_domain>%s)\.)?go\.com/.*?vdka(?P<id>\w+)' % '|'.join(_BRANDS.keys())
    _TESTS = [{
        'url': 'http://abc.go.com/shows/castle/video/most-recent/vdka0_g86w5onx',
        'info_dict': {
            'id': '0_g86w5onx',
            'ext': 'mp4',
            'title': 'Sneak Peek: Language Arts',
            'description': 'md5:7dcdab3b2d17e5217c953256af964e9c',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://abc.go.com/shows/after-paradise/video/most-recent/vdka3335601',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        sub_domain, video_id = re.match(self._VALID_URL, url).groups()
        video_data = self._download_json(
            'http://api.contents.watchabc.go.com/vp2/ws/contents/3000/videos/%s/001/-1/-1/-1/%s/-1/-1.json' % (self._BRANDS[sub_domain], video_id),
            video_id)['video'][0]
        title = video_data['title']
        formats = []
        for asset in video_data.get('assets', {}).get('asset', []):
            asset_url = asset.get('value')
            if not asset_url:
                continue
            format_id = asset.get('format')
            ext = determine_ext(asset_url)
            if ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    asset_url, video_id, 'mp4', m3u8_id=format_id or 'hls', fatal=False))
            else:
                formats.append({
                    'format_id': format_id,
                    'url': asset_url,
                    'ext': ext,
                })
        self._sort_formats(formats)
        subtitles = {}
        for cc in video_data.get('closedcaption', {}).get('src', []):
            cc_url = cc.get('value')
            if not cc_url:
                continue
            ext = determine_ext(cc_url)
            if ext == 'xml':
                ext = 'ttml'
            subtitles.setdefault(cc.get('lang'), []).append({
                'url': cc_url,
                'ext': ext,
            })
        thumbnails = []
        for thumbnail in video_data.get('thumbnails', {}).get('thumbnail', []):
            thumbnail_url = thumbnail.get('value')
            if not thumbnail_url:
                continue
            thumbnails.append({
                'url': thumbnail_url,
                'width': int_or_none(thumbnail.get('width')),
                'height': int_or_none(thumbnail.get('height')),
            })
        return {
            'id': video_id,
            'title': title,
            'description': video_data.get('longdescription') or video_data.get('description'),
            'duration': int_or_none(video_data.get('duration', {}).get('value'), 1000),
            'age_limit': parse_age_limit(video_data.get('tvrating', {}).get('rating')),
            'episode_number': int_or_none(video_data.get('episodenumber')),
            'series': video_data.get('show', {}).get('title'),
            'season_number': int_or_none(video_data.get('season', {}).get('num')),
            'thumbnails': thumbnails,
            'formats': formats,
            'subtitles': subtitles,
        }
--- a/youtube_dl/extractor/goldenmoustache.py
+++ b/youtube_dl/extractor/goldenmoustache.py
@@ -1,48 +0,0 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 class GoldenMoustacheIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?goldenmoustache\.com/(?P<display_id>[\w-]+)-(?P<id>\d+)'
    _TESTS = [{
        'url': 'http://www.goldenmoustache.com/suricate-le-poker-3700/',
        'md5': '0f904432fa07da5054d6c8beb5efb51a',
        'info_dict': {
            'id': '3700',
            'ext': 'mp4',
            'title': 'Suricate - Le Poker',
            'description': 'md5:3d1f242f44f8c8cb0a106f1fd08e5dc9',
            'thumbnail': 're:^https?://.*\.jpg$',
        }
    }, {
        'url': 'http://www.goldenmoustache.com/le-lab-tout-effacer-mc-fly-et-carlito-55249/',
        'md5': '27f0c50fb4dd5f01dc9082fc67cd5700',
        'info_dict': {
            'id': '55249',
            'ext': 'mp4',
            'title': 'Le LAB - Tout Effacer (Mc Fly et Carlito)',
            'description': 'md5:9b7fbf11023fb2250bd4b185e3de3b2a',
            'thumbnail': 're:^https?://.*\.(?:png|jpg)$',
        }
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        video_url = self._html_search_regex(
            r'data-src-type="mp4" data-src="([^"]+)"', webpage, 'video URL')
        title = self._html_search_regex(
            r'<title>(.*?)(?: - Golden Moustache)?</title>', webpage, 'title')
        thumbnail = self._og_search_thumbnail(webpage)
        description = self._og_search_description(webpage)
        return {
            'id': video_id,
            'url': video_url,
            'ext': 'mp4',
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
        }
--- a/youtube_dl/extractor/hgtv.py
+++ b/youtube_dl/extractor/hgtv.py
@@ -0,0 +1,79 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    js_to_json,
    smuggle_url,
 )
 class HGTVIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?hgtv\.ca/[^/]+/video/(?P<id>[^/]+)/video.html'
    _TEST = {
        'url': 'http://www.hgtv.ca/homefree/video/overnight-success/video.html?v=738081859718&p=1&s=da#video',
        'md5': '',
        'info_dict': {
            'id': 'aFH__I_5FBOX',
            'ext': 'mp4',
            'title': 'Overnight Success',
            'description': 'After weeks of hard work, high stakes, breakdowns and pep talks, the final 2 contestants compete to win the ultimate dream.',
            'uploader': 'SHWM-NEW',
            'timestamp': 1470320034,
            'upload_date': '20160804',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        embed_vars = self._parse_json(self._search_regex(
            r'(?s)embed_vars\s*=\s*({.*?});',
            webpage, 'embed vars'), display_id, js_to_json)
        return {
            '_type': 'url_transparent',
            'url': smuggle_url(
                'http://link.theplatform.com/s/dtjsEC/%s?mbr=true&manifest=m3u' % embed_vars['pid'], {
                    'force_smil_url': True
                }),
            'series': embed_vars.get('show'),
            'season_number': int_or_none(embed_vars.get('season')),
            'episode_number': int_or_none(embed_vars.get('episode')),
            'ie_key': 'ThePlatform',
        }
 class HGTVComShowIE(InfoExtractor):
    IE_NAME = 'hgtv.com:show'
    _VALID_URL = r'https?://(?:www\.)?hgtv\.com/shows/[^/]+/(?P<id>[^/?#&]+)'
    _TEST = {
        'url': 'http://www.hgtv.com/shows/flip-or-flop/flip-or-flop-full-episodes-videos',
        'info_dict': {
            'id': 'flip-or-flop-full-episodes-videos',
            'title': 'Flip or Flop Full Episodes',
        },
        'playlist_mincount': 15,
    }
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        config = self._parse_json(
            self._search_regex(
                r'(?s)data-module=["\']video["\'][^>]*>.*?<script[^>]+type=["\']text/x-config["\'][^>]*>(.+?)</script',
                webpage, 'video config'),
            display_id)['channels'][0]
        entries = [
            self.url_result(video['releaseUrl'])
            for video in config['videos'] if video.get('releaseUrl')]
        return self.playlist_result(
            entries, display_id, config.get('title'), config.get('description'))
--- a/youtube_dl/extractor/imdb.py
+++ b/youtube_dl/extractor/imdb.py
@@ -6,6 +6,7 @@ from .common import InfoExtractor
 from ..utils import (
    mimetype2ext,
    qualities,
    remove_end,
 )
@@ -19,7 +20,7 @@ class ImdbIE(InfoExtractor):
        'info_dict': {
            'id': '2524815897',
            'ext': 'mp4',
-            'title': 'Ice Age: Continental Drift Trailer (No. 2) - IMDb',
+            'title': 'Ice Age: Continental Drift Trailer (No. 2)',
            'description': 'md5:9061c2219254e5d14e03c25c98e96a81',
        }
    }, {
@@ -83,10 +84,10 @@ class ImdbIE(InfoExtractor):
        return {
            'id': video_id,
-            'title': self._og_search_title(webpage),
+            'title': remove_end(self._og_search_title(webpage), ' - IMDb'),
            'formats': formats,
            'description': descr,
-            'thumbnail': format_info['slate'],
+            'thumbnail': format_info.get('slate'),
        }
--- a/youtube_dl/extractor/imgur.py
+++ b/youtube_dl/extractor/imgur.py
@@ -50,12 +50,10 @@ class ImgurIE(InfoExtractor):
        webpage = self._download_webpage(
            compat_urlparse.urljoin(url, video_id), video_id)
-        width = int_or_none(self._search_regex(
+        width = int_or_none(self._og_search_property(
-            r'<param name="width" value="([0-9]+)"',
+            'video:width', webpage, default=None))
-            webpage, 'width', fatal=False))
+        height = int_or_none(self._og_search_property(
-        height = int_or_none(self._search_regex(
+            'video:height', webpage, default=None))
            r'<param name="height" value="([0-9]+)"',
            webpage, 'height', fatal=False))
        video_elements = self._search_regex(
            r'(?s)<div class="video-elements">(.*?)</div>',
--- a/youtube_dl/extractor/ivi.py
+++ b/youtube_dl/extractor/ivi.py
@@ -1,4 +1,4 @@
-# encoding: utf-8
+# coding: utf-8
 from __future__ import unicode_literals
 import re
@@ -8,7 +8,7 @@ from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
-    sanitized_Request,
+    qualities,
 )
@@ -49,11 +49,27 @@ class IviIE(InfoExtractor):
                'thumbnail': 're:^https?://.*\.jpg$',
            },
            'skip': 'Only works from Russia',
        },
        {
            # with MP4-HD720 format
            'url': 'http://www.ivi.ru/watch/146500',
            'md5': 'd63d35cdbfa1ea61a5eafec7cc523e1e',
            'info_dict': {
                'id': '146500',
                'ext': 'mp4',
                'title': 'Кукла',
                'description': 'md5:ffca9372399976a2d260a407cc74cce6',
                'duration': 5599,
                'thumbnail': 're:^https?://.*\.jpg$',
            },
            'skip': 'Only works from Russia',
        }
    ]
    # Sorted by quality
-    _KNOWN_FORMATS = ['MP4-low-mobile', 'MP4-mobile', 'FLV-lo', 'MP4-lo', 'FLV-hi', 'MP4-hi', 'MP4-SHQ']
+    _KNOWN_FORMATS = (
        'MP4-low-mobile', 'MP4-mobile', 'FLV-lo', 'MP4-lo', 'FLV-hi', 'MP4-hi',
        'MP4-SHQ', 'MP4-HD720', 'MP4-HD1080')
    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -69,10 +85,9 @@ class IviIE(InfoExtractor):
            ]
        }
        request = sanitized_Request(
            'http://api.digitalaccess.ru/api/json/', json.dumps(data))
        video_json = self._download_json(
-            request, video_id, 'Downloading video JSON')
+            'http://api.digitalaccess.ru/api/json/', video_id,
            'Downloading video JSON', data=json.dumps(data))
        if 'error' in video_json:
            error = video_json['error']
@@ -84,11 +99,13 @@ class IviIE(InfoExtractor):
        result = video_json['result']
        quality = qualities(self._KNOWN_FORMATS)
        formats = [{
            'url': x['url'],
-            'format_id': x['content_format'],
+            'format_id': x.get('content_format'),
-            'preference': self._KNOWN_FORMATS.index(x['content_format']),
+            'quality': quality(x.get('content_format')),
-        } for x in result['files'] if x['content_format'] in self._KNOWN_FORMATS]
+        } for x in result['files'] if x.get('url')]
        self._sort_formats(formats)
@@ -115,7 +132,7 @@ class IviIE(InfoExtractor):
            webpage, 'season number', default=None))
        episode_number = int_or_none(self._search_regex(
-            r'<meta[^>]+itemprop="episode"[^>]*>\s*<meta[^>]+itemprop="episodeNumber"[^>]+content="(\d+)',
+            r'[^>]+itemprop="episode"[^>]*>\s*<meta[^>]+itemprop="episodeNumber"[^>]+content="(\d+)',
            webpage, 'episode number', default=None))
        description = self._og_search_description(webpage, default=None) or self._html_search_meta(
--- a/youtube_dl/extractor/jwplatform.py
+++ b/youtube_dl/extractor/jwplatform.py
@@ -30,7 +30,7 @@ class JWPlatformBaseIE(InfoExtractor):
        return self._parse_jwplayer_data(
            jwplayer_data, video_id, *args, **kwargs)
-    def _parse_jwplayer_data(self, jwplayer_data, video_id, require_title=True, m3u8_id=None, rtmp_params=None, base_url=None):
+    def _parse_jwplayer_data(self, jwplayer_data, video_id=None, require_title=True, m3u8_id=None, rtmp_params=None, base_url=None):
        # JWPlayer backward compatibility: flattened playlists
        # https://github.com/jwplayer/jwplayer/blob/v7.4.3/src/js/api/config.js#L81-L96
        if 'playlist' not in jwplayer_data:
@@ -43,6 +43,8 @@ class JWPlatformBaseIE(InfoExtractor):
            if 'sources' not in video_data:
                video_data['sources'] = [video_data]
            this_video_id = video_id or video_data['mediaid']
            formats = []
            for source in video_data['sources']:
                source_url = self._proto_relative_url(source['file'])
@@ -52,7 +54,7 @@ class JWPlatformBaseIE(InfoExtractor):
                ext = mimetype2ext(source_type) or determine_ext(source_url)
                if source_type == 'hls' or ext == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
-                        source_url, video_id, 'mp4', 'm3u8_native', m3u8_id=m3u8_id, fatal=False))
+                        source_url, this_video_id, 'mp4', 'm3u8_native', m3u8_id=m3u8_id, fatal=False))
                # https://github.com/jwplayer/jwplayer/blob/master/src/js/providers/default.js#L67
                elif source_type.startswith('audio') or ext in ('oga', 'aac', 'mp3', 'mpeg', 'vorbis'):
                    formats.append({
@@ -68,7 +70,7 @@ class JWPlatformBaseIE(InfoExtractor):
                        'ext': ext,
                    }
                    if source_url.startswith('rtmp'):
-                        a_format['ext'] = 'flv',
+                        a_format['ext'] = 'flv'
                        # See com/longtailvideo/jwplayer/media/RTMPMediaProvider.as
                        # of jwplayer.flash.swf
@@ -95,7 +97,7 @@ class JWPlatformBaseIE(InfoExtractor):
                        })
            entries.append({
-                'id': video_id,
+                'id': this_video_id,
                'title': video_data['title'] if require_title else video_data.get('title'),
                'description': video_data.get('description'),
                'thumbnail': self._proto_relative_url(video_data.get('image')),
--- a/youtube_dl/extractor/kaltura.py
+++ b/youtube_dl/extractor/kaltura.py
@@ -36,6 +36,12 @@ class KalturaIE(InfoExtractor):
                '''
    _SERVICE_URL = 'http://cdnapi.kaltura.com'
    _SERVICE_BASE = '/api_v3/index.php'
    # See https://github.com/kaltura/server/blob/master/plugins/content/caption/base/lib/model/enums/CaptionType.php
    _CAPTION_TYPES = {
        1: 'srt',
        2: 'ttml',
        3: 'vtt',
    }
    _TESTS = [
        {
            'url': 'kaltura:269692:1_1jc2y3e4',
@@ -67,6 +73,27 @@ class KalturaIE(InfoExtractor):
            # video with subtitles
            'url': 'kaltura:111032:1_cw786r8q',
            'only_matching': True,
        },
        {
            # video with ttml subtitles (no fileExt)
            'url': 'kaltura:1926081:0_l5ye1133',
            'info_dict': {
                'id': '0_l5ye1133',
                'ext': 'mp4',
                'title': 'What Can You Do With Python?',
                'upload_date': '20160221',
                'uploader_id': 'stork',
                'thumbnail': 're:^https?://.*/thumbnail/.*',
                'timestamp': int,
                'subtitles': {
                    'en': [{
                        'ext': 'ttml',
                    }],
                },
            },
            'params': {
                'skip_download': True,
            },
        }
    ]
@@ -122,18 +149,6 @@ class KalturaIE(InfoExtractor):
        return data
    def _get_kaltura_signature(self, video_id, partner_id, service_url=None):
        actions = [{
            'apiVersion': '3.1',
            'expiry': 86400,
            'format': 1,
            'service': 'session',
            'action': 'startWidgetSession',
            'widgetId': '_%s' % partner_id,
        }]
        return self._kaltura_api_call(
            video_id, actions, service_url, note='Downloading Kaltura signature')['ks']
    def _get_video_info(self, video_id, partner_id, service_url=None):
        actions = [
            {
@@ -208,6 +223,17 @@ class KalturaIE(InfoExtractor):
                    reference_id)['entryResult']
                info, flavor_assets = entry_data['meta'], entry_data['contextData']['flavorAssets']
                entry_id = info['id']
                # Unfortunately, data returned in kalturaIframePackageData lacks
                # captions so we will try requesting the complete data using
                # regular approach since we now know the entry_id
                try:
                    _, info, flavor_assets, captions = self._get_video_info(
                        entry_id, partner_id)
                except ExtractorError:
                    # Regular scenario failed but we already have everything
                    # extracted apart from captions and can process at least
                    # with this
                    pass
            else:
                raise ExtractorError('Invalid URL', expected=True)
            ks = params.get('flashvars[ks]', [None])[0]
@@ -265,9 +291,12 @@ class KalturaIE(InfoExtractor):
                # Continue if caption is not ready
                if f.get('status') != 2:
                    continue
                if not caption.get('id'):
                    continue
                caption_format = int_or_none(caption.get('format'))
                subtitles.setdefault(caption.get('languageCode') or caption.get('language'), []).append({
                    'url': '%s/api_v3/service/caption_captionasset/action/serve/captionAssetId/%s' % (self._SERVICE_URL, caption['id']),
-                    'ext': caption.get('fileExt'),
+                    'ext': caption.get('fileExt') or self._CAPTION_TYPES.get(caption_format) or 'ttml',
                })
        return {
--- a/youtube_dl/extractor/keezmovies.py
+++ b/youtube_dl/extractor/keezmovies.py
@@ -3,64 +3,126 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..aes import aes_decrypt_text
 from ..compat import (
    compat_str,
    compat_urllib_parse_unquote,
 )
 from ..utils import (
-    sanitized_Request,
+    determine_ext,
-    url_basename,
+    ExtractorError,
    int_or_none,
    str_to_int,
    strip_or_none,
 )
 class KeezMoviesIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?keezmovies\.com/video/.+?(?P<id>[0-9]+)(?:[/?&]|$)'
+    _VALID_URL = r'https?://(?:www\.)?keezmovies\.com/video/(?:(?P<display_id>[^/]+)-)?(?P<id>\d+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://www.keezmovies.com/video/petite-asian-lady-mai-playing-in-bathtub-1214711',
        'md5': '1c1e75d22ffa53320f45eeb07bc4cdc0',
        'info_dict': {
            'id': '1214711',
            'display_id': 'petite-asian-lady-mai-playing-in-bathtub',
            'ext': 'mp4',
            'title': 'Petite Asian Lady Mai Playing In Bathtub',
            'age_limit': 18,
            'thumbnail': 're:^https?://.*\.jpg$',
            'view_count': int,
            'age_limit': 18,
        }
-    }
+    }, {
        'url': 'http://www.keezmovies.com/video/1214711',
        'only_matching': True,
    }]
-    def _real_extract(self, url):
+    def _extract_info(self, url):
-        video_id = self._match_id(url)
+        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('id')
        display_id = (mobj.group('display_id')
                      if 'display_id' in mobj.groupdict()
                      else None) or mobj.group('id')
-        req = sanitized_Request(url)
+        webpage = self._download_webpage(
-        req.add_header('Cookie', 'age_verified=1')
+            url, display_id, headers={'Cookie': 'age_verified=1'})
        webpage = self._download_webpage(req, video_id)
        # embedded video
        mobj = re.search(r'href="([^"]+)"></iframe>', webpage)
        if mobj:
            embedded_url = mobj.group(1)
            return self.url_result(embedded_url)
        video_title = self._html_search_regex(
            r'<h1 [^>]*>([^<]+)', webpage, 'title')
        flashvars = self._parse_json(self._search_regex(
            r'var\s+flashvars\s*=\s*([^;]+);', webpage, 'flashvars'), video_id)
        formats = []
-        for height in (180, 240, 480):
+        format_urls = set()
            if flashvars.get('quality_%dp' % height):
                video_url = flashvars['quality_%dp' % height]
                a_format = {
                    'url': video_url,
                    'height': height,
                    'format_id': '%dp' % height,
                }
                filename_parts = url_basename(video_url).split('_')
                if len(filename_parts) >= 2 and re.match(r'\d+[Kk]', filename_parts[1]):
                    a_format['tbr'] = int(filename_parts[1][:-1])
                formats.append(a_format)
-        age_limit = self._rta_search(webpage)
+        title = None
        thumbnail = None
        duration = None
        encrypted = False
-        return {
+        def extract_format(format_url, height=None):
            if not isinstance(format_url, compat_str) or not format_url.startswith('http'):
                return
            if format_url in format_urls:
                return
            format_urls.add(format_url)
            tbr = int_or_none(self._search_regex(
                r'[/_](\d+)[kK][/_]', format_url, 'tbr', default=None))
            if not height:
                height = int_or_none(self._search_regex(
                    r'[/_](\d+)[pP][/_]', format_url, 'height', default=None))
            if encrypted:
                format_url = aes_decrypt_text(
                    video_url, title, 32).decode('utf-8')
            formats.append({
                'url': format_url,
                'format_id': '%dp' % height if height else None,
                'height': height,
                'tbr': tbr,
            })
        flashvars = self._parse_json(
            self._search_regex(
                r'flashvars\s*=\s*({.+?});', webpage,
                'flashvars', default='{}'),
            display_id, fatal=False)
        if flashvars:
            title = flashvars.get('video_title')
            thumbnail = flashvars.get('image_url')
            duration = int_or_none(flashvars.get('video_duration'))
            encrypted = flashvars.get('encrypted') is True
            for key, value in flashvars.items():
                mobj = re.search(r'quality_(\d+)[pP]', key)
                if mobj:
                    extract_format(value, int(mobj.group(1)))
            video_url = flashvars.get('video_url')
            if video_url and determine_ext(video_url, None):
                extract_format(video_url)
        video_url = self._html_search_regex(
            r'flashvars\.video_url\s*=\s*(["\'])(?P<url>http.+?)\1',
            webpage, 'video url', default=None, group='url')
        if video_url:
            extract_format(compat_urllib_parse_unquote(video_url))
        if not formats:
            if 'title="This video is no longer available"' in webpage:
                raise ExtractorError(
                    'Video %s is no longer available' % video_id, expected=True)
        self._sort_formats(formats)
        if not title:
            title = self._html_search_regex(
                r'<h1[^>]*>([^<]+)', webpage, 'title')
        return webpage, {
            'id': video_id,
-            'title': video_title,
+            'display_id': display_id,
            'title': strip_or_none(title),
            'thumbnail': thumbnail,
            'duration': duration,
            'age_limit': 18,
            'formats': formats,
            'age_limit': age_limit,
            'thumbnail': flashvars.get('image_url')
        }
    def _real_extract(self, url):
        webpage, info = self._extract_info(url)
        info['view_count'] = str_to_int(self._search_regex(
            r'<b>([\d,.]+)</b> Views?', webpage, 'view count', fatal=False))
        return info
--- a/youtube_dl/extractor/kickstarter.py
+++ b/youtube_dl/extractor/kickstarter.py
@@ -37,7 +37,6 @@ class KickStarterIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'Power Drive 2000',
        },
        'expected_warnings': ['OpenGraph description'],
    }]
    def _real_extract(self, url):
@@ -67,6 +66,6 @@ class KickStarterIE(InfoExtractor):
            'id': video_id,
            'url': video_url,
            'title': title,
-            'description': self._og_search_description(webpage),
+            'description': self._og_search_description(webpage, default=None),
            'thumbnail': thumbnail,
        }
--- a/youtube_dl/extractor/kusi.py
+++ b/youtube_dl/extractor/kusi.py
@@ -18,31 +18,20 @@ from ..utils import (
 class KUSIIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?kusi\.com/(?P<path>story/.+|video\?clipId=(?P<clipId>\d+))'
    _TESTS = [{
-        'url': 'http://www.kusi.com/story/31183873/turko-files-case-closed-put-on-hold',
+        'url': 'http://www.kusi.com/story/32849881/turko-files-refused-to-help-it-aint-right',
-        'md5': 'f926e7684294cf8cb7bdf8858e1b3988',
+        'md5': '4e76ce8e53660ce9697d06c0ba6fc47d',
        'info_dict': {
-            'id': '12203019',
+            'id': '12689020',
            'ext': 'mp4',
-            'title': 'Turko Files: Case Closed! & Put On Hold!',
+            'title': "Turko Files: Refused to Help, It Ain't Right!",
-            'duration': 231.0,
+            'duration': 223.586,
-            'upload_date': '20160210',
+            'upload_date': '20160826',
-            'timestamp': 1455087571,
+            'timestamp': 1472233118,
            'thumbnail': 're:^https?://.*\.jpg$'
        },
    }, {
        'url': 'http://kusi.com/video?clipId=12203019',
-        'info_dict': {
+        'only_matching': True,
            'id': '12203019',
            'ext': 'mp4',
            'title': 'Turko Files: Case Closed! & Put On Hold!',
            'duration': 231.0,
            'upload_date': '20160210',
            'timestamp': 1455087571,
            'thumbnail': 're:^https?://.*\.jpg$'
        },
        'params': {
            'skip_download': True,  # Same as previous one
        },
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/kuwo.py
+++ b/youtube_dl/extractor/kuwo.py
@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
    get_element_by_id,
    clean_html,
@@ -242,8 +243,9 @@ class KuwoSingerIE(InfoExtractor):
                query={'artistId': artist_id, 'pn': page_num, 'rn': self.PAGE_SIZE})
            return [
-                self.url_result(song_url, 'Kuwo') for song_url in re.findall(
+                self.url_result(compat_urlparse.urljoin(url, song_url), 'Kuwo')
-                    r'<div[^>]+class="name"><a[^>]+href="(http://www\.kuwo\.cn/yinyue/\d+)',
+                for song_url in re.findall(
                    r'<div[^>]+class="name"><a[^>]+href="(/yinyue/\d+)',
                    webpage)
            ]
--- a/youtube_dl/extractor/lifenews.py
+++ b/youtube_dl/extractor/lifenews.py
@@ -4,7 +4,10 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..compat import compat_urlparse
+from ..compat import (
    compat_str,
    compat_urlparse,
 )
 from ..utils import (
    determine_ext,
    ExtractorError,
@@ -96,7 +99,7 @@ class LifeNewsIE(InfoExtractor):
            r'<video[^>]+><source[^>]+src=["\'](.+?)["\']', webpage)
        iframe_links = re.findall(
-            r'<iframe[^>]+src=["\']((?:https?:)?//embed\.life\.ru/embed/.+?)["\']',
+            r'<iframe[^>]+src=["\']((?:https?:)?//embed\.life\.ru/(?:embed|video)/.+?)["\']',
            webpage)
        if not video_urls and not iframe_links:
@@ -164,9 +167,9 @@ class LifeNewsIE(InfoExtractor):
 class LifeEmbedIE(InfoExtractor):
    IE_NAME = 'life:embed'
-    _VALID_URL = r'https?://embed\.life\.ru/embed/(?P<id>[\da-f]{32})'
+    _VALID_URL = r'https?://embed\.life\.ru/(?:embed|video)/(?P<id>[\da-f]{32})'
-    _TEST = {
+    _TESTS = [{
        'url': 'http://embed.life.ru/embed/e50c2dec2867350528e2574c899b8291',
        'md5': 'b889715c9e49cb1981281d0e5458fbbe',
        'info_dict': {
@@ -175,30 +178,57 @@ class LifeEmbedIE(InfoExtractor):
            'title': 'e50c2dec2867350528e2574c899b8291',
            'thumbnail': 're:http://.*\.jpg',
        }
-    }
+    }, {
        # with 1080p
        'url': 'https://embed.life.ru/video/e50c2dec2867350528e2574c899b8291',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        thumbnail = None
        formats = []
-        for video_url in re.findall(r'"file"\s*:\s*"([^"]+)', webpage):
+
-            video_url = compat_urlparse.urljoin(url, video_url)
+        def extract_m3u8(manifest_url):
-            ext = determine_ext(video_url)
+            formats.extend(self._extract_m3u8_formats(
-            if ext == 'm3u8':
+                manifest_url, video_id, 'mp4',
-                formats.extend(self._extract_m3u8_formats(
+                entry_protocol='m3u8_native', m3u8_id='m3u8'))
-                    video_url, video_id, 'mp4',
+
-                    entry_protocol='m3u8_native', m3u8_id='m3u8'))
+        def extract_original(original_url):
-            else:
+            formats.append({
-                formats.append({
+                'url': original_url,
-                    'url': video_url,
+                'format_id': determine_ext(original_url, None),
-                    'format_id': ext,
+                'preference': 1,
-                    'preference': 1,
+            })
-                })
+
        playlist = self._parse_json(
            self._search_regex(
                r'options\s*=\s*({.+?});', webpage, 'options', default='{}'),
            video_id).get('playlist', {})
        if playlist:
            master = playlist.get('master')
            if isinstance(master, compat_str) and determine_ext(master) == 'm3u8':
                extract_m3u8(compat_urlparse.urljoin(url, master))
            original = playlist.get('original')
            if isinstance(original, compat_str):
                extract_original(original)
            thumbnail = playlist.get('image')
        # Old rendition fallback
        if not formats:
            for video_url in re.findall(r'"file"\s*:\s*"([^"]+)', webpage):
                video_url = compat_urlparse.urljoin(url, video_url)
                if determine_ext(video_url) == 'm3u8':
                    extract_m3u8(video_url)
                else:
                    extract_original(video_url)
        self._sort_formats(formats)
-        thumbnail = self._search_regex(
+        thumbnail = thumbnail or self._search_regex(
            r'"image"\s*:\s*"([^"]+)', webpage, 'thumbnail', default=None)
        return {
--- a/youtube_dl/extractor/limelight.py
+++ b/youtube_dl/extractor/limelight.py
@@ -34,11 +34,12 @@ class LimelightBaseIE(InfoExtractor):
    def _extract_info(self, streams, mobile_urls, properties):
        video_id = properties['media_id']
        formats = []
-
+        urls = []
        for stream in streams:
            stream_url = stream.get('url')
-            if not stream_url or stream.get('drmProtected'):
+            if not stream_url or stream.get('drmProtected') or stream_url in urls:
                continue
            urls.append(stream_url)
            ext = determine_ext(stream_url)
            if ext == 'f4m':
                formats.extend(self._extract_f4m_formats(
@@ -58,9 +59,11 @@ class LimelightBaseIE(InfoExtractor):
                    format_id = 'rtmp'
                    if stream.get('videoBitRate'):
                        format_id += '-%d' % int_or_none(stream['videoBitRate'])
                    http_url = 'http://%s/%s' % (rtmp.group('host').replace('csl.', 'cpl.'), rtmp.group('playpath')[4:])
                    urls.append(http_url)
                    http_fmt = fmt.copy()
                    http_fmt.update({
-                        'url': 'http://%s/%s' % (rtmp.group('host').replace('csl.', 'cpl.'), rtmp.group('playpath')[4:]),
+                        'url': http_url,
                        'format_id': format_id.replace('rtmp', 'http'),
                    })
                    formats.append(http_fmt)
@@ -76,8 +79,9 @@ class LimelightBaseIE(InfoExtractor):
        for mobile_url in mobile_urls:
            media_url = mobile_url.get('mobileUrl')
            format_id = mobile_url.get('targetMediaPlatform')
-            if not media_url or format_id == 'Widevine':
+            if not media_url or format_id in ('Widevine', 'SmoothStreaming') or media_url in urls:
                continue
            urls.append(media_url)
            ext = determine_ext(media_url)
            if ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
--- a/youtube_dl/extractor/litv.py
+++ b/youtube_dl/extractor/litv.py
@@ -14,7 +14,7 @@ from ..utils import (
 class LiTVIE(InfoExtractor):
-    _VALID_URL = r'https?://www\.litv\.tv/vod/[^/]+/content\.do\?.*?\bid=(?P<id>[^&]+)'
+    _VALID_URL = r'https?://www\.litv\.tv/(?:vod|promo)/[^/]+/(?:content\.do)?\?.*?\b(?:content_)?id=(?P<id>[^&]+)'
    _URL_TEMPLATE = 'https://www.litv.tv/vod/%s/content.do?id=%s'
@@ -27,6 +27,7 @@ class LiTVIE(InfoExtractor):
        'playlist_count': 50,
    }, {
        'url': 'https://www.litv.tv/vod/drama/content.do?brc_id=root&id=VOD00041610&isUHEnabled=true&autoPlay=1',
        'md5': '969e343d9244778cb29acec608e53640',
        'info_dict': {
            'id': 'VOD00041610',
            'ext': 'mp4',
@@ -37,7 +38,16 @@ class LiTVIE(InfoExtractor):
        },
        'params': {
            'noplaylist': True,
-            'skip_download': True,  # m3u8 download
+        },
        'skip': 'Georestricted to Taiwan',
    }, {
        'url': 'https://www.litv.tv/promo/miyuezhuan/?content_id=VOD00044841&',
        'md5': '88322ea132f848d6e3e18b32a832b918',
        'info_dict': {
            'id': 'VOD00044841',
            'ext': 'mp4',
            'title': '芈月傳第1集　霸星芈月降世楚國',
            'description': '楚威王二年，太史令唐昧夜觀星象，發現霸星即將現世。王后得知霸星的預言後，想盡辦法不讓孩子順利出生，幸得莒姬相護化解危機。沒想到眾人期待下出生的霸星卻是位公主，楚威王對此失望至極。楚王后命人將女嬰丟棄河中，居然奇蹟似的被少司命像攔下，楚威王認為此女非同凡響，為她取名芈月。',
        },
        'skip': 'Georestricted to Taiwan',
    }]
@@ -92,13 +102,18 @@ class LiTVIE(InfoExtractor):
        # endpoint gives the same result as the data embedded in the webpage.
        # If georestricted, there are no embedded data, so an extra request is
        # necessary to get the error code
        if 'assetId' not in view_data:
            view_data = self._download_json(
                'https://www.litv.tv/vod/ajax/getProgramInfo', video_id,
                query={'contentId': video_id},
                headers={'Accept': 'application/json'})
        video_data = self._parse_json(self._search_regex(
            r'uiHlsUrl\s*=\s*testBackendData\(([^;]+)\);',
            webpage, 'video data', default='{}'), video_id)
        if not video_data:
            payload = {
                'assetId': view_data['assetId'],
-                'watchDevices': vod_data['watchDevices'],
+                'watchDevices': view_data['watchDevices'],
                'contentType': view_data['contentType'],
            }
            video_data = self._download_json(
@@ -115,7 +130,8 @@ class LiTVIE(InfoExtractor):
            raise ExtractorError('Unexpected result from %s' % self.IE_NAME)
        formats = self._extract_m3u8_formats(
-            video_data['fullpath'], video_id, ext='mp4', m3u8_id='hls')
+            video_data['fullpath'], video_id, ext='mp4',
            entry_protocol='m3u8_native', m3u8_id='hls')
        for a_format in formats:
            # LiTV HLS segments doesn't like compressions
            a_format.setdefault('http_headers', {})['Youtubedl-no-compression'] = True
--- a/youtube_dl/extractor/miomio.py
+++ b/youtube_dl/extractor/miomio.py
@@ -25,10 +25,7 @@ class MioMioIE(InfoExtractor):
            'title': '【SKY】字幕 铠武昭和VS平成 假面骑士大战FEAT战队 魔星字幕组 字幕',
            'duration': 5923,
        },
-        'params': {
+        'skip': 'Unable to load videos',
            # The server provides broken file
            'skip_download': True,
        }
    }, {
        'url': 'http://www.miomio.tv/watch/cc184024/',
        'info_dict': {
@@ -47,16 +44,12 @@ class MioMioIE(InfoExtractor):
        'skip': 'Unable to load videos',
    }, {
        # new 'h5' player
-        'url': 'http://www.miomio.tv/watch/cc273295/',
+        'url': 'http://www.miomio.tv/watch/cc273997/',
-        'md5': '',
+        'md5': '0b27a4b4495055d826813f8c3a6b2070',
        'info_dict': {
-            'id': '273295',
+            'id': '273997',
            'ext': 'mp4',
-            'title': 'アウト×デラックス 20160526',
+            'title': 'マツコの知らない世界【劇的進化SP！ビニール傘＆冷凍食品2016】 1_2 - 16 05 31',
        },
        'params': {
            # intermittent HTTP 500
            'skip_download': True,
        },
    }]
@@ -116,7 +109,7 @@ class MioMioIE(InfoExtractor):
            player_webpage = self._download_webpage(
                player_url, video_id,
                note='Downloading player webpage', headers={'Referer': url})
-            entries = self._parse_html5_media_entries(player_url, player_webpage)
+            entries = self._parse_html5_media_entries(player_url, player_webpage, video_id)
            http_headers = {'Referer': player_url}
        else:
            http_headers = {'Referer': 'http://www.miomio.tv%s' % mioplayer_path}
--- a/youtube_dl/extractor/mofosex.py
+++ b/youtube_dl/extractor/mofosex.py
@@ -1,53 +1,56 @@
 from __future__ import unicode_literals
-import os
+from ..utils import (
-import re
+    int_or_none,
-
+    str_to_int,
-from .common import InfoExtractor
+    unified_strdate,
 from ..compat import (
    compat_urllib_parse_unquote,
    compat_urllib_parse_urlparse,
 )
-from ..utils import sanitized_Request
+from .keezmovies import KeezMoviesIE
-class MofosexIE(InfoExtractor):
+class MofosexIE(KeezMoviesIE):
-    _VALID_URL = r'https?://(?:www\.)?(?P<url>mofosex\.com/videos/(?P<id>[0-9]+)/.*?\.html)'
+    _VALID_URL = r'https?://(?:www\.)?mofosex\.com/videos/(?P<id>\d+)/(?P<display_id>[^/?#&.]+)\.html'
-    _TEST = {
+    _TESTS = [{
-        'url': 'http://www.mofosex.com/videos/5018/japanese-teen-music-video.html',
+        'url': 'http://www.mofosex.com/videos/318131/amateur-teen-playing-and-masturbating-318131.html',
-        'md5': '1b2eb47ac33cc75d4a80e3026b613c5a',
+        'md5': '39a15853632b7b2e5679f92f69b78e91',
        'info_dict': {
-            'id': '5018',
+            'id': '318131',
            'display_id': 'amateur-teen-playing-and-masturbating-318131',
            'ext': 'mp4',
-            'title': 'Japanese Teen Music Video',
+            'title': 'amateur teen playing and masturbating',
            'thumbnail': 're:^https?://.*\.jpg$',
            'upload_date': '20121114',
            'view_count': int,
            'like_count': int,
            'dislike_count': int,
            'age_limit': 18,
        }
-    }
+    }, {
        # This video is no longer available
        'url': 'http://www.mofosex.com/videos/5018/japanese-teen-music-video.html',
        'only_matching': True,
    }]
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        webpage, info = self._extract_info(url)
        video_id = mobj.group('id')
        url = 'http://www.' + mobj.group('url')
-        req = sanitized_Request(url)
+        view_count = str_to_int(self._search_regex(
-        req.add_header('Cookie', 'age_verified=1')
+            r'VIEWS:</span>\s*([\d,.]+)', webpage, 'view count', fatal=False))
-        webpage = self._download_webpage(req, video_id)
+        like_count = int_or_none(self._search_regex(
            r'id=["\']amountLikes["\'][^>]*>(\d+)', webpage,
            'like count', fatal=False))
        dislike_count = int_or_none(self._search_regex(
            r'id=["\']amountDislikes["\'][^>]*>(\d+)', webpage,
            'like count', fatal=False))
        upload_date = unified_strdate(self._html_search_regex(
            r'Added:</span>([^<]+)', webpage, 'upload date', fatal=False))
-        video_title = self._html_search_regex(r'<h1>(.+?)<', webpage, 'title')
+        info.update({
-        video_url = compat_urllib_parse_unquote(self._html_search_regex(r'flashvars.video_url = \'([^\']+)', webpage, 'video_url'))
+            'view_count': view_count,
-        path = compat_urllib_parse_urlparse(video_url).path
+            'like_count': like_count,
-        extension = os.path.splitext(path)[1][1:]
+            'dislike_count': dislike_count,
-        format = path.split('/')[5].split('_')[:2]
+            'upload_date': upload_date,
-        format = '-'.join(format)
+            'thumbnail': self._og_search_thumbnail(webpage),
        })
-        age_limit = self._rta_search(webpage)
+        return info
        return {
            'id': video_id,
            'title': video_title,
            'url': video_url,
            'ext': extension,
            'format': format,
            'format_id': format,
            'age_limit': age_limit,
        }
--- a/youtube_dl/extractor/movingimage.py
+++ b/youtube_dl/extractor/movingimage.py
@@ -7,22 +7,19 @@ from ..utils import (
 )
-class SSAIE(InfoExtractor):
+class MovingImageIE(InfoExtractor):
-    _VALID_URL = r'https?://ssa\.nls\.uk/film/(?P<id>\d+)'
+    _VALID_URL = r'https?://movingimage\.nls\.uk/film/(?P<id>\d+)'
    _TEST = {
-        'url': 'http://ssa.nls.uk/film/3561',
+        'url': 'http://movingimage.nls.uk/film/3561',
        'md5': '4caa05c2b38453e6f862197571a7be2f',
        'info_dict': {
            'id': '3561',
-            'ext': 'flv',
+            'ext': 'mp4',
            'title': 'SHETLAND WOOL',
            'description': 'md5:c5afca6871ad59b4271e7704fe50ab04',
            'duration': 900,
            'thumbnail': 're:^https?://.*\.jpg$',
        },
        'params': {
            # rtmp download
            'skip_download': True,
        },
    }
    def _real_extract(self, url):
@@ -30,10 +27,9 @@ class SSAIE(InfoExtractor):
        webpage = self._download_webpage(url, video_id)
-        streamer = self._search_regex(
+        formats = self._extract_m3u8_formats(
-            r"'streamer'\s*,\S*'(rtmp[^']+)'", webpage, 'streamer')
+            self._html_search_regex(r'file\s*:\s*"([^"]+)"', webpage, 'm3u8 manifest URL'),
-        play_path = self._search_regex(
+            video_id, ext='mp4', entry_protocol='m3u8_native')
            r"'file'\s*,\s*'([^']+)'", webpage, 'file').rpartition('.')[0]
        def search_field(field_name, fatal=False):
            return self._search_regex(
@@ -44,13 +40,11 @@ class SSAIE(InfoExtractor):
        description = unescapeHTML(search_field('Description'))
        duration = parse_duration(search_field('Running time'))
        thumbnail = self._search_regex(
-            r"'image'\s*,\s*'([^']+)'", webpage, 'thumbnails', fatal=False)
+            r"image\s*:\s*'([^']+)'", webpage, 'thumbnail', fatal=False)
        return {
            'id': video_id,
-            'url': streamer,
+            'formats': formats,
            'play_path': play_path,
            'ext': 'flv',
            'title': title,
            'description': description,
            'duration': duration,
--- a/youtube_dl/extractor/mtv.py
+++ b/youtube_dl/extractor/mtv.py
@@ -4,7 +4,6 @@ import re
 from .common import InfoExtractor
 from ..compat import (
    compat_urllib_parse_urlencode,
    compat_str,
    compat_xpath,
 )
@@ -14,12 +13,13 @@ from ..utils import (
    fix_xml_ampersands,
    float_or_none,
    HEADRequest,
    RegexNotFoundError,
    sanitized_Request,
    strip_or_none,
    timeconvert,
    unescapeHTML,
    update_url_query,
    url_basename,
    RegexNotFoundError,
    xpath_text,
 )
@@ -36,6 +36,11 @@ class MTVServicesInfoExtractor(InfoExtractor):
    def _id_from_uri(uri):
        return uri.split(':')[-1]
    @staticmethod
    def _remove_template_parameter(url):
        # Remove the templates, like &device={device}
        return re.sub(r'&[^=]*?={.*?}(?=(&|$))', '', url)
    # This was originally implemented for ComedyCentral, but it also works here
    @classmethod
    def _transform_rtmp_url(cls, rtmp_video_url):
@@ -117,9 +122,7 @@ class MTVServicesInfoExtractor(InfoExtractor):
        video_id = self._id_from_uri(uri)
        self.report_extraction(video_id)
        content_el = itemdoc.find('%s/%s' % (_media_xml_tag('group'), _media_xml_tag('content')))
-        mediagen_url = content_el.attrib['url']
+        mediagen_url = self._remove_template_parameter(content_el.attrib['url'])
        # Remove the templates, like &device={device}
        mediagen_url = re.sub(r'&[^=]*?={.*?}(?=(&|$))', '', mediagen_url)
        if 'acceptMethods' not in mediagen_url:
            mediagen_url += '&' if '?' in mediagen_url else '?'
            mediagen_url += 'acceptMethods=fms'
@@ -178,12 +181,12 @@ class MTVServicesInfoExtractor(InfoExtractor):
        data = {'uri': uri}
        if self._LANG:
            data['lang'] = self._LANG
-        return compat_urllib_parse_urlencode(data)
+        return data
    def _get_videos_info(self, uri):
        video_id = self._id_from_uri(uri)
        feed_url = self._get_feed_url(uri)
-        info_url = feed_url + '?' + self._get_feed_query(uri)
+        info_url = update_url_query(feed_url, self._get_feed_query(uri))
        return self._get_videos_info_from_url(info_url, video_id)
    def _get_videos_info_from_url(self, url, video_id):
@@ -256,13 +259,9 @@ class MTVServicesEmbeddedIE(MTVServicesInfoExtractor):
    def _get_feed_url(self, uri):
        video_id = self._id_from_uri(uri)
-        site_id = uri.replace(video_id, '')
+        config = self._download_json(
-        config_url = ('http://media.mtvnservices.com/pmt/e1/players/{0}/'
+            'http://media.mtvnservices.com/pmt/e1/access/index.html?uri=%s&configtype=edge' % uri, video_id)
-                      'context4/context5/config.xml'.format(site_id))
+        return self._remove_template_parameter(config['feedWithQueryParams'])
        config_doc = self._download_xml(config_url, video_id)
        feed_node = config_doc.find('.//feed')
        feed_url = feed_node.text.strip().split('?')[0]
        return feed_url
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
--- a/youtube_dl/extractor/muenchentv.py
+++ b/youtube_dl/extractor/muenchentv.py
@@ -36,7 +36,7 @@ class MuenchenTVIE(InfoExtractor):
        title = self._live_title(self._og_search_title(webpage))
        data_js = self._search_regex(
-            r'(?s)\nplaylist:\s*(\[.*?}\]),related:',
+            r'(?s)\nplaylist:\s*(\[.*?}\]),',
            webpage, 'playlist configuration')
        data_json = js_to_json(data_js)
        data = json.loads(data_json)[0]
--- a/youtube_dl/extractor/myvidster.py
+++ b/youtube_dl/extractor/myvidster.py
@@ -13,7 +13,7 @@ class MyVidsterIE(InfoExtractor):
            'id': '3685814',
            'title': 'md5:7d8427d6d02c4fbcef50fe269980c749',
            'upload_date': '20141027',
-            'uploader_id': 'utkualp',
+            'uploader': 'utkualp',
            'ext': 'mp4',
            'age_limit': 18,
        },
--- a/youtube_dl/extractor/nationalgeographic.py
+++ b/youtube_dl/extractor/nationalgeographic.py
@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from .theplatform import ThePlatformIE
+from .adobepass import AdobePassIE
 from ..utils import (
    smuggle_url,
    url_basename,
@@ -65,7 +65,7 @@ class NationalGeographicVideoIE(InfoExtractor):
        }
-class NationalGeographicIE(ThePlatformIE):
+class NationalGeographicIE(AdobePassIE):
    IE_NAME = 'natgeo'
    _VALID_URL = r'https?://channel\.nationalgeographic\.com/(?:wild/)?[^/]+/(?:videos|episodes)/(?P<id>[^/?]+)'
@@ -119,7 +119,7 @@ class NationalGeographicIE(ThePlatformIE):
            auth_resource_id = self._search_regex(
                r"video_auth_resourceId\s*=\s*'([^']+)'",
                webpage, 'auth resource id')
-            query['auth'] = self._extract_mvpd_auth(url, display_id, 'natgeo', auth_resource_id) or ''
+            query['auth'] = self._extract_mvpd_auth(url, display_id, 'natgeo', auth_resource_id)
        return {
            '_type': 'url_transparent',
@@ -131,7 +131,7 @@ class NationalGeographicIE(ThePlatformIE):
        }
-class NationalGeographicEpisodeGuideIE(ThePlatformIE):
+class NationalGeographicEpisodeGuideIE(InfoExtractor):
    IE_NAME = 'natgeo:episodeguide'
    _VALID_URL = r'https?://channel\.nationalgeographic\.com/(?:wild/)?(?P<id>[^/]+)/episode-guide'
    _TESTS = [
--- a/youtube_dl/extractor/nba.py
+++ b/youtube_dl/extractor/nba.py
@@ -1,25 +1,20 @@
 from __future__ import unicode_literals
 import functools
 import os.path
 import re
-from .common import InfoExtractor
+from .turner import TurnerBaseIE
 from ..compat import (
    compat_urllib_parse_urlencode,
    compat_urlparse,
 )
 from ..utils import (
    int_or_none,
    OnDemandPagedList,
    parse_duration,
    remove_start,
    xpath_text,
    xpath_attr,
 )
-class NBAIE(InfoExtractor):
+class NBAIE(TurnerBaseIE):
    _VALID_URL = r'https?://(?:watch\.|www\.)?nba\.com/(?P<path>(?:[^/]+/)+(?P<id>[^?]*?))/?(?:/index\.html)?(?:\?.*)?$'
    _TESTS = [{
        'url': 'http://www.nba.com/video/games/nets/2012/12/04/0021200253-okc-bkn-recap.nba/index.html',
@@ -44,28 +39,30 @@ class NBAIE(InfoExtractor):
        'url': 'http://watch.nba.com/video/channels/playoffs/2015/05/20/0041400301-cle-atl-recap.nba',
        'md5': 'b2b39b81cf28615ae0c3360a3f9668c4',
        'info_dict': {
-            'id': '0041400301-cle-atl-recap',
+            'id': 'channels/playoffs/2015/05/20/0041400301-cle-atl-recap.nba',
            'ext': 'mp4',
            'title': 'Hawks vs. Cavaliers Game 1',
            'description': 'md5:8094c3498d35a9bd6b1a8c396a071b4d',
            'duration': 228,
            'timestamp': 1432134543,
            'upload_date': '20150520',
-        }
+        },
        'expected_warnings': ['Unable to download f4m manifest'],
    }, {
        'url': 'http://www.nba.com/clippers/news/doc-rivers-were-not-trading-blake',
        'info_dict': {
-            'id': '1455672027478-Doc_Feb16_720',
+            'id': 'teams/clippers/2016/02/17/1455672027478-Doc_Feb16_720.mov-297324',
            'ext': 'mp4',
            'title': 'Practice: Doc Rivers - 2/16/16',
            'description': 'Head Coach Doc Rivers addresses the media following practice.',
-            'upload_date': '20160217',
+            'upload_date': '20160216',
            'timestamp': 1455672000,
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
        'expected_warnings': ['Unable to download f4m manifest'],
    }, {
        'url': 'http://www.nba.com/timberwolves/wiggins-shootaround#',
        'info_dict': {
@@ -80,7 +77,7 @@ class NBAIE(InfoExtractor):
    }, {
        'url': 'http://www.nba.com/timberwolves/wiggins-shootaround#',
        'info_dict': {
-            'id': 'Wigginsmp4',
+            'id': 'teams/timberwolves/2014/12/12/Wigginsmp4-3462601',
            'ext': 'mp4',
            'title': 'Shootaround Access - Dec. 12 | Andrew Wiggins',
            'description': 'Wolves rookie Andrew Wiggins addresses the media after Friday\'s shootaround.',
@@ -92,6 +89,7 @@ class NBAIE(InfoExtractor):
            # m3u8 download
            'skip_download': True,
        },
        'expected_warnings': ['Unable to download f4m manifest'],
    }]
    _PAGE_SIZE = 30
@@ -145,53 +143,12 @@ class NBAIE(InfoExtractor):
            if path.startswith('video/teams'):
                path = 'video/channels/proxy/' + path[6:]
-        video_info = self._download_xml('http://www.nba.com/%s.xml' % path, video_id)
+        return self._extract_cvp_info(
-        video_id = os.path.splitext(xpath_text(video_info, 'slug'))[0]
+            'http://www.nba.com/%s.xml' % path, video_id, {
-        title = xpath_text(video_info, 'headline')
+                'default': {
-        description = xpath_text(video_info, 'description')
+                    'media_src': 'http://nba.cdn.turner.com/nba/big',
-        duration = parse_duration(xpath_text(video_info, 'length'))
+                },
-        timestamp = int_or_none(xpath_attr(video_info, 'dateCreated', 'uts'))
+                'm3u8': {
-
+                    'media_src': 'http://nbavod-f.akamaihd.net',
-        thumbnails = []
+                },
        for image in video_info.find('images'):
            thumbnails.append({
                'id': image.attrib.get('cut'),
                'url': image.text,
                'width': int_or_none(image.attrib.get('width')),
                'height': int_or_none(image.attrib.get('height')),
            })
        formats = []
        for video_file in video_info.findall('.//file'):
            video_url = video_file.text
            if video_url.startswith('/'):
                continue
            if video_url.endswith('.m3u8'):
                formats.extend(self._extract_m3u8_formats(video_url, video_id, ext='mp4', m3u8_id='hls', fatal=False))
            elif video_url.endswith('.f4m'):
                formats.extend(self._extract_f4m_formats(video_url + '?hdcore=3.4.1.1', video_id, f4m_id='hds', fatal=False))
            else:
                key = video_file.attrib.get('bitrate')
                format_info = {
                    'format_id': key,
                    'url': video_url,
                }
                mobj = re.search(r'(\d+)x(\d+)(?:_(\d+))?', key)
                if mobj:
                    format_info.update({
                        'width': int(mobj.group(1)),
                        'height': int(mobj.group(2)),
                        'tbr': int_or_none(mobj.group(3)),
                    })
                formats.append(format_info)
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': title,
            'description': description,
            'duration': duration,
            'timestamp': timestamp,
            'thumbnails': thumbnails,
            'formats': formats,
        }
--- a/youtube_dl/extractor/nhk.py
+++ b/youtube_dl/extractor/nhk.py
@@ -0,0 +1,50 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 class NhkVodIE(InfoExtractor):
    _VALID_URL = r'https?://www3\.nhk\.or\.jp/nhkworld/en/vod/(?P<id>.+?)\.html'
    _TEST = {
        # Videos available only for a limited period of time. Visit
        # http://www3.nhk.or.jp/nhkworld/en/vod/ for working samples.
        'url': 'http://www3.nhk.or.jp/nhkworld/en/vod/tokyofashion/20160815.html',
        'info_dict': {
            'id': 'A1bnNiNTE6nY3jLllS-BIISfcC_PpvF5',
            'ext': 'flv',
            'title': 'TOKYO FASHION EXPRESS - The Kimono as Global Fashion',
            'description': 'md5:db338ee6ce8204f415b754782f819824',
            'series': 'TOKYO FASHION EXPRESS',
            'episode': 'The Kimono as Global Fashion',
        },
        'skip': 'Videos available only for a limited period of time',
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        embed_code = self._search_regex(
            r'nw_vod_ooplayer\([^,]+,\s*(["\'])(?P<id>(?:(?!\1).)+)\1',
            webpage, 'ooyala embed code', group='id')
        title = self._search_regex(
            r'<div[^>]+class=["\']episode-detail["\']>\s*<h\d+>([^<]+)',
            webpage, 'title', default=None)
        description = self._html_search_regex(
            r'(?s)<p[^>]+class=["\']description["\'][^>]*>(.+?)</p>',
            webpage, 'description', default=None)
        series = self._search_regex(
            r'<h2[^>]+class=["\']detail-top-player-title[^>]+><a[^>]+>([^<]+)',
            webpage, 'series', default=None)
        return {
            '_type': 'url_transparent',
            'ie_key': 'Ooyala',
            'url': 'ooyala:%s' % embed_code,
            'title': '%s - %s' % (series, title) if series and title else title,
            'description': description,
            'series': series,
            'episode': title,
        }
--- a/youtube_dl/extractor/nick.py
+++ b/youtube_dl/extractor/nick.py
@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 from .mtv import MTVServicesInfoExtractor
 from ..compat import compat_urllib_parse_urlencode
 from ..utils import update_url_query
@@ -59,10 +58,10 @@ class NickIE(MTVServicesInfoExtractor):
    }]
    def _get_feed_query(self, uri):
-        return compat_urllib_parse_urlencode({
+        return {
            'feed': 'nick_arc_player_prime',
            'mgid': uri,
-        })
+        }
    def _extract_mgid(self, webpage):
        return self._search_regex(r'data-contenturi="([^"]+)', webpage, 'mgid')
--- a/youtube_dl/extractor/ninecninemedia.py
+++ b/youtube_dl/extractor/ninecninemedia.py
@@ -4,40 +4,36 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    parse_iso8601,
-    parse_duration,
+    float_or_none,
-    ExtractorError
+    ExtractorError,
    int_or_none,
 )
-class NineCNineMediaIE(InfoExtractor):
+class NineCNineMediaBaseIE(InfoExtractor):
-    _VALID_URL = r'9c9media:(?P<destination_code>[^:]+):(?P<id>\d+)'
+    _API_BASE_TEMPLATE = 'http://capi.9c9media.com/destinations/%s/platforms/desktop/contents/%s/'
 class NineCNineMediaStackIE(NineCNineMediaBaseIE):
    IE_NAME = '9c9media:stack'
    _VALID_URL = r'9c9media:stack:(?P<destination_code>[^:]+):(?P<content_id>\d+):(?P<content_package>\d+):(?P<id>\d+)'
    def _real_extract(self, url):
-        destination_code, video_id = re.match(self._VALID_URL, url).groups()
+        destination_code, content_id, package_id, stack_id = re.match(self._VALID_URL, url).groups()
-        api_base_url = 'http://capi.9c9media.com/destinations/%s/platforms/desktop/contents/%s/' % (destination_code, video_id)
+        stack_base_url_template = self._API_BASE_TEMPLATE + 'contentpackages/%s/stacks/%s/manifest.'
-        content = self._download_json(api_base_url, video_id, query={
+        stack_base_url = stack_base_url_template % (destination_code, content_id, package_id, stack_id)
-            '$include': '[contentpackages]',
+
        })
        title = content['Name']
        if len(content['ContentPackages']) > 1:
            raise ExtractorError('multiple content packages')
        content_package = content['ContentPackages'][0]
        stacks_base_url = api_base_url + 'contentpackages/%s/stacks/' % content_package['Id']
        stacks = self._download_json(stacks_base_url, video_id)['Items']
        if len(stacks) > 1:
            raise ExtractorError('multiple stacks')
        stack = stacks[0]
        stack_base_url = '%s%s/manifest.' % (stacks_base_url, stack['Id'])
        formats = []
        formats.extend(self._extract_m3u8_formats(
-            stack_base_url + 'm3u8', video_id, 'mp4',
+            stack_base_url + 'm3u8', stack_id, 'mp4',
            'm3u8_native', m3u8_id='hls', fatal=False))
        formats.extend(self._extract_f4m_formats(
-            stack_base_url + 'f4m', video_id,
+            stack_base_url + 'f4m', stack_id,
            f4m_id='hds', fatal=False))
-        mp4_url = self._download_webpage(stack_base_url + 'pd', video_id, fatal=False)
+        mp4_url = self._download_webpage(stack_base_url + 'pd', stack_id, fatal=False)
        if mp4_url:
            formats.append({
                'url': mp4_url,
@@ -46,10 +42,86 @@ class NineCNineMediaIE(InfoExtractor):
        self._sort_formats(formats)
        return {
-            'id': video_id,
+            'id': stack_id,
            'title': title,
            'description': content.get('Desc') or content.get('ShortDesc'),
            'timestamp': parse_iso8601(content.get('BroadcastDateTime')),
            'duration': parse_duration(content.get('BroadcastTime')),
            'formats': formats,
        }
 class NineCNineMediaIE(NineCNineMediaBaseIE):
    IE_NAME = '9c9media'
    _VALID_URL = r'9c9media:(?P<destination_code>[^:]+):(?P<id>\d+)'
    def _real_extract(self, url):
        destination_code, content_id = re.match(self._VALID_URL, url).groups()
        api_base_url = self._API_BASE_TEMPLATE % (destination_code, content_id)
        content = self._download_json(api_base_url, content_id, query={
            '$include': '[Media,Season,ContentPackages]',
        })
        title = content['Name']
        if len(content['ContentPackages']) > 1:
            raise ExtractorError('multiple content packages')
        content_package = content['ContentPackages'][0]
        package_id = content_package['Id']
        content_package_url = api_base_url + 'contentpackages/%s/' % package_id
        content_package = self._download_json(content_package_url, content_id)
        if content_package.get('Constraints', {}).get('Security', {}).get('Type') == 'adobe-drm':
            raise ExtractorError('This video is DRM protected.', expected=True)
        stacks = self._download_json(content_package_url + 'stacks/', package_id)['Items']
        multistacks = len(stacks) > 1
        thumbnails = []
        for image in content.get('Images', []):
            image_url = image.get('Url')
            if not image_url:
                continue
            thumbnails.append({
                'url': image_url,
                'width': int_or_none(image.get('Width')),
                'height': int_or_none(image.get('Height')),
            })
        tags, categories = [], []
        for source_name, container in (('Tags', tags), ('Genres', categories)):
            for e in content.get(source_name, []):
                e_name = e.get('Name')
                if not e_name:
                    continue
                container.append(e_name)
        description = content.get('Desc') or content.get('ShortDesc')
        season = content.get('Season', {})
        base_info = {
            'description': description,
            'timestamp': parse_iso8601(content.get('BroadcastDateTime')),
            'episode_number': int_or_none(content.get('Episode')),
            'season': season.get('Name'),
            'season_number': season.get('Number'),
            'season_id': season.get('Id'),
            'series': content.get('Media', {}).get('Name'),
            'tags': tags,
            'categories': categories,
        }
        entries = []
        for stack in stacks:
            stack_id = compat_str(stack['Id'])
            entry = {
                '_type': 'url_transparent',
                'url': '9c9media:stack:%s:%s:%s:%s' % (destination_code, content_id, package_id, stack_id),
                'id': stack_id,
                'title': '%s_part%s' % (title, stack['Name']) if multistacks else title,
                'duration': float_or_none(stack.get('Duration')),
                'ie_key': 'NineCNineMediaStack',
            }
            entry.update(base_info)
            entries.append(entry)
        return {
            '_type': 'multi_video',
            'id': content_id,
            'title': title,
            'description': description,
            'entries': entries,
        }
--- a/youtube_dl/extractor/nrk.py
+++ b/youtube_dl/extractor/nrk.py
@@ -14,16 +14,6 @@ from ..utils import (
 class NRKBaseIE(InfoExtractor):
    def _extract_formats(self, manifest_url, video_id, fatal=True):
        formats = []
        formats.extend(self._extract_f4m_formats(
            manifest_url + '?hdcore=3.5.0&plugin=aasp-3.5.0.151.81',
            video_id, f4m_id='hds', fatal=fatal))
        formats.extend(self._extract_m3u8_formats(manifest_url.replace(
            'akamaihd.net/z/', 'akamaihd.net/i/').replace('/manifest.f4m', '/master.m3u8'),
            video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=fatal))
        return formats
    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -45,7 +35,7 @@ class NRKBaseIE(InfoExtractor):
                asset_url = asset.get('url')
                if not asset_url:
                    continue
-                formats = self._extract_formats(asset_url, video_id, fatal=False)
+                formats = self._extract_akamai_formats(asset_url, video_id)
                if not formats:
                    continue
                self._sort_formats(formats)
@@ -69,7 +59,7 @@ class NRKBaseIE(InfoExtractor):
        if not entries:
            media_url = data.get('mediaUrl')
            if media_url:
-                formats = self._extract_formats(media_url, video_id)
+                formats = self._extract_akamai_formats(media_url, video_id)
                self._sort_formats(formats)
                duration = parse_duration(data.get('duration'))
                entries = [{
--- a/youtube_dl/extractor/ntvde.py
+++ b/youtube_dl/extractor/ntvde.py
@@ -1,6 +1,8 @@
-# encoding: utf-8
+# coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
@@ -40,8 +42,8 @@ class NTVDeIE(InfoExtractor):
        timestamp = int_or_none(info.get('publishedDateAsUnixTimeStamp'))
        vdata = self._parse_json(self._search_regex(
            r'(?s)\$\(\s*"\#player"\s*\)\s*\.data\(\s*"player",\s*(\{.*?\})\);',
-            webpage, 'player data'),
+            webpage, 'player data'), video_id,
-            video_id, transform_source=js_to_json)
+            transform_source=lambda s: js_to_json(re.sub(r'advertising:\s*{[^}]+},', '', s)))
        duration = parse_duration(vdata.get('duration'))
        formats = []
--- a/youtube_dl/extractor/nytimes.py
+++ b/youtube_dl/extractor/nytimes.py
@@ -1,26 +1,37 @@
 from __future__ import unicode_literals
 import hmac
 import hashlib
 import base64
 from .common import InfoExtractor
 from ..utils import (
    float_or_none,
    int_or_none,
    parse_iso8601,
    mimetype2ext,
    determine_ext,
 )
 class NYTimesBaseIE(InfoExtractor):
    _SECRET = b'pX(2MbU2);4N{7J8)>YwKRJ+/pQ3JkiU2Q^V>mFYv6g6gYvt6v'
    def _extract_video_from_id(self, video_id):
-        video_data = self._download_json(
+        # Authorization generation algorithm is reverse engineered from `signer` in
-            'http://www.nytimes.com/svc/video/api/v2/video/%s' % video_id,
+        # http://graphics8.nytimes.com/video/vhs/vhs-2.x.min.js
-            video_id, 'Downloading video JSON')
+        path = '/svc/video/api/v3/video/' + video_id
        hm = hmac.new(self._SECRET, (path + ':vhs').encode(), hashlib.sha512).hexdigest()
        video_data = self._download_json('http://www.nytimes.com' + path, video_id, 'Downloading video JSON', headers={
            'Authorization': 'NYTV ' + base64.b64encode(hm.encode()).decode(),
            'X-NYTV': 'vhs',
        }, fatal=False)
        if not video_data:
            video_data = self._download_json(
                'http://www.nytimes.com/svc/video/api/v2/video/' + video_id,
                video_id, 'Downloading video JSON')
        title = video_data['headline']
        description = video_data.get('summary')
        duration = float_or_none(video_data.get('duration'), 1000)
        uploader = video_data.get('byline')
        publication_date = video_data.get('publication_date')
        timestamp = parse_iso8601(publication_date[:-8]) if publication_date else None
        def get_file_size(file_size):
            if isinstance(file_size, int):
@@ -28,35 +39,59 @@ class NYTimesBaseIE(InfoExtractor):
            elif isinstance(file_size, dict):
                return int(file_size.get('value', 0))
            else:
-                return 0
+                return None
-        formats = [
+        urls = []
-            {
+        formats = []
-                'url': video['url'],
+        for video in video_data.get('renditions', []):
-                'format_id': video.get('type'),
+            video_url = video.get('url')
-                'vcodec': video.get('video_codec'),
+            format_id = video.get('type')
-                'width': int_or_none(video.get('width')),
+            if not video_url or format_id == 'thumbs' or video_url in urls:
-                'height': int_or_none(video.get('height')),
+                continue
-                'filesize': get_file_size(video.get('fileSize')),
+            urls.append(video_url)
-            } for video in video_data['renditions'] if video.get('url')
+            ext = mimetype2ext(video.get('mimetype')) or determine_ext(video_url)
-        ]
+            if ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    video_url, video_id, 'mp4', 'm3u8_native',
                    m3u8_id=format_id or 'hls', fatal=False))
            elif ext == 'mpd':
                continue
            #     formats.extend(self._extract_mpd_formats(
            #         video_url, video_id, format_id or 'dash', fatal=False))
            else:
                formats.append({
                    'url': video_url,
                    'format_id': format_id,
                    'vcodec': video.get('videoencoding') or video.get('video_codec'),
                    'width': int_or_none(video.get('width')),
                    'height': int_or_none(video.get('height')),
                    'filesize': get_file_size(video.get('file_size') or video.get('fileSize')),
                    'tbr': int_or_none(video.get('bitrate'), 1000),
                    'ext': ext,
                })
        self._sort_formats(formats)
-        thumbnails = [
+        thumbnails = []
-            {
+        for image in video_data.get('images', []):
-                'url': 'http://www.nytimes.com/%s' % image['url'],
+            image_url = image.get('url')
            if not image_url:
                continue
            thumbnails.append({
                'url': 'http://www.nytimes.com/' + image_url,
                'width': int_or_none(image.get('width')),
                'height': int_or_none(image.get('height')),
-            } for image in video_data.get('images', []) if image.get('url')
+            })
-        ]
+
        publication_date = video_data.get('publication_date')
        timestamp = parse_iso8601(publication_date[:-8]) if publication_date else None
        return {
            'id': video_id,
            'title': title,
-            'description': description,
+            'description': video_data.get('summary'),
            'timestamp': timestamp,
-            'uploader': uploader,
+            'uploader': video_data.get('byline'),
-            'duration': duration,
+            'duration': float_or_none(video_data.get('duration'), 1000),
            'formats': formats,
            'thumbnails': thumbnails,
        }
@@ -67,7 +102,7 @@ class NYTimesIE(NYTimesBaseIE):
    _TESTS = [{
        'url': 'http://www.nytimes.com/video/opinion/100000002847155/verbatim-what-is-a-photocopier.html?playlistId=100000001150263',
-        'md5': '18a525a510f942ada2720db5f31644c0',
+        'md5': 'd665342765db043f7e225cff19df0f2d',
        'info_dict': {
            'id': '100000002847155',
            'ext': 'mov',
--- a/youtube_dl/extractor/openload.py
+++ b/youtube_dl/extractor/openload.py
@@ -1,12 +1,12 @@
 # coding: utf-8
 from __future__ import unicode_literals, division
 import math
 from .common import InfoExtractor
-from ..compat import compat_chr
+from ..compat import (
    compat_chr,
    compat_ord,
 )
 from ..utils import (
    decode_png,
    determine_ext,
    ExtractorError,
 )
@@ -42,71 +42,28 @@ class OpenloadIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        webpage = self._download_webpage('https://openload.co/embed/%s/' % video_id, video_id)
-        if 'File not found' in webpage:
+        if 'File not found' in webpage or 'deleted by the owner' in webpage:
            raise ExtractorError('File not found', expected=True)
-        # The following extraction logic is proposed by @Belderak and @gdkchan
+        # The following decryption algorithm is written by @yokrysty and
-        # and declared to be used freely in youtube-dl
+        # declared to be freely used in youtube-dl
-        # See https://github.com/rg3/youtube-dl/issues/9706
+        # See https://github.com/rg3/youtube-dl/issues/10408
        enc_data = self._html_search_regex(
            r'<span[^>]+id="hiddenurl"[^>]*>([^<]+)</span>', webpage, 'encrypted data')
-        numbers_js = self._download_webpage(
+        video_url_chars = []
            'https://openload.co/assets/js/obfuscator/n.js', video_id,
            note='Downloading signature numbers')
        signums = self._search_regex(
            r'window\.signatureNumbers\s*=\s*[\'"](?P<data>[a-z]+)[\'"]',
            numbers_js, 'signature numbers', group='data')
-        linkimg_uri = self._search_regex(
+        for idx, c in enumerate(enc_data):
-            r'<img[^>]+id="linkimg"[^>]+src="([^"]+)"', webpage, 'link image')
+            j = compat_ord(c)
-        linkimg = self._request_webpage(
+            if j >= 33 and j <= 126:
-            linkimg_uri, video_id, note=False).read()
+                j = ((j + 14) % 94) + 33
            if idx == len(enc_data) - 1:
                j += 1
            video_url_chars += compat_chr(j)
-        width, height, pixels = decode_png(linkimg)
+        video_url = 'https://openload.co/stream/%s?mime=true' % ''.join(video_url_chars)
        output = ''
        for y in range(height):
            for x in range(width):
                r, g, b = pixels[y][3 * x:3 * x + 3]
                if r == 0 and g == 0 and b == 0:
                    break
                else:
                    output += compat_chr(r)
                    output += compat_chr(g)
                    output += compat_chr(b)
        img_str_length = len(output) // 200
        img_str = [[0 for x in range(img_str_length)] for y in range(10)]
        sig_str_length = len(signums) // 260
        sig_str = [[0 for x in range(sig_str_length)] for y in range(10)]
        for i in range(10):
            for j in range(img_str_length):
                begin = i * img_str_length * 20 + j * 20
                img_str[i][j] = output[begin:begin + 20]
            for j in range(sig_str_length):
                begin = i * sig_str_length * 26 + j * 26
                sig_str[i][j] = signums[begin:begin + 26]
        parts = []
        # TODO: find better names for str_, chr_ and sum_
        str_ = ''
        for i in [2, 3, 5, 7]:
            str_ = ''
            sum_ = float(99)
            for j in range(len(sig_str[i])):
                for chr_idx in range(len(img_str[i][j])):
                    if sum_ > float(122):
                        sum_ = float(98)
                    chr_ = compat_chr(int(math.floor(sum_)))
                    if sig_str[i][j][chr_idx] == chr_ and j >= len(str_):
                        sum_ += float(2.5)
                        str_ += img_str[i][j][chr_idx]
            parts.append(str_.replace(',', ''))
        video_url = 'https://openload.co/stream/%s~%s~%s~%s' % (parts[3], parts[1], parts[2], parts[0])
        title = self._og_search_title(webpage, default=None) or self._search_regex(
            r'<span[^>]+class=["\']title["\'][^>]*>([^<]+)', webpage,
--- a/Show More
+++ b/Show More