Merge branch 'master' into totalwebcasting

.github/ISSUE_TEMPLATE.md

@@ -6,13 +6,12 @@
 
 ---
 
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2019.02.18*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
-- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2019.02.18**
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.12.31*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2017.12.31**
 
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
 - [ ] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
-- [ ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
 
 ### What is the purpose of your *issue*?
 - [ ] Bug report (encountered problems with youtube-dl)
@@ -36,7 +35,7 @@ Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2019.02.18
+[debug] youtube-dl version 2017.12.31
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}

.github/ISSUE_TEMPLATE_tmpl.md

@@ -12,7 +12,6 @@
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
 - [ ] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
-- [ ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
 
 ### What is the purpose of your *issue*?
 - [ ] Bug report (encountered problems with youtube-dl)

.gitignore

@@ -47,7 +47,3 @@ youtube-dl.zsh
 *.iml
 
 tmp/
-venv/
-
-# VS Code related files
-.vscode

.travis.yml

@@ -15,18 +15,6 @@ env:
   - YTDL_TEST_SET=download
 matrix:
   include:
-    - python: 3.7
-      dist: xenial
-      env: YTDL_TEST_SET=core
-    - python: 3.7
-      dist: xenial
-      env: YTDL_TEST_SET=download
-    - python: 3.8-dev
-      dist: xenial
-      env: YTDL_TEST_SET=core
-    - python: 3.8-dev
-      dist: xenial
-      env: YTDL_TEST_SET=download
     - env: JYTHON=true; YTDL_TEST_SET=core
     - env: JYTHON=true; YTDL_TEST_SET=download
   fast_finish: true

AUTHORS

@@ -231,18 +231,3 @@ John Dong
 Tatsuyuki Ishi
 Daniel Weber
 Kay Bouché
-Yang Hongbo
-Lei Wang
-Petr Novák
-Leonardo Taccari
-Martin Weinelt
-Surya Oktafendri
-TingPing
-Alexandre Macabies
-Bastian de Groot
-Niklas Haas
-András Veres-Szentkirályi
-Enes Solak
-Nathan Rossi
-Thomas van der Berg
-Luca Cherubin

CONTRIBUTING.md

@@ -152,20 +152,16 @@ After you have ensured this site is distributing its content legally, you can fo
     ```
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](http://flake8.pycqa.org/en/latest/index.html#quickstart):
-
-        $ flake8 youtube_dl/extractor/yourextractor.py
-
-9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
+7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+9. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
 
         $ git add youtube_dl/extractor/extractors.py
         $ git add youtube_dl/extractor/yourextractor.py
         $ git commit -m '[yourextractor] Add new extractor'
         $ git push origin yourextractor
 
-11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
@@ -177,7 +173,7 @@ Extractors are very fragile by nature since they depend on the layout of the sou
 
 ### Mandatory and optional metafields
 
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
+For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L75-L257) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
 
  - `id` (media identifier)
  - `title` (media title)
@@ -185,7 +181,7 @@ For extraction to work youtube-dl relies on metadata your extractor extracts and
 
 In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken.
 
-[Any field](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
+[Any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L149-L257) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
 
 #### Example
 
@@ -261,33 +257,11 @@ title = meta.get('title') or self._og_search_title(webpage)
 
 This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
 
-### Regular expressions
+### Make regular expressions flexible
 
-#### Don't capture groups you don't use
-
-Capturing group must be an indication that it's used somewhere in the code. Any group that is not used must be non capturing.
-
-##### Example
-
-Don't capture id attribute name here since you can't use it for anything anyway.
-
-Correct:
-
-```python
-r'(?:id|ID)=(?P<id>\d+)'
-```
-
-Incorrect:
-```python
-r'(id|ID)=(?P<id>\d+)'
-```
-
-
-#### Make regular expressions relaxed and flexible
-
-When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on.
+When using regular expressions try to write them fuzzy and flexible.
  
-##### Example
+#### Example
 
 Say you need to extract `title` from the following HTML code:
 
@@ -320,49 +294,7 @@ title = self._search_regex(
     webpage, 'title', group='title')
 ```
 
-### Long lines policy
+### Use safe conversion functions
 
-There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse.
-
-For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit:
-
-Correct:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-Incorrect:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list='
-'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-### Use convenience conversion and parsing functions
-
-Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
-
-Use `url_or_none` for safe URL processing.
-
-Use `try_get` for safe metadata extraction from parsed JSON.
-
-Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. 
-
-Explore [`youtube_dl/utils.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions.
-
-#### More examples
-
-##### Safely extract optional description from parsed JSON
-```python
-description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str)
-```
-
-##### Safely extract more optional metadata
-```python
-video = try_get(response, lambda x: x['result']['video'][0], dict) or {}
-description = video.get('summary')
-duration = float_or_none(video.get('durationMs'), scale=1000)
-view_count = int_or_none(video.get('views'))
-```
+Wrap all extracted numeric data into safe functions from `utils`: `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
 

Makefile

@@ -14,9 +14,6 @@ PYTHON ?= /usr/bin/env python
 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
 SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
 
-# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
-MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
-
 install: youtube-dl youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
 	install -d $(DESTDIR)$(BINDIR)
 	install -m 755 youtube-dl $(DESTDIR)$(BINDIR)
@@ -85,11 +82,11 @@ supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py docs/supportedsites.md
 
 README.txt: README.md
-	pandoc -f $(MARKDOWN) -t plain README.md -o README.txt
+	pandoc -f markdown -t plain README.md -o README.txt
 
 youtube-dl.1: README.md
 	$(PYTHON) devscripts/prepare_manpage.py youtube-dl.1.temp.md
-	pandoc -s -f $(MARKDOWN) -t man youtube-dl.1.temp.md -o youtube-dl.1
+	pandoc -s -f markdown -t man youtube-dl.1.temp.md -o youtube-dl.1
 	rm -f youtube-dl.1.temp.md
 
 youtube-dl.bash-completion: youtube_dl/*.py youtube_dl/*/*.py devscripts/bash-completion.in

README.md

@@ -17,7 +17,7 @@ youtube-dl - download videos from youtube.com or other video platforms
 
 # INSTALLATION
 
-To install it right away for all UNIX users (Linux, macOS, etc.), type:
+To install it right away for all UNIX users (Linux, OS X, etc.), type:
 
     sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
     sudo chmod a+rx /usr/local/bin/youtube-dl
@@ -35,7 +35,7 @@ You can also use pip:
     
 This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
 
-macOS users can install youtube-dl with [Homebrew](https://brew.sh/):
+OS X users can install youtube-dl with [Homebrew](https://brew.sh/):
 
     brew install youtube-dl
 
@@ -46,7 +46,7 @@ Or with [MacPorts](https://www.macports.org/):
 Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see the [youtube-dl Download Page](https://rg3.github.io/youtube-dl/download.html).
 
 # DESCRIPTION
-**youtube-dl** is a command-line program to download videos from YouTube.com and a few more sites. It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
+**youtube-dl** is a command-line program to download videos from YouTube.com and a few more sites. It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on Mac OS X. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
 
     youtube-dl [OPTIONS] URL [URL...]
 
@@ -93,8 +93,8 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 
 ## Network Options:
     --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
-                                     To enable SOCKS proxy, specify a proper
-                                     scheme. For example
+                                     To enable experimental SOCKS proxy, specify
+                                     a proper scheme. For example
                                      socks5://127.0.0.1:1080/. Pass in an empty
                                      string (--proxy "") for direct connection
     --socket-timeout SECONDS         Time to wait before giving up, in seconds
@@ -106,18 +106,16 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
     --geo-verification-proxy URL     Use this proxy to verify the IP address for
                                      some geo-restricted sites. The default
                                      proxy specified by --proxy (or none, if the
-                                     option is not present) is used for the
+                                     options is not present) is used for the
                                      actual downloading.
     --geo-bypass                     Bypass geographic restriction via faking
-                                     X-Forwarded-For HTTP header
+                                     X-Forwarded-For HTTP header (experimental)
     --no-geo-bypass                  Do not bypass geographic restriction via
                                      faking X-Forwarded-For HTTP header
+                                     (experimental)
     --geo-bypass-country CODE        Force bypass geographic restriction with
                                      explicitly provided two-letter ISO 3166-2
-                                     country code
-    --geo-bypass-ip-block IP_BLOCK   Force bypass geographic restriction with
-                                     explicitly provided IP block in CIDR
-                                     notation
+                                     country code (experimental)
 
 ## Video Selection:
     --playlist-start NUMBER          Playlist video to start at (default is 1)
@@ -200,15 +198,10 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
                                      size. By default, the buffer size is
                                      automatically resized from an initial value
                                      of SIZE.
-    --http-chunk-size SIZE           Size of a chunk for chunk-based HTTP
-                                     downloading (e.g. 10485760 or 10M) (default
-                                     is disabled). May be useful for bypassing
-                                     bandwidth throttling imposed by a webserver
-                                     (experimental)
     --playlist-reverse               Download playlist videos in reverse order
     --playlist-random                Download playlist videos in random order
     --xattr-set-filesize             Set file xattribute ytdl.filesize with
-                                     expected file size
+                                     expected file size (experimental)
     --hls-prefer-native              Use the native HLS downloader instead of
                                      ffmpeg
     --hls-prefer-ffmpeg              Use ffmpeg instead of the native HLS
@@ -225,9 +218,7 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 
 ## Filesystem Options:
     -a, --batch-file FILE            File containing URLs to download ('-' for
-                                     stdin), one URL per line. Lines starting
-                                     with '#', ';' or ']' are considered as
-                                     comments and ignored.
+                                     stdin)
     --id                             Use only video ID in file name
     -o, --output TEMPLATE            Output filename template, see the "OUTPUT
                                      TEMPLATE" for all the info
@@ -427,9 +418,9 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
                                      default; fix file if we can, warn
                                      otherwise)
     --prefer-avconv                  Prefer avconv over ffmpeg for running the
-                                     postprocessors
-    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
                                      postprocessors (default)
+    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
+                                     postprocessors
     --ffmpeg-location PATH           Location of the ffmpeg/avconv binary;
                                      either the path to the binary or its
                                      containing directory.
@@ -442,7 +433,7 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 
 # CONFIGURATION
 
-You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and macOS, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
+You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and OS X, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
 
 For example, with the following configuration file youtube-dl will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
 ```
@@ -496,7 +487,7 @@ The `-o` option allows users to indicate a template for the output file names.
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
-The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dl -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations. Allowed names along with sequence type are:
+The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dl -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by a formatting operations. Allowed names along with sequence type are:
 
  - `id` (string): Video identifier
  - `title` (string): Video title
@@ -511,8 +502,6 @@ The basic usage is not to set any template arguments when downloading a single f
  - `timestamp` (numeric): UNIX timestamp of the moment the video became available
  - `upload_date` (string): Video upload date (YYYYMMDD)
  - `uploader_id` (string): Nickname or id of the video uploader
- - `channel` (string): Full name of the channel the video is uploaded on
- - `channel_id` (string): Id of the channel
  - `location` (string): Physical location where the video was filmed
  - `duration` (numeric): Length of the video in seconds
  - `view_count` (numeric): How many users have watched the video on the platform
@@ -667,7 +656,7 @@ The following numeric meta fields can be used with comparisons `<`, `<=`, `>`, `
  - `asr`: Audio sampling rate in Hertz
  - `fps`: Frame rate
 
-Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends with), `*=` (contains) and following string meta fields:
+Also filtering work for comparisons `=` (equals), `!=` (not equals), `^=` (begins with), `$=` (ends with), `*=` (contains) and following string meta fields:
  - `ext`: File extension
  - `acodec`: Name of the audio codec in use
  - `vcodec`: Name of the video codec in use
@@ -675,8 +664,6 @@ Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends
  - `protocol`: The protocol that will be used for the actual download, lower-case (`http`, `https`, `rtsp`, `rtmp`, `rtmpe`, `mms`, `f4m`, `ism`, `http_dash_segments`, `m3u8`, or `m3u8_native`)
  - `format_id`: A short description of the format
 
-Any string comparison may be prefixed with negation `!` in order to produce an opposite comparison, e.g. `!*=` (does not contain).
-
 Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular extractor, i.e. the metadata offered by the video hoster.
 
 Formats for which the value is not known are excluded unless you put a question mark (`?`) after the operator. You can combine format filters, so `-f "[height <=? 720][tbr>500]"` selects up to 720p videos (or videos where the height is not known) with a bitrate of at least 500 KBit/s.
@@ -874,9 +861,9 @@ Either prepend `https://www.youtube.com/watch?v=` or separate the ID from the op
 
 Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`.
 
-In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) (for Firefox).
+In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [Export Cookies](https://addons.mozilla.org/en-US/firefox/addon/export-cookies/) (for Firefox).
 
-Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows and `LF` (`\n`) for Unix and Unix-like systems (Linux, macOS, etc.). `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
+Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows and `LF` (`\n`) for Unix and Unix-like systems (Linux, Mac OS, etc.). `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
 
 Passing cookies to youtube-dl is a good way to workaround login when a particular extractor does not implement it explicitly. Another use case is working around [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) some websites require you to solve in particular cases in order to get access (e.g. YouTube, CloudFlare).
 
@@ -1026,20 +1013,16 @@ After you have ensured this site is distributing its content legally, you can fo
     ```
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](http://flake8.pycqa.org/en/latest/index.html#quickstart):
-
-        $ flake8 youtube_dl/extractor/yourextractor.py
-
-9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
+7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+9. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
 
         $ git add youtube_dl/extractor/extractors.py
         $ git add youtube_dl/extractor/yourextractor.py
         $ git commit -m '[yourextractor] Add new extractor'
         $ git push origin yourextractor
 
-11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
@@ -1051,7 +1034,7 @@ Extractors are very fragile by nature since they depend on the layout of the sou
 
 ### Mandatory and optional metafields
 
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
+For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L75-L257) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
 
  - `id` (media identifier)
  - `title` (media title)
@@ -1059,7 +1042,7 @@ For extraction to work youtube-dl relies on metadata your extractor extracts and
 
 In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken.
 
-[Any field](https://github.com/rg3/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
+[Any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L149-L257) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
 
 #### Example
 
@@ -1135,33 +1118,11 @@ title = meta.get('title') or self._og_search_title(webpage)
 
 This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
 
-### Regular expressions
+### Make regular expressions flexible
 
-#### Don't capture groups you don't use
-
-Capturing group must be an indication that it's used somewhere in the code. Any group that is not used must be non capturing.
-
-##### Example
-
-Don't capture id attribute name here since you can't use it for anything anyway.
-
-Correct:
-
-```python
-r'(?:id|ID)=(?P<id>\d+)'
-```
-
-Incorrect:
-```python
-r'(id|ID)=(?P<id>\d+)'
-```
-
-
-#### Make regular expressions relaxed and flexible
-
-When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on.
+When using regular expressions try to write them fuzzy and flexible.
  
-##### Example
+#### Example
 
 Say you need to extract `title` from the following HTML code:
 
@@ -1194,51 +1155,9 @@ title = self._search_regex(
     webpage, 'title', group='title')
 ```
 
-### Long lines policy
+### Use safe conversion functions
 
-There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse.
-
-For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit:
-
-Correct:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-Incorrect:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list='
-'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-### Use convenience conversion and parsing functions
-
-Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
-
-Use `url_or_none` for safe URL processing.
-
-Use `try_get` for safe metadata extraction from parsed JSON.
-
-Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. 
-
-Explore [`youtube_dl/utils.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions.
-
-#### More examples
-
-##### Safely extract optional description from parsed JSON
-```python
-description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str)
-```
-
-##### Safely extract more optional metadata
-```python
-video = try_get(response, lambda x: x['result']['video'][0], dict) or {}
-description = video.get('summary')
-duration = float_or_none(video.get('durationMs'), scale=1000)
-view_count = int_or_none(video.get('views'))
-```
+Wrap all extracted numeric data into safe functions from `utils`: `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
 
 # EMBEDDING YOUTUBE-DL
 

devscripts/gh-pages/generate-download.py

@@ -1,22 +1,27 @@
 #!/usr/bin/env python3
 from __future__ import unicode_literals
 
+import hashlib
+import urllib.request
 import json
 
 versions_info = json.load(open('update/versions.json'))
 version = versions_info['latest']
-version_dict = versions_info['versions'][version]
+URL = versions_info['versions'][version]['bin'][0]
+
+data = urllib.request.urlopen(URL).read()
 
 # Read template page
 with open('download.html.in', 'r', encoding='utf-8') as tmplf:
     template = tmplf.read()
 
+sha256sum = hashlib.sha256(data).hexdigest()
 template = template.replace('@PROGRAM_VERSION@', version)
-template = template.replace('@PROGRAM_URL@', version_dict['bin'][0])
-template = template.replace('@PROGRAM_SHA256SUM@', version_dict['bin'][1])
-template = template.replace('@EXE_URL@', version_dict['exe'][0])
-template = template.replace('@EXE_SHA256SUM@', version_dict['exe'][1])
-template = template.replace('@TAR_URL@', version_dict['tar'][0])
-template = template.replace('@TAR_SHA256SUM@', version_dict['tar'][1])
+template = template.replace('@PROGRAM_URL@', URL)
+template = template.replace('@PROGRAM_SHA256SUM@', sha256sum)
+template = template.replace('@EXE_URL@', versions_info['versions'][version]['exe'][0])
+template = template.replace('@EXE_SHA256SUM@', versions_info['versions'][version]['exe'][1])
+template = template.replace('@TAR_URL@', versions_info['versions'][version]['tar'][0])
+template = template.replace('@TAR_SHA256SUM@', versions_info['versions'][version]['tar'][1])
 with open('download.html', 'w', encoding='utf-8') as dlf:
     dlf.write(template)

devscripts/gh-pages/update-copyright.py

@@ -13,7 +13,7 @@ year = str(datetime.datetime.now().year)
 for fn in glob.glob('*.html*'):
     with io.open(fn, encoding='utf-8') as f:
         content = f.read()
-    newc = re.sub(r'(?P<copyright>Copyright © 2011-)(?P<year>[0-9]{4})', 'Copyright © 2011-' + year, content)
+    newc = re.sub(r'(?P<copyright>Copyright © 2006-)(?P<year>[0-9]{4})', 'Copyright © 2006-' + year, content)
     if content != newc:
         tmpFn = fn + '.part'
         with io.open(tmpFn, 'wt', encoding='utf-8') as outf:

docs/supportedsites.md

@@ -15,6 +15,7 @@
  - **8tracks**
  - **91porn**
  - **9c9media**
+ - **9c9media:stack**
  - **9gag**
  - **9now.com.au**
  - **abc.net.au**
@@ -33,7 +34,7 @@
  - **AdobeTVShow**
  - **AdobeTVVideo**
  - **AdultSwim**
- - **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault
+ - **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network
  - **afreecatv**: afreecatv.com
  - **AirMozilla**
  - **AliExpressLive**
@@ -47,7 +48,6 @@
  - **anitube.se**
  - **Anvato**
  - **AnySex**
- - **APA**
  - **Aparat**
  - **AppleConnect**
  - **AppleDaily**: 臺灣蘋果日報
@@ -56,7 +56,6 @@
  - **archive.org**: archive.org videos
  - **ARD**
  - **ARD:mediathek**
- - **ARDBetaMediathek**
  - **Arkena**
  - **arte.tv**
  - **arte.tv:+7**
@@ -84,6 +83,8 @@
  - **awaan:season**
  - **awaan:video**
  - **AZMedien**: AZ Medien videos
+ - **AZMedienPlaylist**: AZ Medien playlists
+ - **AZMedienShowPlaylist**: AZ Medien show playlists
  - **BaiduVideo**: 百度视频
  - **bambuser**
  - **bambuser:channel**
@@ -96,11 +97,9 @@
  - **bbc.co.uk:article**: BBC articles
  - **bbc.co.uk:iplayer:playlist**
  - **bbc.co.uk:playlist**
- - **BBVTV**
  - **Beatport**
  - **Beeg**
  - **BehindKink**
- - **Bellator**
  - **BellMedia**
  - **Bet**
  - **Bigflix**
@@ -108,8 +107,6 @@
  - **BiliBili**
  - **BioBioChileTV**
  - **BIQLE**
- - **BitChute**
- - **BitChuteChannel**
  - **BleacherReport**
  - **BleacherReportCMS**
  - **blinkx**
@@ -125,23 +122,19 @@
  - **BRMediathek**: Bayerischer Rundfunk Mediathek
  - **bt:article**: Bergens Tidende Articles
  - **bt:vestlendingen**: Bergens Tidende - Vestlendingen
- - **BusinessInsider**
  - **BuzzFeed**
  - **BYUtv**
  - **Camdemy**
  - **CamdemyFolder**
- - **CamModels**
- - **CamTube**
  - **CamWithHer**
  - **canalc2.tv**
- - **Canalplus**: mycanal.fr and piwiplus.fr
+ - **Canalplus**: canalplus.fr, piwiplus.fr and d8.tv
  - **Canvas**
  - **CanvasEen**: canvas.be and een.be
  - **CarambaTV**
  - **CarambaTVPage**
  - **CartoonNetwork**
  - **cbc.ca**
- - **cbc.ca:olympics**
  - **cbc.ca:player**
  - **cbc.ca:watch**
  - **cbc.ca:watch:video**
@@ -163,22 +156,18 @@
  - **chirbit**
  - **chirbit:profile**
  - **Cinchcast**
- - **CiscoLiveSearch**
- - **CiscoLiveSession**
  - **CJSW**
  - **cliphunter**
  - **Clippit**
  - **ClipRs**
  - **Clipsyndicate**
  - **CloserToTruth**
- - **CloudflareStream**
  - **cloudtime**: CloudTime
  - **Cloudy**
  - **Clubic**
  - **Clyp**
  - **cmt.com**
  - **CNBC**
- - **CNBCVideo**
  - **CNN**
  - **CNNArticle**
  - **CNNBlogs**
@@ -194,13 +183,13 @@
  - **Crackle**
  - **Criterion**
  - **CrooksAndLiars**
- - **crunchyroll**
+ - **Crunchyroll**
  - **crunchyroll:playlist**
  - **CSNNE**
  - **CSpan**: C-SPAN
  - **CtsNews**: 華視新聞
  - **CTVNews**
- - **Culturebox**
+ - **culturebox.francetvinfo.fr**
  - **CultureUnplugged**
  - **curiositystream**
  - **curiositystream:collection**
@@ -221,7 +210,6 @@
  - **defense.gouv.fr**
  - **democracynow**
  - **DHM**: Filmarchiv - Deutsches Historisches Museum
- - **Digg**
  - **DigitallySpeaking**
  - **Digiteka**
  - **Discovery**
@@ -242,7 +230,6 @@
  - **DrTuber**
  - **drtv**
  - **drtv:live**
- - **DTube**
  - **Dumpert**
  - **dvtv**: http://video.aktualne.cz/
  - **dw**
@@ -252,9 +239,7 @@
  - **EchoMsk**
  - **egghead:course**: egghead.io course
  - **egghead:lesson**: egghead.io lesson
- - **ehftv**
  - **eHow**
- - **EinsUndEinsTV**
  - **Einthusan**
  - **eitb.tv**
  - **EllenTube**
@@ -270,11 +255,10 @@
  - **ESPN**
  - **ESPNArticle**
  - **EsriVideo**
+ - **ETOnline**
  - **Europa**
  - **EveryonesMixtape**
- - **EWETV**
  - **ExpoTV**
- - **Expressen**
  - **ExtremeTube**
  - **EyedoTV**
  - **facebook**
@@ -298,6 +282,7 @@
  - **Foxgay**
  - **foxnews**: Fox News and Fox Business Video
  - **foxnews:article**
+ - **foxnews:insider**
  - **FoxSports**
  - **france2.fr:generation-what**
  - **FranceCulture**
@@ -305,22 +290,15 @@
  - **FranceTV**
  - **FranceTVEmbed**
  - **francetvinfo.fr**
- - **FranceTVJeunesse**
- - **FranceTVSite**
  - **Freesound**
  - **freespeech.org**
  - **FreshLive**
- - **FrontendMasters**
- - **FrontendMastersCourse**
- - **FrontendMastersLesson**
  - **Funimation**
- - **FunkChannel**
- - **FunkMix**
+ - **Funk**
  - **FunnyOrDie**
  - **Fusion**
  - **Fux**
  - **FXNetworks**
- - **Gaia**
  - **GameInformer**
  - **GameOne**
  - **gameone:playlist**
@@ -333,7 +311,6 @@
  - **Gfycat**
  - **GiantBomb**
  - **Giga**
- - **GlattvisionTV**
  - **Glide**: Glide mobile video messages (glide.me)
  - **Globo**
  - **GloboArticle**
@@ -355,16 +332,14 @@
  - **HentaiStigma**
  - **hetklokhuis**
  - **hgtv.com:show**
- - **HiDive**
  - **HistoricFilms**
  - **history:topic**: History.com Topic
  - **hitbox**
  - **hitbox:live**
  - **HitRecord**
- - **hketv**: 香港教育局教育電視 (HKETV) Educational Television, Hong Kong Educational Bureau
  - **HornBunny**
  - **HotNewHipHop**
- - **hotstar**
+ - **HotStar**
  - **hotstar:playlist**
  - **Howcast**
  - **HowStuffWorks**
@@ -372,22 +347,19 @@
  - **HRTiPlaylist**
  - **Huajiao**: 花椒直播
  - **HuffPost**: Huffington Post
- - **Hungama**
- - **HungamaSong**
  - **Hypem**
  - **Iconosquare**
  - **ign.com**
  - **imdb**: Internet Movie Database trailers
  - **imdb:list**: Internet Movie Database lists
  - **Imgur**
- - **imgur:album**
- - **imgur:gallery**
+ - **ImgurAlbum**
  - **Ina**
  - **Inc**
+ - **Indavideo**
  - **IndavideoEmbed**
  - **InfoQ**
  - **Instagram**
- - **instagram:tag**: Instagram hashtag search
  - **instagram:user**: Instagram user profile
  - **Internazionale**
  - **InternetVideoArchive**
@@ -396,7 +368,6 @@
  - **Ir90Tv**
  - **ITTF**
  - **ITV**
- - **ITVBTCC**
  - **ivi**: ivi.ru
  - **ivi:compilation**: ivi.ru compilations
  - **ivideon**: Ivideon TV
@@ -411,6 +382,7 @@
  - **JWPlatform**
  - **Kakao**
  - **Kaltura**
+ - **Kamcord**
  - **KanalPlay**: Kanal 5/9/11 Play
  - **Kankan**
  - **Karaoketv**
@@ -420,7 +392,6 @@
  - **Ketnet**
  - **KhanAcademy**
  - **KickStarter**
- - **KinoPoisk**
  - **KonserthusetPlay**
  - **kontrtube**: KontrTube.ru - Труба зовёт
  - **KrasView**: Красвью
@@ -441,12 +412,8 @@
  - **Le**: 乐视网
  - **Learnr**
  - **Lecture2Go**
- - **Lecturio**
- - **LecturioCourse**
- - **LecturioDeCourse**
  - **LEGO**
  - **Lemonde**
- - **Lenta**
  - **LePlaylist**
  - **LetvCloud**: 乐视云
  - **Libsyn**
@@ -455,10 +422,6 @@
  - **limelight**
  - **limelight:channel**
  - **limelight:channel_list**
- - **LineTV**
- - **linkedin:learning**
- - **linkedin:learning:course**
- - **LinuxAcademy**
  - **LiTV**
  - **LiveLeak**
  - **LiveLeakEmbed**
@@ -474,15 +437,11 @@
  - **m6**
  - **macgamestore**: MacGameStore trailers
  - **mailru**: Видео@Mail.Ru
- - **mailru:music**: Музыка@Mail.Ru
- - **mailru:music:search**: Музыка@Mail.Ru
+ - **MakersChannel**
  - **MakerTV**
- - **MallTV**
  - **mangomolo:live**
  - **mangomolo:video**
  - **ManyVids**
- - **Markiza**
- - **MarkizaPage**
  - **massengeschmack.tv**
  - **MatchTV**
  - **MDR**: MDR.DE and KiKA
@@ -513,13 +472,12 @@
  - **Mixer:vod**
  - **MLB**
  - **Mnet**
- - **MNetTV**
  - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
  - **Mofosex**
  - **Mojvideo**
+ - **Moniker**: allmyvideos.net and vidspot.net
  - **Morningstar**: morningstar.com
  - **Motherless**
- - **MotherlessGroup**
  - **Motorsport**: motorsport.com
  - **MovieClips**
  - **MovieFap**
@@ -538,25 +496,21 @@
  - **mva:course**: Microsoft Virtual Academy courses
  - **Mwave**
  - **MwaveMeetGreet**
- - **MyChannels**
  - **MySpace**
  - **MySpace:album**
  - **MySpass**
  - **Myvi**
  - **MyVidster**
- - **MyviEmbed**
- - **MyVisionTV**
  - **n-tv.de**
+ - **natgeo**
+ - **natgeo:episodeguide**
  - **natgeo:video**
- - **NationalGeographicTV**
  - **Naver**
  - **NBA**
  - **NBC**
  - **NBCNews**
- - **nbcolympics**
- - **nbcolympics:stream**
+ - **NBCOlympics**
  - **NBCSports**
- - **NBCSportsStream**
  - **NBCSportsVPlayer**
  - **ndr**: NDR.de - Norddeutscher Rundfunk
  - **ndr:embed**
@@ -570,7 +524,6 @@
  - **netease:program**: 网易云音乐 - 电台节目
  - **netease:singer**: 网易云音乐 - 歌手
  - **netease:song**: 网易云音乐
- - **NetPlus**
  - **Netzkino**
  - **Newgrounds**
  - **NewgroundsPlaylist**
@@ -584,6 +537,9 @@
  - **nfl.com**
  - **NhkVod**
  - **nhl.com**
+ - **nhl.com:news**: NHL news
+ - **nhl.com:videocenter**
+ - **nhl.com:videocenter:category**: NHL videocenter category
  - **nick.com**
  - **nick.de**
  - **nickelodeon:br**
@@ -602,7 +558,6 @@
  - **Normalboots**
  - **NosVideo**
  - **Nova**: TN.cz, Prásk.tv, Nova.cz, Novaplus.cz, FANDA.tv, Krásná.cz and Doma.cz
- - **NovaEmbed**
  - **nowness**
  - **nowness:playlist**
  - **nowness:series**
@@ -618,9 +573,7 @@
  - **NRKSkole**: NRK Skole
  - **NRKTV**: NRK TV and NRK Radio
  - **NRKTVDirekte**: NRK TV Direkte and NRK Radio Direkte
- - **NRKTVEpisode**
  - **NRKTVEpisodes**
- - **NRKTVSeason**
  - **NRKTVSeries**
  - **ntv.ru**
  - **Nuvid**
@@ -647,19 +600,15 @@
  - **orf:iptv**: iptv.ORF.at
  - **orf:oe1**: Radio Österreich 1
  - **orf:tvthek**: ORF TVthek
- - **OsnatelTV**
- - **OutsideTV**
  - **PacktPub**
  - **PacktPubCourse**
  - **PandaTV**: 熊猫TV
  - **pandora.tv**: 판도라TV
- - **ParamountNetwork**
  - **parliamentlive.tv**: UK parliament videos
  - **Patreon**
  - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
  - **pcmag**
  - **PearVideo**
- - **PeerTube**
  - **People**
  - **PerformGroup**
  - **periscope**: Periscope
@@ -667,13 +616,10 @@
  - **PhilharmonieDeParis**: Philharmonie de Paris
  - **phoenix.de**
  - **Photobucket**
- - **Picarto**
- - **PicartoVod**
  - **Piksel**
  - **Pinkbike**
  - **Pladform**
  - **play.fm**
- - **PlayPlusTV**
  - **PlaysTV**
  - **Playtvak**: Playtvak.cz, iDNES.cz and Lidovky.cz
  - **Playvid**
@@ -701,8 +647,6 @@
  - **PrimeShareTV**
  - **PromptFile**
  - **prosiebensat1**: ProSiebenSat.1 Digital
- - **puhutv**
- - **puhutv:serie**
  - **Puls4**
  - **Pyvideo**
  - **qqmusic**: QQ音乐
@@ -710,9 +654,6 @@
  - **qqmusic:playlist**: QQ音乐 - 歌单
  - **qqmusic:singer**: QQ音乐 - 歌手
  - **qqmusic:toplist**: QQ音乐 - 排行榜
- - **QuantumTV**
- - **Quickline**
- - **QuicklineLive**
  - **R7**
  - **R7Article**
  - **radio.de**
@@ -725,8 +666,6 @@
  - **RaiPlay**
  - **RaiPlayLive**
  - **RaiPlayPlaylist**
- - **RayWenderlich**
- - **RayWenderlichCourse**
  - **RBMARadio**
  - **RDS**: RDS.ca
  - **RedBullTV**
@@ -742,6 +681,7 @@
  - **revision**
  - **revision3:embed**
  - **RICE**
+ - **RingTV**
  - **RMCDecouverte**
  - **RockstarGames**
  - **RoosterTeeth**
@@ -762,7 +702,6 @@
  - **rtve.es:live**: RTVE.es live streams
  - **rtve.es:television**
  - **RTVNH**
- - **RTVS**
  - **Rudo**
  - **RUHD**
  - **RulePorn**
@@ -778,8 +717,6 @@
  - **safari**: safaribooksonline.com online video
  - **safari:api**
  - **safari:course**: safaribooksonline.com online courses
- - **SAKTV**
- - **SaltTV**
  - **Sapo**: SAPO Vídeos
  - **savefrom.net**
  - **SBS**: sbs.com.au
@@ -794,8 +731,6 @@
  - **ServingSys**
  - **Servus**
  - **Sexu**
- - **SeznamZpravy**
- - **SeznamZpravyArticle**
  - **Shahid**
  - **ShahidShow**
  - **Shared**: shared.sx
@@ -829,16 +764,15 @@
  - **southpark.nl**
  - **southparkstudios.dk**
  - **SpankBang**
- - **SpankBangPlaylist**
  - **Spankwire**
  - **Spiegel**
  - **Spiegel:Article**: Articles on spiegel.de
  - **Spiegeltv**
- - **sport.francetvinfo.fr**
+ - **Spike**
  - **Sport5**
- - **SportBox**
+ - **SportBoxEmbed**
  - **SportDeutschland**
- - **SpringboardPlatform**
+ - **Sportschau**
  - **Sprout**
  - **sr:mediathek**: Saarländischer Rundfunk
  - **SRGSSR**
@@ -854,9 +788,7 @@
  - **StretchInternet**
  - **SunPorno**
  - **SVT**
- - **SVTPage**
  - **SVTPlay**: SVT Play and Öppet arkiv
- - **SVTSeries**
  - **SWRMediathek**
  - **Syfy**
  - **SztvHu**
@@ -867,8 +799,6 @@
  - **TastyTrade**
  - **TBS**
  - **TDSLifeway**
- - **Teachable**
- - **TeachableCourse**
  - **teachertube**: teachertube.com videos
  - **teachertube:user:collection**: teachertube.com user and collection videos
  - **TeachingChannel**
@@ -877,17 +807,13 @@
  - **techtv.mit.edu**
  - **ted**
  - **Tele13**
- - **Tele5**
  - **TeleBruxelles**
  - **Telecinco**: telecinco.es, cuatro.com and mediaset.es
  - **Telegraaf**
  - **TeleMB**
  - **TeleQuebec**
- - **TeleQuebecEmission**
- - **TeleQuebecLive**
  - **TeleTask**
  - **Telewebion**
- - **TennisTV**
  - **TF1**
  - **TFO**
  - **TheIntercept**
@@ -895,14 +821,13 @@
  - **ThePlatform**
  - **ThePlatformFeed**
  - **TheScene**
+ - **TheSixtyOne**
  - **TheStar**
  - **TheSun**
  - **TheWeatherChannel**
  - **ThisAmericanLife**
  - **ThisAV**
  - **ThisOldHouse**
- - **TikTok**
- - **TikTokUser**
  - **tinypic**: tinypic.com videos
  - **TMZ**
  - **TMZArticle**
@@ -916,7 +841,6 @@
  - **ToypicsUser**: Toypics user profile
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
- - **TruNews**
  - **TruTV**
  - **Tube8**
  - **TubiTv**
@@ -932,6 +856,7 @@
  - **TV2**
  - **tv2.hu**
  - **TV2Article**
+ - **TV3**
  - **TV4**: tv4.se and tv4play.se
  - **TV5MondePlus**: TV5MONDE+
  - **TVA**
@@ -942,20 +867,14 @@
  - **tvigle**: Интернет-телевидение Tvigle.ru
  - **tvland.com**
  - **TVN24**
- - **TVNet**
  - **TVNoe**
  - **TVNow**
- - **TVNowAnnual**
- - **TVNowNew**
- - **TVNowSeason**
- - **TVNowShow**
+ - **TVNowList**
  - **tvp**: Telewizja Polska
  - **tvp:embed**: Telewizja Polska
  - **tvp:series**
  - **TVPlayer**
- - **TVPlayHome**
  - **Tweakers**
- - **TwitCasting**
  - **twitch:chapter**
  - **twitch:clips**
  - **twitch:profile**
@@ -980,6 +899,8 @@
  - **uol.com.br**
  - **uplynk**
  - **uplynk:preplay**
+ - **Upskill**
+ - **UpskillCourse**
  - **Urort**: NRK P3 Urørt
  - **URPlay**
  - **USANetwork**
@@ -998,11 +919,11 @@
  - **VevoPlaylist**
  - **VGTV**: VGTV, BTTV, FTV, Aftenposten and Aftonbladet
  - **vh1.com**
- - **vhx:embed**
  - **Viafree**
  - **vice**
  - **vice:article**
  - **vice:show**
+ - **Viceland**
  - **Vidbit**
  - **Viddler**
  - **Videa**
@@ -1018,7 +939,6 @@
  - **VideoPress**
  - **videoweed**: VideoWeed
  - **Vidio**
- - **VidLii**
  - **vidme**
  - **vidme:user**
  - **vidme:user:likes**
@@ -1043,7 +963,6 @@
  - **Vimple**: Vimple - one-click video hosting
  - **Vine**
  - **vine:user**
- - **Viqeo**
  - **Viu**
  - **viu:ott**
  - **viu:playlist**
@@ -1061,6 +980,7 @@
  - **Voot**
  - **VoxMedia**
  - **VoxMediaVolume**
+ - **Vporn**
  - **vpro**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **Vrak**
  - **VRT**: deredactie.be, sporza.be, cobra.be and cobra.canvas.be
@@ -1068,15 +988,12 @@
  - **vrv**
  - **vrv:series**
  - **VShare**
- - **VTXTV**
  - **vube**: Vube.com
  - **VuClip**
  - **VVVVID**
  - **VyboryMos**
  - **Vzaar**
- - **Wakanim**
  - **Walla**
- - **WalyTV**
  - **washingtonpost**
  - **washingtonpost:article**
  - **wat.tv**
@@ -1084,14 +1001,10 @@
  - **WatchIndianPorn**: Watch Indian Porn
  - **WDR**
  - **wdr:mobile**
- - **WDRElefant**
- - **WDRPage**
  - **Webcaster**
  - **WebcasterFeed**
  - **WebOfStories**
  - **WebOfStoriesPlaylist**
- - **Weibo**
- - **WeiboMobile**
  - **WeiqiTV**: WQTV
  - **wholecloud**: WholeCloud
  - **Wimp**
@@ -1102,7 +1015,6 @@
  - **wrzuta.pl:playlist**
  - **WSJ**: Wall Street Journal
  - **WSJArticle**
- - **WWE**
  - **XBef**
  - **XboxClips**
  - **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To, XVIDSTAGE, Vid ABC, VidBom, vidlo, RapidVideo.TV, FastVideo.me
@@ -1112,8 +1024,6 @@
  - **xiami:artist**: 虾米音乐 - 歌手
  - **xiami:collection**: 虾米音乐 - 精选集
  - **xiami:song**: 虾米音乐
- - **ximalaya**: 喜马拉雅FM
- - **ximalaya:album**: 喜马拉雅FM 专辑
  - **XMinus**
  - **XNXX**
  - **Xstream**
@@ -1127,7 +1037,6 @@
  - **yandexmusic:album**: Яндекс.Музыка - Альбом
  - **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
  - **yandexmusic:track**: Яндекс.Музыка - Трек
- - **YapFiles**
  - **YesJapan**
  - **yinyuetai:video**: 音悦Tai
  - **Ynet**
@@ -1138,7 +1047,6 @@
  - **YouNowLive**
  - **YouNowMoment**
  - **YouPorn**
- - **YourPorn**
  - **YourUpload**
  - **youtube**: YouTube.com
  - **youtube:channel**: YouTube.com channels
@@ -1157,9 +1065,6 @@
  - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
  - **Zapiks**
  - **Zaq1**
- - **Zattoo**
- - **ZattooLive**
  - **ZDF**
  - **ZDFChannel**
  - **zingmp3**: mp3.zing.vn
- - **Zype**

setup.cfg

@@ -2,5 +2,5 @@
 universal = True
 
 [flake8]
-exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv
-ignore = E402,E501,E731,E741
+exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git
+ignore = E402,E501,E731

setup.py

@@ -124,8 +124,6 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
         'License :: Public Domain',
-        'Programming Language :: Python',
-        'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
@@ -134,13 +132,6 @@ setup(
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        'Programming Language :: Python :: Implementation',
-        'Programming Language :: Python :: Implementation :: CPython',
-        'Programming Language :: Python :: Implementation :: IronPython',
-        'Programming Language :: Python :: Implementation :: Jython',
-        'Programming Language :: Python :: Implementation :: PyPy',
     ],
 
     cmdclass={'build_lazy_extractors': build_lazy_extractors},

test/helper.py

@@ -7,7 +7,6 @@ import json
 import os.path
 import re
 import types
-import ssl
 import sys
 
 import youtube_dl.extractor
@@ -153,27 +152,15 @@ def expect_value(self, got, expected, field):
                 isinstance(got, compat_str),
                 'Expected field %s to be a unicode object, but got value %r of type %r' % (field, got, type(got)))
             got = 'md5:' + md5(got)
-        elif isinstance(expected, compat_str) and re.match(r'^(?:min|max)?count:\d+', expected):
+        elif isinstance(expected, compat_str) and expected.startswith('mincount:'):
             self.assertTrue(
                 isinstance(got, (list, dict)),
                 'Expected field %s to be a list or a dict, but it is of type %s' % (
                     field, type(got).__name__))
-            op, _, expected_num = expected.partition(':')
-            expected_num = int(expected_num)
-            if op == 'mincount':
-                assert_func = assertGreaterEqual
-                msg_tmpl = 'Expected %d items in field %s, but only got %d'
-            elif op == 'maxcount':
-                assert_func = assertLessEqual
-                msg_tmpl = 'Expected maximum %d items in field %s, but got %d'
-            elif op == 'count':
-                assert_func = assertEqual
-                msg_tmpl = 'Expected exactly %d items in field %s, but got %d'
-            else:
-                assert False
-            assert_func(
+            expected_num = int(expected.partition(':')[2])
+            assertGreaterEqual(
                 self, len(got), expected_num,
-                msg_tmpl % (expected_num, field, len(got)))
+                'Expected %d items in field %s, but only got %d' % (expected_num, field, len(got)))
             return
         self.assertEqual(
             expected, got,
@@ -249,20 +236,6 @@ def assertGreaterEqual(self, got, expected, msg=None):
         self.assertTrue(got >= expected, msg)
 
 
-def assertLessEqual(self, got, expected, msg=None):
-    if not (got <= expected):
-        if msg is None:
-            msg = '%r not less than or equal to %r' % (got, expected)
-        self.assertTrue(got <= expected, msg)
-
-
-def assertEqual(self, got, expected, msg=None):
-    if not (got == expected):
-        if msg is None:
-            msg = '%r not equal to %r' % (got, expected)
-        self.assertTrue(got == expected, msg)
-
-
 def expect_warnings(ydl, warnings_re):
     real_warning = ydl.report_warning
 
@@ -271,12 +244,3 @@ def expect_warnings(ydl, warnings_re):
             real_warning(w)
 
     ydl.report_warning = _report_warning
-
-
-def http_server_port(httpd):
-    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
-        # In Jython SSLSocket is not a subclass of socket.socket
-        sock = httpd.socket.sock
-    else:
-        sock = httpd.socket
-    return sock.getsockname()[1]

test/test_InfoExtractor.py

@@ -9,30 +9,11 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL, expect_dict, expect_value, http_server_port
-from youtube_dl.compat import compat_etree_fromstring, compat_http_server
+from test.helper import FakeYDL, expect_dict, expect_value
+from youtube_dl.compat import compat_etree_fromstring
 from youtube_dl.extractor.common import InfoExtractor
 from youtube_dl.extractor import YoutubeIE, get_info_extractor
 from youtube_dl.utils import encode_data_uri, strip_jsonp, ExtractorError, RegexNotFoundError
-import threading
-
-
-TEAPOT_RESPONSE_STATUS = 418
-TEAPOT_RESPONSE_BODY = "<h1>418 I'm a teapot</h1>"
-
-
-class InfoExtractorTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
-    def log_message(self, format, *args):
-        pass
-
-    def do_GET(self):
-        if self.path == '/teapot':
-            self.send_response(TEAPOT_RESPONSE_STATUS)
-            self.send_header('Content-Type', 'text/html; charset=utf-8')
-            self.end_headers()
-            self.wfile.write(TEAPOT_RESPONSE_BODY.encode())
-        else:
-            assert False
 
 
 class TestIE(InfoExtractor):
@@ -61,7 +42,6 @@ class TestInfoExtractor(unittest.TestCase):
             <meta content='Foo' property=og:foobar>
             <meta name="og:test1" content='foo > < bar'/>
             <meta name="og:test2" content="foo >//< bar"/>
-            <meta property=og-test3 content='Ill-formatted opengraph'/>
             '''
         self.assertEqual(ie._og_search_title(html), 'Foo')
         self.assertEqual(ie._og_search_description(html), 'Some video\'s description ')
@@ -70,7 +50,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertEqual(ie._og_search_property('foobar', html), 'Foo')
         self.assertEqual(ie._og_search_property('test1', html), 'foo > < bar')
         self.assertEqual(ie._og_search_property('test2', html), 'foo >//< bar')
-        self.assertEqual(ie._og_search_property('test3', html), 'Ill-formatted opengraph')
         self.assertEqual(ie._og_search_property(('test0', 'test1'), html), 'foo > < bar')
         self.assertRaises(RegexNotFoundError, ie._og_search_property, 'test0', html, None, fatal=True)
         self.assertRaises(RegexNotFoundError, ie._og_search_property, ('test0', 'test00'), html, None, fatal=True)
@@ -499,64 +478,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 1280,
                     'height': 720,
                 }]
-            ),
-            (
-                # https://github.com/rg3/youtube-dl/issues/18923
-                # https://www.ted.com/talks/boris_hesser_a_grassroots_healthcare_revolution_in_africa
-                'ted_18923',
-                'http://hls.ted.com/talks/31241.m3u8',
-                [{
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '600k-Audio',
-                    'vcodec': 'none',
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '68',
-                    'vcodec': 'none',
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/64k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '163',
-                    'acodec': 'none',
-                    'width': 320,
-                    'height': 180,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/180k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '481',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/320k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '769',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/450k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '984',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '1255',
-                    'acodec': 'none',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/950k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '1693',
-                    'acodec': 'none',
-                    'width': 853,
-                    'height': 480,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/1500k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '2462',
-                    'acodec': 'none',
-                    'width': 1280,
-                    'height': 720,
-                }]
-            ),
+            )
         ]
 
         for m3u8_file, m3u8_url, expected_formats in _TEST_CASES:
@@ -772,74 +694,6 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
 
-    def test_parse_xspf(self):
-        _TEST_CASES = [
-            (
-                'foo_xspf',
-                'https://example.org/src/foo_xspf.xspf',
-                [{
-                    'id': 'foo_xspf',
-                    'title': 'Pandemonium',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 202.416,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/src/cd1/track%201.mp3',
-                    }],
-                }, {
-                    'id': 'foo_xspf',
-                    'title': 'Final Cartridge (Nichico Twelve Remix)',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 255.857,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/%E3%83%88%E3%83%A9%E3%83%83%E3%82%AF%E3%80%80%EF%BC%92.mp3',
-                    }],
-                }, {
-                    'id': 'foo_xspf',
-                    'title': 'Rebuilding Nightingale',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 287.915,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/src/track3.mp3',
-                    }, {
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.com/track3.mp3',
-                    }]
-                }]
-            ),
-        ]
-
-        for xspf_file, xspf_url, expected_entries in _TEST_CASES:
-            with io.open('./test/testdata/xspf/%s.xspf' % xspf_file,
-                         mode='r', encoding='utf-8') as f:
-                entries = self.ie._parse_xspf(
-                    compat_etree_fromstring(f.read().encode('utf-8')),
-                    xspf_file, xspf_url=xspf_url, xspf_base_url=xspf_url)
-                expect_value(self, entries, expected_entries, None)
-                for i in range(len(entries)):
-                    expect_dict(self, entries[i], expected_entries[i])
-
-    def test_response_with_expected_status_returns_content(self):
-        # Checks for mitigations against the effects of
-        # <https://bugs.python.org/issue15002> that affect Python 3.4.1+, which
-        # manifest as `_download_webpage`, `_download_xml`, `_download_json`,
-        # or the underlying `_download_webpage_handle` returning no content
-        # when a response matches `expected_status`.
-
-        httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), InfoExtractorTestRequestHandler)
-        port = http_server_port(httpd)
-        server_thread = threading.Thread(target=httpd.serve_forever)
-        server_thread.daemon = True
-        server_thread.start()
-
-        (content, urlh) = self.ie._download_webpage_handle(
-            'http://127.0.0.1:%d/teapot' % port, None,
-            expected_status=TEAPOT_RESPONSE_STATUS)
-        self.assertEqual(content, TEAPOT_RESPONSE_BODY)
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_YoutubeDL.py

@@ -239,76 +239,6 @@ class TestFormatSelection(unittest.TestCase):
         downloaded = ydl.downloaded_info_dicts[0]
         self.assertEqual(downloaded['format_id'], 'vid-vcodec-dot')
 
-    def test_format_selection_string_ops(self):
-        formats = [
-            {'format_id': 'abc-cba', 'ext': 'mp4', 'url': TEST_URL},
-            {'format_id': 'zxc-cxz', 'ext': 'webm', 'url': TEST_URL},
-        ]
-        info_dict = _make_result(formats)
-
-        # equals (=)
-        ydl = YDL({'format': '[format_id=abc-cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not equal (!=)
-        ydl = YDL({'format': '[format_id!=abc-cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!=abc-cba][format_id!=zxc-cxz]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # starts with (^=)
-        ydl = YDL({'format': '[format_id^=abc]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not start with (!^=)
-        ydl = YDL({'format': '[format_id!^=abc]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!^=abc][format_id!^=zxc]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # ends with ($=)
-        ydl = YDL({'format': '[format_id$=cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not end with (!$=)
-        ydl = YDL({'format': '[format_id!$=cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!$=cba][format_id!$=cxz]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # contains (*=)
-        ydl = YDL({'format': '[format_id*=bc-cb]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not contain (!*=)
-        ydl = YDL({'format': '[format_id!*=bc-cb]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!*=abc][format_id!*=zxc]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        ydl = YDL({'format': '[format_id!*=-]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
     def test_youtube_format_selection(self):
         order = [
             '38', '37', '46', '22', '45', '35', '44', '18', '34', '43', '6', '5', '17', '36', '13',

test/test_YoutubeDLCookieJar.py

@@ -1,34 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-from __future__ import unicode_literals
-
-import os
-import re
-import sys
-import tempfile
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from youtube_dl.utils import YoutubeDLCookieJar
-
-
-class TestYoutubeDLCookieJar(unittest.TestCase):
-    def test_keep_session_cookies(self):
-        cookiejar = YoutubeDLCookieJar('./test/testdata/cookies/session_cookies.txt')
-        cookiejar.load(ignore_discard=True, ignore_expires=True)
-        tf = tempfile.NamedTemporaryFile(delete=False)
-        try:
-            cookiejar.save(filename=tf.name, ignore_discard=True, ignore_expires=True)
-            temp = tf.read().decode('utf-8')
-            self.assertTrue(re.search(
-                r'www\.foobar\.foobar\s+FALSE\s+/\s+TRUE\s+0\s+YoutubeDLExpiresEmpty\s+YoutubeDLExpiresEmptyValue', temp))
-            self.assertTrue(re.search(
-                r'www\.foobar\.foobar\s+FALSE\s+/\s+TRUE\s+0\s+YoutubeDLExpires0\s+YoutubeDLExpires0Value', temp))
-        finally:
-            tf.close()
-            os.remove(tf.name)
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_compat.py

@@ -39,7 +39,7 @@ class TestCompat(unittest.TestCase):
 
     def test_compat_expanduser(self):
         old_home = os.environ.get('HOME')
-        test_str = r'C:\Documents and Settings\тест\Application Data'
+        test_str = 'C:\Documents and Settings\тест\Application Data'
         compat_setenv('HOME', test_str)
         self.assertEqual(compat_expanduser('~'), test_str)
         compat_setenv('HOME', old_home or '')

test/test_download.py

@@ -92,8 +92,8 @@ class TestDownload(unittest.TestCase):
 def generator(test_case, tname):
 
     def test_template(self):
-        ie = youtube_dl.extractor.get_info_extractor(test_case['name'])()
-        other_ies = [get_info_extractor(ie_key)() for ie_key in test_case.get('add_ie', [])]
+        ie = youtube_dl.extractor.get_info_extractor(test_case['name'])
+        other_ies = [get_info_extractor(ie_key) for ie_key in test_case.get('add_ie', [])]
         is_playlist = any(k.startswith('playlist') for k in test_case)
         test_cases = test_case.get(
             'playlist', [] if is_playlist else [test_case])

test/test_downloader_http.py

@@ -1,115 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-from __future__ import unicode_literals
-
-# Allow direct execution
-import os
-import re
-import sys
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from test.helper import http_server_port, try_rm
-from youtube_dl import YoutubeDL
-from youtube_dl.compat import compat_http_server
-from youtube_dl.downloader.http import HttpFD
-from youtube_dl.utils import encodeFilename
-import threading
-
-TEST_DIR = os.path.dirname(os.path.abspath(__file__))
-
-
-TEST_SIZE = 10 * 1024
-
-
-class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
-    def log_message(self, format, *args):
-        pass
-
-    def send_content_range(self, total=None):
-        range_header = self.headers.get('Range')
-        start = end = None
-        if range_header:
-            mobj = re.search(r'^bytes=(\d+)-(\d+)', range_header)
-            if mobj:
-                start = int(mobj.group(1))
-                end = int(mobj.group(2))
-        valid_range = start is not None and end is not None
-        if valid_range:
-            content_range = 'bytes %d-%d' % (start, end)
-            if total:
-                content_range += '/%d' % total
-            self.send_header('Content-Range', content_range)
-        return (end - start + 1) if valid_range else total
-
-    def serve(self, range=True, content_length=True):
-        self.send_response(200)
-        self.send_header('Content-Type', 'video/mp4')
-        size = TEST_SIZE
-        if range:
-            size = self.send_content_range(TEST_SIZE)
-        if content_length:
-            self.send_header('Content-Length', size)
-        self.end_headers()
-        self.wfile.write(b'#' * size)
-
-    def do_GET(self):
-        if self.path == '/regular':
-            self.serve()
-        elif self.path == '/no-content-length':
-            self.serve(content_length=False)
-        elif self.path == '/no-range':
-            self.serve(range=False)
-        elif self.path == '/no-range-no-content-length':
-            self.serve(range=False, content_length=False)
-        else:
-            assert False
-
-
-class FakeLogger(object):
-    def debug(self, msg):
-        pass
-
-    def warning(self, msg):
-        pass
-
-    def error(self, msg):
-        pass
-
-
-class TestHttpFD(unittest.TestCase):
-    def setUp(self):
-        self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
-        self.port = http_server_port(self.httpd)
-        self.server_thread = threading.Thread(target=self.httpd.serve_forever)
-        self.server_thread.daemon = True
-        self.server_thread.start()
-
-    def download(self, params, ep):
-        params['logger'] = FakeLogger()
-        ydl = YoutubeDL(params)
-        downloader = HttpFD(ydl, params)
-        filename = 'testfile.mp4'
-        try_rm(encodeFilename(filename))
-        self.assertTrue(downloader.real_download(filename, {
-            'url': 'http://127.0.0.1:%d/%s' % (self.port, ep),
-        }))
-        self.assertEqual(os.path.getsize(encodeFilename(filename)), TEST_SIZE)
-        try_rm(encodeFilename(filename))
-
-    def download_all(self, params):
-        for ep in ('regular', 'no-content-length', 'no-range', 'no-range-no-content-length'):
-            self.download(params, ep)
-
-    def test_regular(self):
-        self.download_all({})
-
-    def test_chunked(self):
-        self.download_all({
-            'http_chunk_size': 1000,
-        })
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_http.py

@@ -8,7 +8,6 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import http_server_port
 from youtube_dl import YoutubeDL
 from youtube_dl.compat import compat_http_server, compat_urllib_request
 import ssl
@@ -17,6 +16,15 @@ import threading
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 
 
+def http_server_port(httpd):
+    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
+        # In Jython SSLSocket is not a subclass of socket.socket
+        sock = httpd.socket.sock
+    else:
+        sock = httpd.socket
+    return sock.getsockname()[1]
+
+
 class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
     def log_message(self, format, *args):
         pass
@@ -39,7 +47,7 @@ class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
                 self.end_headers()
                 return
 
-            new_url = 'http://127.0.0.1:%d/中文.html' % http_server_port(self.server)
+            new_url = 'http://localhost:%d/中文.html' % http_server_port(self.server)
             self.send_response(302)
             self.send_header(b'Location', new_url.encode('utf-8'))
             self.end_headers()
@@ -66,7 +74,7 @@ class FakeLogger(object):
 class TestHTTP(unittest.TestCase):
     def setUp(self):
         self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
+            ('localhost', 0), HTTPTestRequestHandler)
         self.port = http_server_port(self.httpd)
         self.server_thread = threading.Thread(target=self.httpd.serve_forever)
         self.server_thread.daemon = True
@@ -78,15 +86,15 @@ class TestHTTP(unittest.TestCase):
             return
 
         ydl = YoutubeDL({'logger': FakeLogger()})
-        r = ydl.extract_info('http://127.0.0.1:%d/302' % self.port)
-        self.assertEqual(r['entries'][0]['url'], 'http://127.0.0.1:%d/vid.mp4' % self.port)
+        r = ydl.extract_info('http://localhost:%d/302' % self.port)
+        self.assertEqual(r['entries'][0]['url'], 'http://localhost:%d/vid.mp4' % self.port)
 
 
 class TestHTTPS(unittest.TestCase):
     def setUp(self):
         certfn = os.path.join(TEST_DIR, 'testcert.pem')
         self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
+            ('localhost', 0), HTTPTestRequestHandler)
         self.httpd.socket = ssl.wrap_socket(
             self.httpd.socket, certfile=certfn, server_side=True)
         self.port = http_server_port(self.httpd)
@@ -99,11 +107,11 @@ class TestHTTPS(unittest.TestCase):
             ydl = YoutubeDL({'logger': FakeLogger()})
             self.assertRaises(
                 Exception,
-                ydl.extract_info, 'https://127.0.0.1:%d/video.html' % self.port)
+                ydl.extract_info, 'https://localhost:%d/video.html' % self.port)
 
         ydl = YoutubeDL({'logger': FakeLogger(), 'nocheckcertificate': True})
-        r = ydl.extract_info('https://127.0.0.1:%d/video.html' % self.port)
-        self.assertEqual(r['entries'][0]['url'], 'https://127.0.0.1:%d/vid.mp4' % self.port)
+        r = ydl.extract_info('https://localhost:%d/video.html' % self.port)
+        self.assertEqual(r['entries'][0]['url'], 'https://localhost:%d/vid.mp4' % self.port)
 
 
 def _build_proxy_handler(name):
@@ -124,23 +132,23 @@ def _build_proxy_handler(name):
 class TestProxy(unittest.TestCase):
     def setUp(self):
         self.proxy = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), _build_proxy_handler('normal'))
+            ('localhost', 0), _build_proxy_handler('normal'))
         self.port = http_server_port(self.proxy)
         self.proxy_thread = threading.Thread(target=self.proxy.serve_forever)
         self.proxy_thread.daemon = True
         self.proxy_thread.start()
 
         self.geo_proxy = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), _build_proxy_handler('geo'))
+            ('localhost', 0), _build_proxy_handler('geo'))
         self.geo_port = http_server_port(self.geo_proxy)
         self.geo_proxy_thread = threading.Thread(target=self.geo_proxy.serve_forever)
         self.geo_proxy_thread.daemon = True
         self.geo_proxy_thread.start()
 
     def test_proxy(self):
-        geo_proxy = '127.0.0.1:{0}'.format(self.geo_port)
+        geo_proxy = 'localhost:{0}'.format(self.geo_port)
         ydl = YoutubeDL({
-            'proxy': '127.0.0.1:{0}'.format(self.port),
+            'proxy': 'localhost:{0}'.format(self.port),
             'geo_verification_proxy': geo_proxy,
         })
         url = 'http://foo.com/bar'
@@ -154,7 +162,7 @@ class TestProxy(unittest.TestCase):
 
     def test_proxy_with_idn(self):
         ydl = YoutubeDL({
-            'proxy': '127.0.0.1:{0}'.format(self.port),
+            'proxy': 'localhost:{0}'.format(self.port),
         })
         url = 'http://中文.tw/'
         response = ydl.urlopen(url).read().decode('utf-8')

test/test_postprocessors.py

@@ -14,4 +14,4 @@ from youtube_dl.postprocessor import MetadataFromTitlePP
 class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(pp._titleregex, '(?P<title>.+)\ \-\ (?P<artist>.+)')

test/test_subtitles.py

@@ -232,7 +232,7 @@ class TestNPOSubtitles(BaseTestSubtitles):
 
 
 class TestMTVSubtitles(BaseTestSubtitles):
-    url = 'http://www.cc.com/video-clips/p63lk0/adam-devine-s-house-party-chasing-white-swans'
+    url = 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother'
     IE = ComedyCentralIE
 
     def getInfoDict(self):
@@ -243,7 +243,7 @@ class TestMTVSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['en']))
-        self.assertEqual(md5(subtitles['en']), '78206b8d8a0cfa9da64dc026eea48961')
+        self.assertEqual(md5(subtitles['en']), 'b9f6ca22a6acf597ec76f61749765e65')
 
 
 class TestNRKSubtitles(BaseTestSubtitles):

test/test_utils.py

@@ -42,7 +42,6 @@ from youtube_dl.utils import (
     is_html,
     js_to_json,
     limit_length,
-    merge_dicts,
     mimetype2ext,
     month_by_name,
     multipart_encode,
@@ -54,12 +53,10 @@ from youtube_dl.utils import (
     parse_filesize,
     parse_count,
     parse_iso8601,
-    parse_resolution,
     pkcs1pad,
     read_batch_urls,
     sanitize_filename,
     sanitize_path,
-    sanitize_url,
     expand_path,
     prepend_extension,
     replace_extension,
@@ -78,7 +75,6 @@ from youtube_dl.utils import (
     uppercase_escape,
     lowercase_escape,
     url_basename,
-    url_or_none,
     base_url,
     urljoin,
     urlencode_postdata,
@@ -223,12 +219,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(sanitize_path('./abc'), 'abc')
         self.assertEqual(sanitize_path('./../abc'), '..\\abc')
 
-    def test_sanitize_url(self):
-        self.assertEqual(sanitize_url('//foo.bar'), 'http://foo.bar')
-        self.assertEqual(sanitize_url('httpss://foo.bar'), 'https://foo.bar')
-        self.assertEqual(sanitize_url('rmtps://foo.bar'), 'rtmps://foo.bar')
-        self.assertEqual(sanitize_url('https://foo.bar'), 'https://foo.bar')
-
     def test_expand_path(self):
         def env(var):
             return '%{0}%'.format(var) if sys.platform == 'win32' else '${0}'.format(var)
@@ -354,7 +344,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(unified_timestamp('2017-03-30T17:52:41Q'), 1490896361)
         self.assertEqual(unified_timestamp('Sep 11, 2013 | 5:49 AM'), 1378878540)
         self.assertEqual(unified_timestamp('December 15, 2017 at 7:49 am'), 1513324140)
-        self.assertEqual(unified_timestamp('2018-03-14T08:32:43.1493874+00:00'), 1521016363)
 
     def test_determine_ext(self):
         self.assertEqual(determine_ext('http://example.com/foo/bar.mp4/?download'), 'mp4')
@@ -362,7 +351,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(determine_ext('http://example.com/foo/bar.nonext/?download', None), None)
         self.assertEqual(determine_ext('http://example.com/foo/bar/mp4?download', None), None)
         self.assertEqual(determine_ext('http://example.com/foo/bar.m3u8//?download'), 'm3u8')
-        self.assertEqual(determine_ext('foobar', None), None)
 
     def test_find_xpath_attr(self):
         testxml = '''<root>
@@ -507,18 +495,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(urljoin('http://foo.de/', ''), None)
         self.assertEqual(urljoin('http://foo.de/', ['foobar']), None)
         self.assertEqual(urljoin('http://foo.de/a/b/c.txt', '.././../d.txt'), 'http://foo.de/d.txt')
-        self.assertEqual(urljoin('http://foo.de/a/b/c.txt', 'rtmp://foo.de'), 'rtmp://foo.de')
-        self.assertEqual(urljoin(None, 'rtmp://foo.de'), 'rtmp://foo.de')
-
-    def test_url_or_none(self):
-        self.assertEqual(url_or_none(None), None)
-        self.assertEqual(url_or_none(''), None)
-        self.assertEqual(url_or_none('foo'), None)
-        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
-        self.assertEqual(url_or_none('https://foo.de'), 'https://foo.de')
-        self.assertEqual(url_or_none('http$://foo.de'), None)
-        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
-        self.assertEqual(url_or_none('//foo.de'), '//foo.de')
 
     def test_parse_age_limit(self):
         self.assertEqual(parse_age_limit(None), None)
@@ -533,8 +509,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_age_limit('PG-13'), 13)
         self.assertEqual(parse_age_limit('TV-14'), 14)
         self.assertEqual(parse_age_limit('TV-MA'), 17)
-        self.assertEqual(parse_age_limit('TV14'), 14)
-        self.assertEqual(parse_age_limit('TV_G'), 0)
 
     def test_parse_duration(self):
         self.assertEqual(parse_duration(None), None)
@@ -686,17 +660,6 @@ class TestUtil(unittest.TestCase):
             self.assertEqual(dict_get(d, ('b', 'c', key, )), None)
             self.assertEqual(dict_get(d, ('b', 'c', key, ), skip_false_values=False), false_value)
 
-    def test_merge_dicts(self):
-        self.assertEqual(merge_dicts({'a': 1}, {'b': 2}), {'a': 1, 'b': 2})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': 2}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': None}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': ''}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': None}, {'a': 1}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': ''}, {'a': 1}), {'a': ''})
-        self.assertEqual(merge_dicts({'a': ''}, {'a': 'abc'}), {'a': 'abc'})
-        self.assertEqual(merge_dicts({'a': None}, {'a': ''}, {'a': 'abc'}), {'a': 'abc'})
-
     def test_encode_compat_str(self):
         self.assertEqual(encode_compat_str(b'\xd1\x82\xd0\xb5\xd1\x81\xd1\x82', 'utf-8'), 'тест')
         self.assertEqual(encode_compat_str('тест', 'utf-8'), 'тест')
@@ -730,10 +693,6 @@ class TestUtil(unittest.TestCase):
         d = json.loads(stripped)
         self.assertEqual(d, {'status': 'success'})
 
-        stripped = strip_jsonp('({"status": "success"});')
-        d = json.loads(stripped)
-        self.assertEqual(d, {'status': 'success'})
-
     def test_uppercase_escape(self):
         self.assertEqual(uppercase_escape('aä'), 'aä')
         self.assertEqual(uppercase_escape('\\U0001d550'), '𝕐')
@@ -787,10 +746,6 @@ class TestUtil(unittest.TestCase):
             'vcodec': 'h264',
             'acodec': 'aac',
         })
-        self.assertEqual(parse_codecs('av01.0.05M.08'), {
-            'vcodec': 'av01.0.05M.08',
-            'acodec': 'none',
-        })
 
     def test_escape_rfc3986(self):
         reserved = "!*'();:@&=+$,/?#[]"
@@ -859,9 +814,6 @@ class TestUtil(unittest.TestCase):
         inp = '''{"duration": "00:01:07"}'''
         self.assertEqual(js_to_json(inp), '''{"duration": "00:01:07"}''')
 
-        inp = '''{segments: [{"offset":-3.885780586188048e-16,"duration":39.75000000000001}]}'''
-        self.assertEqual(js_to_json(inp), '''{"segments": [{"offset":-3.885780586188048e-16,"duration":39.75000000000001}]}''')
-
     def test_js_to_json_edgecases(self):
         on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
         self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
@@ -933,13 +885,6 @@ class TestUtil(unittest.TestCase):
         on = js_to_json('{/*comment\n*/42/*comment\n*/:/*comment\n*/42/*comment\n*/}')
         self.assertEqual(json.loads(on), {'42': 42})
 
-        on = js_to_json('{42:4.2e1}')
-        self.assertEqual(json.loads(on), {'42': 42.0})
-
-    def test_js_to_json_malformed(self):
-        self.assertEqual(js_to_json('42a1'), '42"a1"')
-        self.assertEqual(js_to_json('42a-1'), '42"a"-1')
-
     def test_extract_attributes(self):
         self.assertEqual(extract_attributes('<e x="y">'), {'x': 'y'})
         self.assertEqual(extract_attributes("<e x='y'>"), {'x': 'y'})
@@ -1020,16 +965,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_count('1.1kk '), 1100000)
         self.assertEqual(parse_count('1.1kk views'), 1100000)
 
-    def test_parse_resolution(self):
-        self.assertEqual(parse_resolution(None), {})
-        self.assertEqual(parse_resolution(''), {})
-        self.assertEqual(parse_resolution('1920x1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920×1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920 x 1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('720p'), {'height': 720})
-        self.assertEqual(parse_resolution('4k'), {'height': 2160})
-        self.assertEqual(parse_resolution('8K'), {'height': 4320})
-
     def test_version_tuple(self):
         self.assertEqual(version_tuple('1'), (1,))
         self.assertEqual(version_tuple('10.23.344'), (10, 23, 344))
@@ -1108,18 +1043,6 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
         self.assertFalse(match_str(
             'like_count > 100 & dislike_count <? 50 & description',
             {'like_count': 190, 'dislike_count': 10}))
-        self.assertTrue(match_str('is_live', {'is_live': True}))
-        self.assertFalse(match_str('is_live', {'is_live': False}))
-        self.assertFalse(match_str('is_live', {'is_live': None}))
-        self.assertFalse(match_str('is_live', {}))
-        self.assertFalse(match_str('!is_live', {'is_live': True}))
-        self.assertTrue(match_str('!is_live', {'is_live': False}))
-        self.assertTrue(match_str('!is_live', {'is_live': None}))
-        self.assertTrue(match_str('!is_live', {}))
-        self.assertTrue(match_str('title', {'title': 'abc'}))
-        self.assertTrue(match_str('title', {'title': ''}))
-        self.assertFalse(match_str('!title', {'title': 'abc'}))
-        self.assertFalse(match_str('!title', {'title': ''}))
 
     def test_parse_dfxp_time_expr(self):
         self.assertEqual(parse_dfxp_time_expr(None), None)

test/test_youtube_lists.py

@@ -61,7 +61,7 @@ class TestYoutubeLists(unittest.TestCase):
         dl = FakeYDL()
         dl.params['extract_flat'] = True
         ie = YoutubePlaylistIE(dl)
-        result = ie.extract('https://www.youtube.com/playlist?list=PL-KKIb8rvtMSrAO9YFbeM6UQrAqoFTUWv')
+        result = ie.extract('https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re')
         self.assertIsPlaylist(result)
         for entry in result['entries']:
             self.assertTrue(entry.get('title'))

test/testdata/cookies/session_cookies.txt

@@ -1,6 +0,0 @@
-# Netscape HTTP Cookie File
-# http://curl.haxx.se/rfc/cookie_spec.html
-# This is a generated file!  Do not edit.
-
-www.foobar.foobar	FALSE	/	TRUE		YoutubeDLExpiresEmpty	YoutubeDLExpiresEmptyValue
-www.foobar.foobar	FALSE	/	TRUE	0	YoutubeDLExpires0	YoutubeDLExpires0Value

test/testdata/m3u8/ted_18923.m3u8

@@ -1,28 +0,0 @@
-#EXTM3U
-#EXT-X-VERSION:4
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=1255659,PROGRAM-ID=1,CODECS="avc1.42c01e,mp4a.40.2",RESOLUTION=640x360
-/videos/BorisHesser_2018S/video/600k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=163154,PROGRAM-ID=1,CODECS="avc1.42c00c,mp4a.40.2",RESOLUTION=320x180
-/videos/BorisHesser_2018S/video/64k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=481701,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/180k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=769968,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/320k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=984037,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/450k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=1693925,PROGRAM-ID=1,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=853x480
-/videos/BorisHesser_2018S/video/950k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=2462469,PROGRAM-ID=1,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1280x720
-/videos/BorisHesser_2018S/video/1500k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=68101,PROGRAM-ID=1,CODECS="mp4a.40.2",DEFAULT=YES
-/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b
-
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=74298,PROGRAM-ID=1,CODECS="avc1.42c00c",RESOLUTION=320x180,URI="/videos/BorisHesser_2018S/video/64k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=216200,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/180k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=304717,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/320k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=350933,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/450k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=495850,PROGRAM-ID=1,CODECS="avc1.42c01e",RESOLUTION=640x360,URI="/videos/BorisHesser_2018S/video/600k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=810750,PROGRAM-ID=1,CODECS="avc1.4d401f",RESOLUTION=853x480,URI="/videos/BorisHesser_2018S/video/950k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1273700,PROGRAM-ID=1,CODECS="avc1.640028",RESOLUTION=1280x720,URI="/videos/BorisHesser_2018S/video/1500k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-
-#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="600k",LANGUAGE="en",NAME="Audio",AUTOSELECT=YES,DEFAULT=YES,URI="/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b",BANDWIDTH=614400

test/testdata/xspf/foo_xspf.xspf

@@ -1,34 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<playlist version="1" xmlns="http://xspf.org/ns/0/">
-    <date>2018-03-09T18:01:43Z</date>
-    <trackList>
-        <track>
-            <location>cd1/track%201.mp3</location>
-            <title>Pandemonium</title>
-            <creator>Foilverb</creator>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <album>Pandemonium EP</album>
-            <trackNum>1</trackNum>
-            <duration>202416</duration>
-        </track>
-        <track>
-            <location>../%E3%83%88%E3%83%A9%E3%83%83%E3%82%AF%E3%80%80%EF%BC%92.mp3</location>
-            <title>Final Cartridge (Nichico Twelve Remix)</title>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <creator>Foilverb</creator>
-            <album>Pandemonium EP</album>
-            <trackNum>2</trackNum>
-            <duration>255857</duration>
-        </track>
-        <track>
-            <location>track3.mp3</location>
-            <location>https://example.com/track3.mp3</location>
-            <title>Rebuilding Nightingale</title>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <creator>Foilverb</creator>
-            <album>Pandemonium EP</album>
-            <trackNum>3</trackNum>
-            <duration>287915</duration>
-        </track>
-    </trackList>
-</playlist>

youtube_dl/YoutubeDL.py

@@ -82,14 +82,12 @@ from .utils import (
     sanitize_url,
     sanitized_Request,
     std_headers,
-    str_or_none,
     subtitles_filename,
     UnavailableVideoError,
     url_basename,
     version_tuple,
     write_json_file,
     write_string,
-    YoutubeDLCookieJar,
     YoutubeDLCookieProcessor,
     YoutubeDLHandler,
 )
@@ -213,7 +211,7 @@ class YoutubeDL(object):
                        At the moment, this is only supported by YouTube.
     proxy:             URL of the proxy server to use
     geo_verification_proxy:  URL of the proxy to use for IP address verification
-                       on geo-restricted sites.
+                       on geo-restricted sites. (Experimental)
     socket_timeout:    Time to wait for unresponsive hosts, in seconds
     bidi_workaround:   Work around buggy terminals without bidirectional text
                        support, using fridibi
@@ -261,7 +259,7 @@ class YoutubeDL(object):
                        - "warn": only emit a warning
                        - "detect_or_warn": check whether we can do anything
                                            about it, warn otherwise (default)
-    source_address:    Client-side IP address to bind to.
+    source_address:    (Experimental) Client-side IP address to bind to.
     call_home:         Boolean, true iff we are allowed to contact the
                        youtube-dl servers for debugging.
     sleep_interval:    Number of seconds to sleep before each download when
@@ -283,14 +281,11 @@ class YoutubeDL(object):
                        match_filter_func in utils.py is one example for this.
     no_color:          Do not emit color codes in output.
     geo_bypass:        Bypass geographic restriction via faking X-Forwarded-For
-                       HTTP header
+                       HTTP header (experimental)
     geo_bypass_country:
                        Two-letter ISO 3166-2 country code that will be used for
                        explicit geographic restriction bypassing via faking
-                       X-Forwarded-For HTTP header
-    geo_bypass_ip_block:
-                       IP range in CIDR notation that will be used similarly to
-                       geo_bypass_country
+                       X-Forwarded-For HTTP header (experimental)
 
     The following options determine which downloader is picked:
     external_downloader: Executable of the external downloader to call.
@@ -303,12 +298,11 @@ class YoutubeDL(object):
     the downloader (see youtube_dl/downloader/common.py):
     nopart, updatetime, buffersize, ratelimit, min_filesize, max_filesize, test,
     noresizebuffer, retries, continuedl, noprogress, consoletitle,
-    xattr_set_filesize, external_downloader_args, hls_use_mpegts,
-    http_chunk_size.
+    xattr_set_filesize, external_downloader_args, hls_use_mpegts.
 
     The following options are used by the post processors:
-    prefer_ffmpeg:     If False, use avconv instead of ffmpeg if both are available,
-                       otherwise prefer ffmpeg.
+    prefer_ffmpeg:     If True, use ffmpeg instead of avconv if both are available,
+                       otherwise prefer avconv.
     postprocessor_args: A list of additional command-line arguments for the
                         postprocessor.
 
@@ -537,8 +531,6 @@ class YoutubeDL(object):
     def save_console_title(self):
         if not self.params.get('consoletitle', False):
             return
-        if self.params.get('simulate', False):
-            return
         if compat_os_name != 'nt' and 'TERM' in os.environ:
             # Save the title on stack
             self._write_string('\033[22;0t', self._screen_file)
@@ -546,8 +538,6 @@ class YoutubeDL(object):
     def restore_console_title(self):
         if not self.params.get('consoletitle', False):
             return
-        if self.params.get('simulate', False):
-            return
         if compat_os_name != 'nt' and 'TERM' in os.environ:
             # Restore the title from stack
             self._write_string('\033[23;0t', self._screen_file)
@@ -560,7 +550,7 @@ class YoutubeDL(object):
         self.restore_console_title()
 
         if self.params.get('cookiefile') is not None:
-            self.cookiejar.save(ignore_discard=True, ignore_expires=True)
+            self.cookiejar.save()
 
     def trouble(self, message=None, tb=None):
         """Determine action to take when a download problem appears.
@@ -1042,7 +1032,7 @@ class YoutubeDL(object):
             '!=': operator.ne,
         }
         operator_rex = re.compile(r'''(?x)\s*
-            (?P<key>width|height|tbr|abr|vbr|asr|filesize|filesize_approx|fps)
+            (?P<key>width|height|tbr|abr|vbr|asr|filesize|fps)
             \s*(?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
             (?P<value>[0-9.]+(?:[kKmMgGtTpPeEzZyY]i?[Bb]?)?)
             $
@@ -1064,24 +1054,21 @@ class YoutubeDL(object):
         if not m:
             STR_OPERATORS = {
                 '=': operator.eq,
+                '!=': operator.ne,
                 '^=': lambda attr, value: attr.startswith(value),
                 '$=': lambda attr, value: attr.endswith(value),
                 '*=': lambda attr, value: value in attr,
             }
             str_operator_rex = re.compile(r'''(?x)
                 \s*(?P<key>ext|acodec|vcodec|container|protocol|format_id)
-                \s*(?P<negation>!\s*)?(?P<op>%s)(?P<none_inclusive>\s*\?)?
+                \s*(?P<op>%s)(?P<none_inclusive>\s*\?)?
                 \s*(?P<value>[a-zA-Z0-9._-]+)
                 \s*$
                 ''' % '|'.join(map(re.escape, STR_OPERATORS.keys())))
             m = str_operator_rex.search(filter_spec)
             if m:
                 comparison_value = m.group('value')
-                str_op = STR_OPERATORS[m.group('op')]
-                if m.group('negation'):
-                    op = lambda attr, value: not str_op(attr, value)
-                else:
-                    op = str_op
+                op = STR_OPERATORS[m.group('op')]
 
         if not m:
             raise ValueError('Invalid filter specification %r' % filter_spec)
@@ -1487,28 +1474,23 @@ class YoutubeDL(object):
             if info_dict.get('%s_number' % field) is not None and not info_dict.get(field):
                 info_dict[field] = '%s %d' % (field.capitalize(), info_dict['%s_number' % field])
 
-        for cc_kind in ('subtitles', 'automatic_captions'):
-            cc = info_dict.get(cc_kind)
-            if cc:
-                for _, subtitle in cc.items():
-                    for subtitle_format in subtitle:
-                        if subtitle_format.get('url'):
-                            subtitle_format['url'] = sanitize_url(subtitle_format['url'])
-                        if subtitle_format.get('ext') is None:
-                            subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
-
-        automatic_captions = info_dict.get('automatic_captions')
         subtitles = info_dict.get('subtitles')
+        if subtitles:
+            for _, subtitle in subtitles.items():
+                for subtitle_format in subtitle:
+                    if subtitle_format.get('url'):
+                        subtitle_format['url'] = sanitize_url(subtitle_format['url'])
+                    if subtitle_format.get('ext') is None:
+                        subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
 
         if self.params.get('listsubtitles', False):
             if 'automatic_captions' in info_dict:
-                self.list_subtitles(
-                    info_dict['id'], automatic_captions, 'automatic captions')
+                self.list_subtitles(info_dict['id'], info_dict.get('automatic_captions'), 'automatic captions')
             self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
             return
-
         info_dict['requested_subtitles'] = self.process_subtitles(
-            info_dict['id'], subtitles, automatic_captions)
+            info_dict['id'], subtitles,
+            info_dict.get('automatic_captions'))
 
         # We now pick which formats have to be downloaded
         if info_dict.get('formats') is None:
@@ -1866,7 +1848,7 @@ class YoutubeDL(object):
                     def compatible_formats(formats):
                         video, audio = formats
                         # Check extension
-                        video_ext, audio_ext = video.get('ext'), audio.get('ext')
+                        video_ext, audio_ext = audio.get('ext'), video.get('ext')
                         if video_ext and audio_ext:
                             COMPATIBLE_EXTS = (
                                 ('mp3', 'mp4', 'm4a', 'm4p', 'm4b', 'm4r', 'm4v', 'ismv', 'isma'),
@@ -2061,24 +2043,15 @@ class YoutubeDL(object):
                         self.report_warning('Unable to remove downloaded original file')
 
     def _make_archive_id(self, info_dict):
-        video_id = info_dict.get('id')
-        if not video_id:
-            return
         # Future-proof against any change in case
         # and backwards compatibility with prior versions
-        extractor = info_dict.get('extractor_key') or info_dict.get('ie_key')  # key in a playlist
+        extractor = info_dict.get('extractor_key')
         if extractor is None:
-            url = str_or_none(info_dict.get('url'))
-            if not url:
-                return
-            # Try to find matching extractor for the URL and take its ie_key
-            for ie in self._ies:
-                if ie.suitable(url):
-                    extractor = ie.ie_key()
-                    break
-            else:
-                return
-        return extractor.lower() + ' ' + video_id
+            if 'id' in info_dict:
+                extractor = info_dict.get('ie_key')  # key in a playlist
+        if extractor is None:
+            return None  # Incomplete video information
+        return extractor.lower() + ' ' + info_dict['id']
 
     def in_download_archive(self, info_dict):
         fn = self.params.get('download_archive')
@@ -2086,7 +2059,7 @@ class YoutubeDL(object):
             return False
 
         vid_id = self._make_archive_id(info_dict)
-        if not vid_id:
+        if vid_id is None:
             return False  # Incomplete video information
 
         try:
@@ -2311,9 +2284,10 @@ class YoutubeDL(object):
             self.cookiejar = compat_cookiejar.CookieJar()
         else:
             opts_cookiefile = expand_path(opts_cookiefile)
-            self.cookiejar = YoutubeDLCookieJar(opts_cookiefile)
+            self.cookiejar = compat_cookiejar.MozillaCookieJar(
+                opts_cookiefile)
             if os.access(opts_cookiefile, os.R_OK):
-                self.cookiejar.load(ignore_discard=True, ignore_expires=True)
+                self.cookiejar.load()
 
         cookie_processor = YoutubeDLCookieProcessor(self.cookiejar)
         if opts_proxy is not None:

youtube_dl/__init__.py

@@ -191,11 +191,6 @@ def _real_main(argv=None):
         if numeric_buffersize is None:
             parser.error('invalid buffer size specified')
         opts.buffersize = numeric_buffersize
-    if opts.http_chunk_size is not None:
-        numeric_chunksize = FileDownloader.parse_bytes(opts.http_chunk_size)
-        if not numeric_chunksize:
-            parser.error('invalid http chunk size specified')
-        opts.http_chunk_size = numeric_chunksize
     if opts.playliststart <= 0:
         raise ValueError('Playlist start must be positive')
     if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
@@ -351,7 +346,6 @@ def _real_main(argv=None):
         'keep_fragments': opts.keep_fragments,
         'buffersize': opts.buffersize,
         'noresizebuffer': opts.noresizebuffer,
-        'http_chunk_size': opts.http_chunk_size,
         'continuedl': opts.continue_dl,
         'noprogress': opts.noprogress,
         'progress_with_newline': opts.progress_with_newline,
@@ -430,7 +424,6 @@ def _real_main(argv=None):
         'config_location': opts.config_location,
         'geo_bypass': opts.geo_bypass,
         'geo_bypass_country': opts.geo_bypass_country,
-        'geo_bypass_ip_block': opts.geo_bypass_ip_block,
         # just for deprecation check
         'autonumber': opts.autonumber if opts.autonumber is True else None,
         'usetitle': opts.usetitle if opts.usetitle is True else None,

youtube_dl/aes.py

@@ -1,8 +1,8 @@
 from __future__ import unicode_literals
 
+import base64
 from math import ceil
 
-from .compat import compat_b64decode
 from .utils import bytes_to_intlist, intlist_to_bytes
 
 BLOCK_SIZE_BYTES = 16
@@ -180,7 +180,7 @@ def aes_decrypt_text(data, password, key_size_bytes):
     """
     NONCE_LENGTH_BYTES = 8
 
-    data = bytes_to_intlist(compat_b64decode(data))
+    data = bytes_to_intlist(base64.b64decode(data.encode('utf-8')))
     password = bytes_to_intlist(password.encode('utf-8'))
 
     key = password[:key_size_bytes] + [0] * (key_size_bytes - len(password))

youtube_dl/compat.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import base64
 import binascii
 import collections
 import ctypes
@@ -2787,12 +2786,6 @@ except NameError:  # Python 3
     compat_numeric_types = (int, float, complex)
 
 
-try:
-    compat_integer_types = (int, long)
-except NameError:  # Python 3
-    compat_integer_types = (int, )
-
-
 if sys.version_info < (2, 7):
     def compat_socket_create_connection(address, timeout, source_address=None):
         host, port = address
@@ -2903,24 +2896,9 @@ except TypeError:
         if isinstance(spec, compat_str):
             spec = spec.encode('ascii')
         return struct.unpack(spec, *args)
-
-    class compat_Struct(struct.Struct):
-        def __init__(self, fmt):
-            if isinstance(fmt, compat_str):
-                fmt = fmt.encode('ascii')
-            super(compat_Struct, self).__init__(fmt)
 else:
     compat_struct_pack = struct.pack
     compat_struct_unpack = struct.unpack
-    if platform.python_implementation() == 'IronPython' and sys.version_info < (2, 7, 8):
-        class compat_Struct(struct.Struct):
-            def unpack(self, string):
-                if not isinstance(string, buffer):  # noqa: F821
-                    string = buffer(string)  # noqa: F821
-                return super(compat_Struct, self).unpack(string)
-    else:
-        compat_Struct = struct.Struct
-
 
 try:
     from future_builtins import zip as compat_zip
@@ -2930,16 +2908,6 @@ except ImportError:  # not 2.6+ or is 3.x
     except ImportError:
         compat_zip = zip
 
-
-if sys.version_info < (3, 3):
-    def compat_b64decode(s, *args, **kwargs):
-        if isinstance(s, compat_str):
-            s = s.encode('ascii')
-        return base64.b64decode(s, *args, **kwargs)
-else:
-    compat_b64decode = base64.b64decode
-
-
 if platform.python_implementation() == 'PyPy' and sys.pypy_version_info < (5, 4, 0):
     # PyPy2 prior to version 5.4.0 expects byte strings as Windows function
     # names, see the original PyPy issue [1] and the youtube-dl one [2].
@@ -2962,8 +2930,6 @@ __all__ = [
     'compat_HTMLParseError',
     'compat_HTMLParser',
     'compat_HTTPError',
-    'compat_Struct',
-    'compat_b64decode',
     'compat_basestring',
     'compat_chr',
     'compat_cookiejar',
@@ -2980,7 +2946,6 @@ __all__ = [
     'compat_http_client',
     'compat_http_server',
     'compat_input',
-    'compat_integer_types',
     'compat_itertools_count',
     'compat_kwargs',
     'compat_numeric_types',

youtube_dl/downloader/common.py

@@ -45,12 +45,10 @@ class FileDownloader(object):
     min_filesize:       Skip files smaller than this size
     max_filesize:       Skip files larger than this size
     xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
+                        (experimental)
     external_downloader_args:  A list of additional command-line arguments for the
                         external downloader.
     hls_use_mpegts:     Use the mpegts container for HLS videos.
-    http_chunk_size:    Size of a chunk for chunk-based HTTP downloading. May be
-                        useful for bypassing bandwidth throttling imposed by
-                        a webserver (experimental)
 
     Subclasses of this one must re-define the real_download method.
     """
@@ -248,13 +246,12 @@ class FileDownloader(object):
             if self.params.get('noprogress', False):
                 self.to_screen('[download] Download completed')
             else:
-                msg_template = '100%%'
-                if s.get('total_bytes') is not None:
-                    s['_total_bytes_str'] = format_bytes(s['total_bytes'])
-                    msg_template += ' of %(_total_bytes_str)s'
+                s['_total_bytes_str'] = format_bytes(s['total_bytes'])
                 if s.get('elapsed') is not None:
                     s['_elapsed_str'] = self.format_seconds(s['elapsed'])
-                    msg_template += ' in %(_elapsed_str)s'
+                    msg_template = '100%% of %(_total_bytes_str)s in %(_elapsed_str)s'
+                else:
+                    msg_template = '100%% of %(_total_bytes_str)s'
                 self._report_progress_status(
                     msg_template % s, is_last_line=True)
 

youtube_dl/downloader/dash.py

@@ -2,10 +2,7 @@ from __future__ import unicode_literals
 
 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
-from ..utils import (
-    DownloadError,
-    urljoin,
-)
+from ..utils import urljoin
 
 
 class DashSegmentsFD(FragmentFD):
@@ -60,14 +57,6 @@ class DashSegmentsFD(FragmentFD):
                     count += 1
                     if count <= fragment_retries:
                         self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                except DownloadError:
-                    # Don't retry fragment if error occurred during HTTP downloading
-                    # itself since it has own retry settings
-                    if not fatal:
-                        self.report_skip_fragment(frag_index)
-                        break
-                    raise
-
             if count > fragment_retries:
                 if not fatal:
                     self.report_skip_fragment(frag_index)

youtube_dl/downloader/external.py

@@ -1,10 +1,9 @@
 from __future__ import unicode_literals
 
 import os.path
-import re
 import subprocess
 import sys
-import time
+import re
 
 from .common import FileDownloader
 from ..compat import (
@@ -31,7 +30,6 @@ class ExternalFD(FileDownloader):
         tmpfilename = self.temp_name(filename)
 
         try:
-            started = time.time()
             retval = self._call_downloader(tmpfilename, info_dict)
         except KeyboardInterrupt:
             if not info_dict.get('is_live'):
@@ -43,20 +41,15 @@ class ExternalFD(FileDownloader):
             self.to_screen('[%s] Interrupted by user' % self.get_basename())
 
         if retval == 0:
-            status = {
+            fsize = os.path.getsize(encodeFilename(tmpfilename))
+            self.to_screen('\r[%s] Downloaded %s bytes' % (self.get_basename(), fsize))
+            self.try_rename(tmpfilename, filename)
+            self._hook_progress({
+                'downloaded_bytes': fsize,
+                'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-                'elapsed': time.time() - started,
-            }
-            if filename != '-':
-                fsize = os.path.getsize(encodeFilename(tmpfilename))
-                self.to_screen('\r[%s] Downloaded %s bytes' % (self.get_basename(), fsize))
-                self.try_rename(tmpfilename, filename)
-                status.update({
-                    'downloaded_bytes': fsize,
-                    'total_bytes': fsize,
-                })
-            self._hook_progress(status)
+            })
             return True
         else:
             self.to_stderr('\n')

youtube_dl/downloader/f4m.py

@@ -1,12 +1,12 @@
 from __future__ import division, unicode_literals
 
+import base64
 import io
 import itertools
 import time
 
 from .fragment import FragmentFD
 from ..compat import (
-    compat_b64decode,
     compat_etree_fromstring,
     compat_urlparse,
     compat_urllib_error,
@@ -312,7 +312,7 @@ class F4mFD(FragmentFD):
             boot_info = self._get_bootstrap_from_url(bootstrap_url)
         else:
             bootstrap_url = None
-            bootstrap = compat_b64decode(node.text)
+            bootstrap = base64.b64decode(node.text.encode('ascii'))
             boot_info = read_bootstrap_info(bootstrap)
         return boot_info, bootstrap_url
 
@@ -349,7 +349,7 @@ class F4mFD(FragmentFD):
         live = boot_info['live']
         metadata_node = media.find(_add_ns('metadata'))
         if metadata_node is not None:
-            metadata = compat_b64decode(metadata_node.text)
+            metadata = base64.b64decode(metadata_node.text.encode('ascii'))
         else:
             metadata = None
 

youtube_dl/downloader/fragment.py

@@ -74,14 +74,9 @@ class FragmentFD(FileDownloader):
         return not ctx['live'] and not ctx['tmpfilename'] == '-'
 
     def _read_ytdl_file(self, ctx):
-        assert 'ytdl_corrupt' not in ctx
         stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'r')
-        try:
-            ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
-        except Exception:
-            ctx['ytdl_corrupt'] = True
-        finally:
-            stream.close()
+        ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
+        stream.close()
 
     def _write_ytdl_file(self, ctx):
         frag_index_stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'w')
@@ -163,17 +158,11 @@ class FragmentFD(FileDownloader):
         if self.__do_ytdl_file(ctx):
             if os.path.isfile(encodeFilename(self.ytdl_filename(ctx['filename']))):
                 self._read_ytdl_file(ctx)
-                is_corrupt = ctx.get('ytdl_corrupt') is True
-                is_inconsistent = ctx['fragment_index'] > 0 and resume_len == 0
-                if is_corrupt or is_inconsistent:
-                    message = (
-                        '.ytdl file is corrupt' if is_corrupt else
-                        'Inconsistent state of incomplete fragment download')
+                if ctx['fragment_index'] > 0 and resume_len == 0:
                     self.report_warning(
-                        '%s. Restarting from the beginning...' % message)
+                        'Inconsistent state of incomplete fragment download. '
+                        'Restarting from the beginning...')
                     ctx['fragment_index'] = resume_len = 0
-                    if 'ytdl_corrupt' in ctx:
-                        del ctx['ytdl_corrupt']
                     self._write_ytdl_file(ctx)
             else:
                 self._write_ytdl_file(ctx)
@@ -252,16 +241,12 @@ class FragmentFD(FileDownloader):
             if os.path.isfile(ytdl_filename):
                 os.remove(ytdl_filename)
         elapsed = time.time() - ctx['started']
-
-        if ctx['tmpfilename'] == '-':
-            downloaded_bytes = ctx['complete_frags_downloaded_bytes']
-        else:
-            self.try_rename(ctx['tmpfilename'], ctx['filename'])
-            downloaded_bytes = os.path.getsize(encodeFilename(ctx['filename']))
+        self.try_rename(ctx['tmpfilename'], ctx['filename'])
+        fsize = os.path.getsize(encodeFilename(ctx['filename']))
 
         self._hook_progress({
-            'downloaded_bytes': downloaded_bytes,
-            'total_bytes': downloaded_bytes,
+            'downloaded_bytes': fsize,
+            'total_bytes': fsize,
             'filename': ctx['filename'],
             'status': 'finished',
             'elapsed': elapsed,

youtube_dl/downloader/hls.py

@@ -75,13 +75,8 @@ class HlsFD(FragmentFD):
                 fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
-        def is_ad_fragment_start(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s or
-                    s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
-
-        def is_ad_fragment_end(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s or
-                    s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
+        def anvato_ad(s):
+            return s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
 
         media_frags = 0
         ad_frags = 0
@@ -91,13 +86,12 @@ class HlsFD(FragmentFD):
             if not line:
                 continue
             if line.startswith('#'):
-                if is_ad_fragment_start(line):
+                if anvato_ad(line):
+                    ad_frags += 1
                     ad_frag_next = True
-                elif is_ad_fragment_end(line):
-                    ad_frag_next = False
                 continue
             if ad_frag_next:
-                ad_frags += 1
+                ad_frag_next = False
                 continue
             media_frags += 1
 
@@ -128,6 +122,7 @@ class HlsFD(FragmentFD):
             if line:
                 if not line.startswith('#'):
                     if ad_frag_next:
+                        ad_frag_next = False
                         continue
                     frag_index += 1
                     if frag_index <= ctx['fragment_index']:
@@ -200,10 +195,8 @@ class HlsFD(FragmentFD):
                         'start': sub_range_start,
                         'end': sub_range_start + int(splitted_byte_range[0]),
                     }
-                elif is_ad_fragment_start(line):
+                elif anvato_ad(line):
                     ad_frag_next = True
-                elif is_ad_fragment_end(line):
-                    ad_frag_next = False
 
         self._finish_frag_download(ctx)
 

youtube_dl/downloader/http.py

@@ -4,18 +4,13 @@ import errno
 import os
 import socket
 import time
-import random
 import re
 
 from .common import FileDownloader
-from ..compat import (
-    compat_str,
-    compat_urllib_error,
-)
+from ..compat import compat_urllib_error
 from ..utils import (
     ContentTooShortError,
     encodeFilename,
-    int_or_none,
     sanitize_open,
     sanitized_Request,
     write_xattr,
@@ -43,26 +38,21 @@ class HttpFD(FileDownloader):
         add_headers = info_dict.get('http_headers')
         if add_headers:
             headers.update(add_headers)
+        basic_request = sanitized_Request(url, None, headers)
+        request = sanitized_Request(url, None, headers)
 
         is_test = self.params.get('test', False)
-        chunk_size = self._TEST_FILE_SIZE if is_test else (
-            info_dict.get('downloader_options', {}).get('http_chunk_size') or
-            self.params.get('http_chunk_size') or 0)
+
+        if is_test:
+            request.add_header('Range', 'bytes=0-%s' % str(self._TEST_FILE_SIZE - 1))
 
         ctx.open_mode = 'wb'
         ctx.resume_len = 0
-        ctx.data_len = None
-        ctx.block_size = self.params.get('buffersize', 1024)
-        ctx.start_time = time.time()
-        ctx.chunk_size = None
 
         if self.params.get('continuedl', True):
             # Establish possible resume length
             if os.path.isfile(encodeFilename(ctx.tmpfilename)):
-                ctx.resume_len = os.path.getsize(
-                    encodeFilename(ctx.tmpfilename))
-
-        ctx.is_resume = ctx.resume_len > 0
+                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
 
         count = 0
         retries = self.params.get('retries', 0)
@@ -74,36 +64,11 @@ class HttpFD(FileDownloader):
             def __init__(self, source_error):
                 self.source_error = source_error
 
-        class NextFragment(Exception):
-            pass
-
-        def set_range(req, start, end):
-            range_header = 'bytes=%d-' % start
-            if end:
-                range_header += compat_str(end)
-            req.add_header('Range', range_header)
-
         def establish_connection():
-            ctx.chunk_size = (random.randint(int(chunk_size * 0.95), chunk_size)
-                              if not is_test and chunk_size else chunk_size)
-            if ctx.resume_len > 0:
-                range_start = ctx.resume_len
-                if ctx.is_resume:
-                    self.report_resuming_byte(ctx.resume_len)
+            if ctx.resume_len != 0:
+                self.report_resuming_byte(ctx.resume_len)
+                request.add_header('Range', 'bytes=%d-' % ctx.resume_len)
                 ctx.open_mode = 'ab'
-            elif ctx.chunk_size > 0:
-                range_start = 0
-            else:
-                range_start = None
-            ctx.is_resume = False
-            range_end = range_start + ctx.chunk_size - 1 if ctx.chunk_size else None
-            if range_end and ctx.data_len is not None and range_end >= ctx.data_len:
-                range_end = ctx.data_len - 1
-            has_range = range_start is not None
-            ctx.has_range = has_range
-            request = sanitized_Request(url, None, headers)
-            if has_range:
-                set_range(request, range_start, range_end)
             # Establish connection
             try:
                 ctx.data = self.ydl.urlopen(request)
@@ -112,40 +77,29 @@ class HttpFD(FileDownloader):
                 # that don't support resuming and serve a whole file with no Content-Range
                 # set in response despite of requested Range (see
                 # https://github.com/rg3/youtube-dl/issues/6057#issuecomment-126129799)
-                if has_range:
+                if ctx.resume_len > 0:
                     content_range = ctx.data.headers.get('Content-Range')
                     if content_range:
-                        content_range_m = re.search(r'bytes (\d+)-(\d+)?(?:/(\d+))?', content_range)
+                        content_range_m = re.search(r'bytes (\d+)-', content_range)
                         # Content-Range is present and matches requested Range, resume is possible
-                        if content_range_m:
-                            if range_start == int(content_range_m.group(1)):
-                                content_range_end = int_or_none(content_range_m.group(2))
-                                content_len = int_or_none(content_range_m.group(3))
-                                accept_content_len = (
-                                    # Non-chunked download
-                                    not ctx.chunk_size or
-                                    # Chunked download and requested piece or
-                                    # its part is promised to be served
-                                    content_range_end == range_end or
-                                    content_len < range_end)
-                                if accept_content_len:
-                                    ctx.data_len = content_len
-                                    return
+                        if content_range_m and ctx.resume_len == int(content_range_m.group(1)):
+                            return
                     # Content-Range is either not present or invalid. Assuming remote webserver is
                     # trying to send the whole file, resume is not possible, so wiping the local file
                     # and performing entire redownload
                     self.report_unable_to_resume()
                     ctx.resume_len = 0
                     ctx.open_mode = 'wb'
-                ctx.data_len = int_or_none(ctx.data.info().get('Content-length', None))
                 return
             except (compat_urllib_error.HTTPError, ) as err:
-                if err.code == 416:
+                if (err.code < 500 or err.code >= 600) and err.code != 416:
+                    # Unexpected HTTP error
+                    raise
+                elif err.code == 416:
                     # Unable to resume (requested range not satisfiable)
                     try:
                         # Open the connection again without the range header
-                        ctx.data = self.ydl.urlopen(
-                            sanitized_Request(url, None, headers))
+                        ctx.data = self.ydl.urlopen(basic_request)
                         content_length = ctx.data.info()['Content-Length']
                     except (compat_urllib_error.HTTPError, ) as err:
                         if err.code < 500 or err.code >= 600:
@@ -176,9 +130,6 @@ class HttpFD(FileDownloader):
                             ctx.resume_len = 0
                             ctx.open_mode = 'wb'
                             return
-                elif err.code < 500 or err.code >= 600:
-                    # Unexpected HTTP error
-                    raise
                 raise RetryDownload(err)
             except socket.error as err:
                 if err.errno != errno.ECONNRESET:
@@ -209,7 +160,7 @@ class HttpFD(FileDownloader):
                     return False
 
             byte_counter = 0 + ctx.resume_len
-            block_size = ctx.block_size
+            block_size = self.params.get('buffersize', 1024)
             start = time.time()
 
             # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
@@ -217,11 +168,10 @@ class HttpFD(FileDownloader):
             before = start  # start measuring
 
             def retry(e):
-                to_stdout = ctx.tmpfilename == '-'
-                if not to_stdout:
+                if ctx.tmpfilename != '-':
                     ctx.stream.close()
                 ctx.stream = None
-                ctx.resume_len = byte_counter if to_stdout else os.path.getsize(encodeFilename(ctx.tmpfilename))
+                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
                 raise RetryDownload(e)
 
             while True:
@@ -283,30 +233,25 @@ class HttpFD(FileDownloader):
 
                 # Progress message
                 speed = self.calc_speed(start, now, byte_counter - ctx.resume_len)
-                if ctx.data_len is None:
+                if data_len is None:
                     eta = None
                 else:
-                    eta = self.calc_eta(start, time.time(), ctx.data_len - ctx.resume_len, byte_counter - ctx.resume_len)
+                    eta = self.calc_eta(start, time.time(), data_len - ctx.resume_len, byte_counter - ctx.resume_len)
 
                 self._hook_progress({
                     'status': 'downloading',
                     'downloaded_bytes': byte_counter,
-                    'total_bytes': ctx.data_len,
+                    'total_bytes': data_len,
                     'tmpfilename': ctx.tmpfilename,
                     'filename': ctx.filename,
                     'eta': eta,
                     'speed': speed,
-                    'elapsed': now - ctx.start_time,
+                    'elapsed': now - start,
                 })
 
                 if is_test and byte_counter == data_len:
                     break
 
-            if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
-                ctx.resume_len = byte_counter
-                # ctx.block_size = block_size
-                raise NextFragment()
-
             if ctx.stream is None:
                 self.to_stderr('\n')
                 self.report_error('Did not get any data blocks')
@@ -331,7 +276,7 @@ class HttpFD(FileDownloader):
                 'total_bytes': byte_counter,
                 'filename': ctx.filename,
                 'status': 'finished',
-                'elapsed': time.time() - ctx.start_time,
+                'elapsed': time.time() - start,
             })
 
             return True
@@ -345,8 +290,6 @@ class HttpFD(FileDownloader):
                 if count <= retries:
                     self.report_retry(e.source_error, count, retries)
                 continue
-            except NextFragment:
-                continue
             except SucceedDownload:
                 return True
 

youtube_dl/downloader/ism.py

@@ -1,27 +1,25 @@
 from __future__ import unicode_literals
 
 import time
+import struct
 import binascii
 import io
 
 from .fragment import FragmentFD
-from ..compat import (
-    compat_Struct,
-    compat_urllib_error,
-)
+from ..compat import compat_urllib_error
 
 
-u8 = compat_Struct('>B')
-u88 = compat_Struct('>Bx')
-u16 = compat_Struct('>H')
-u1616 = compat_Struct('>Hxx')
-u32 = compat_Struct('>I')
-u64 = compat_Struct('>Q')
+u8 = struct.Struct(b'>B')
+u88 = struct.Struct(b'>Bx')
+u16 = struct.Struct(b'>H')
+u1616 = struct.Struct(b'>Hxx')
+u32 = struct.Struct(b'>I')
+u64 = struct.Struct(b'>Q')
 
-s88 = compat_Struct('>bx')
-s16 = compat_Struct('>h')
-s1616 = compat_Struct('>hxx')
-s32 = compat_Struct('>i')
+s88 = struct.Struct(b'>bx')
+s16 = struct.Struct(b'>h')
+s1616 = struct.Struct(b'>hxx')
+s32 = struct.Struct(b'>i')
 
 unity_matrix = (s32.pack(0x10000) + s32.pack(0) * 3) * 2 + s32.pack(0x40000000)
 
@@ -141,7 +139,7 @@ def write_piff_header(stream, params):
         sample_entry_payload += u16.pack(0x18)  # depth
         sample_entry_payload += s16.pack(-1)  # pre defined
 
-        codec_private_data = binascii.unhexlify(params['codec_private_data'].encode('utf-8'))
+        codec_private_data = binascii.unhexlify(params['codec_private_data'])
         if fourcc in ('H264', 'AVC1'):
             sps, pps = codec_private_data.split(u32.pack(1))[1:]
             avcc_payload = u8.pack(1)  # configuration version

youtube_dl/downloader/rtmp.py

@@ -29,68 +29,66 @@ class RtmpFD(FileDownloader):
             proc = subprocess.Popen(args, stderr=subprocess.PIPE)
             cursor_in_new_line = True
             proc_stderr_closed = False
-            try:
-                while not proc_stderr_closed:
-                    # read line from stderr
-                    line = ''
-                    while True:
-                        char = proc.stderr.read(1)
-                        if not char:
-                            proc_stderr_closed = True
-                            break
-                        if char in [b'\r', b'\n']:
-                            break
-                        line += char.decode('ascii', 'replace')
-                    if not line:
-                        # proc_stderr_closed is True
-                        continue
-                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
+            while not proc_stderr_closed:
+                # read line from stderr
+                line = ''
+                while True:
+                    char = proc.stderr.read(1)
+                    if not char:
+                        proc_stderr_closed = True
+                        break
+                    if char in [b'\r', b'\n']:
+                        break
+                    line += char.decode('ascii', 'replace')
+                if not line:
+                    # proc_stderr_closed is True
+                    continue
+                mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
+                if mobj:
+                    downloaded_data_len = int(float(mobj.group(1)) * 1024)
+                    percent = float(mobj.group(2))
+                    if not resume_percent:
+                        resume_percent = percent
+                        resume_downloaded_data_len = downloaded_data_len
+                    time_now = time.time()
+                    eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
+                    speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
+                    data_len = None
+                    if percent > 0:
+                        data_len = int(downloaded_data_len * 100 / percent)
+                    self._hook_progress({
+                        'status': 'downloading',
+                        'downloaded_bytes': downloaded_data_len,
+                        'total_bytes_estimate': data_len,
+                        'tmpfilename': tmpfilename,
+                        'filename': filename,
+                        'eta': eta,
+                        'elapsed': time_now - start,
+                        'speed': speed,
+                    })
+                    cursor_in_new_line = False
+                else:
+                    # no percent for live streams
+                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
                     if mobj:
                         downloaded_data_len = int(float(mobj.group(1)) * 1024)
-                        percent = float(mobj.group(2))
-                        if not resume_percent:
-                            resume_percent = percent
-                            resume_downloaded_data_len = downloaded_data_len
                         time_now = time.time()
-                        eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
-                        speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
-                        data_len = None
-                        if percent > 0:
-                            data_len = int(downloaded_data_len * 100 / percent)
+                        speed = self.calc_speed(start, time_now, downloaded_data_len)
                         self._hook_progress({
-                            'status': 'downloading',
                             'downloaded_bytes': downloaded_data_len,
-                            'total_bytes_estimate': data_len,
                             'tmpfilename': tmpfilename,
                             'filename': filename,
-                            'eta': eta,
+                            'status': 'downloading',
                             'elapsed': time_now - start,
                             'speed': speed,
                         })
                         cursor_in_new_line = False
-                    else:
-                        # no percent for live streams
-                        mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
-                        if mobj:
-                            downloaded_data_len = int(float(mobj.group(1)) * 1024)
-                            time_now = time.time()
-                            speed = self.calc_speed(start, time_now, downloaded_data_len)
-                            self._hook_progress({
-                                'downloaded_bytes': downloaded_data_len,
-                                'tmpfilename': tmpfilename,
-                                'filename': filename,
-                                'status': 'downloading',
-                                'elapsed': time_now - start,
-                                'speed': speed,
-                            })
-                            cursor_in_new_line = False
-                        elif self.params.get('verbose', False):
-                            if not cursor_in_new_line:
-                                self.to_screen('')
-                            cursor_in_new_line = True
-                            self.to_screen('[rtmpdump] ' + line)
-            finally:
-                proc.wait()
+                    elif self.params.get('verbose', False):
+                        if not cursor_in_new_line:
+                            self.to_screen('')
+                        cursor_in_new_line = True
+                        self.to_screen('[rtmpdump] ' + line)
+            proc.wait()
             if not cursor_in_new_line:
                 self.to_screen('')
             return proc.returncode
@@ -165,15 +163,7 @@ class RtmpFD(FileDownloader):
         RD_INCOMPLETE = 2
         RD_NO_CONNECT = 3
 
-        started = time.time()
-
-        try:
-            retval = run_rtmpdump(args)
-        except KeyboardInterrupt:
-            if not info_dict.get('is_live'):
-                raise
-            retval = RD_SUCCESS
-            self.to_screen('\n[rtmpdump] Interrupted by user')
+        retval = run_rtmpdump(args)
 
         if retval == RD_NO_CONNECT:
             self.report_error('[rtmpdump] Could not connect to RTMP server.')
@@ -181,7 +171,7 @@ class RtmpFD(FileDownloader):
 
         while retval in (RD_INCOMPLETE, RD_FAILED) and not test and not live:
             prevsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] Downloaded %s bytes' % prevsize)
+            self.to_screen('[rtmpdump] %s bytes' % prevsize)
             time.sleep(5.0)  # This seems to be needed
             args = basic_args + ['--resume']
             if retval == RD_FAILED:
@@ -198,14 +188,13 @@ class RtmpFD(FileDownloader):
                 break
         if retval == RD_SUCCESS or (test and retval == RD_INCOMPLETE):
             fsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] Downloaded %s bytes' % fsize)
+            self.to_screen('[rtmpdump] %s bytes' % fsize)
             self.try_rename(tmpfilename, filename)
             self._hook_progress({
                 'downloaded_bytes': fsize,
                 'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-                'elapsed': time.time() - started,
             })
             return True
         else:

youtube_dl/extractor/abc.py

@@ -13,7 +13,6 @@ from ..utils import (
     int_or_none,
     parse_iso8601,
     try_get,
-    unescapeHTML,
     update_url_query,
 )
 
@@ -105,22 +104,21 @@ class ABCIE(InfoExtractor):
 
 class ABCIViewIE(InfoExtractor):
     IE_NAME = 'abc.net.au:iview'
-    _VALID_URL = r'https?://iview\.abc\.net\.au/(?:[^/]+/)*video/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://iview\.abc\.net\.au/programs/[^/]+/(?P<id>[^/?#]+)'
     _GEO_COUNTRIES = ['AU']
 
     # ABC iview programs are normally available for 14 days only.
     _TESTS = [{
-        'url': 'https://iview.abc.net.au/show/ben-and-hollys-little-kingdom/series/0/video/ZX9371A050S00',
+        'url': 'http://iview.abc.net.au/programs/call-the-midwife/ZW0898A003S00',
         'md5': 'cde42d728b3b7c2b32b1b94b4a548afc',
         'info_dict': {
-            'id': 'ZX9371A050S00',
+            'id': 'ZW0898A003S00',
             'ext': 'mp4',
-            'title': "Gaston's Birthday",
-            'series': "Ben And Holly's Little Kingdom",
-            'description': 'md5:f9de914d02f226968f598ac76f105bcf',
-            'upload_date': '20180604',
-            'uploader_id': 'abc4kids',
-            'timestamp': 1528140219,
+            'title': 'Series 5 Ep 3',
+            'description': 'md5:e0ef7d4f92055b86c4f33611f180ed79',
+            'upload_date': '20171228',
+            'uploader_id': 'abc1',
+            'timestamp': 1514499187,
         },
         'params': {
             'skip_download': True,
@@ -129,16 +127,17 @@ class ABCIViewIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_params = self._download_json(
-            'https://iview.abc.net.au/api/programs/' + video_id, video_id)
-        title = unescapeHTML(video_params.get('title') or video_params['seriesTitle'])
-        stream = next(s for s in video_params['playlist'] if s.get('type') in ('program', 'livestream'))
+        webpage = self._download_webpage(url, video_id)
+        video_params = self._parse_json(self._search_regex(
+            r'videoParams\s*=\s*({.+?});', webpage, 'video params'), video_id)
+        title = video_params.get('title') or video_params['seriesTitle']
+        stream = next(s for s in video_params['playlist'] if s.get('type') == 'program')
 
-        house_number = video_params.get('episodeHouseNumber') or video_id
-        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-tablet'.format(
+        house_number = video_params.get('episodeHouseNumber')
+        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-mobile'.format(
             int(time.time()), house_number)
         sig = hmac.new(
-            b'android.content.res.Resources',
+            'android.content.res.Resources'.encode('utf-8'),
             path.encode('utf-8'), hashlib.sha256).hexdigest()
         token = self._download_webpage(
             'http://iview.abc.net.au{0}&sig={1}'.format(path, sig), video_id)
@@ -168,26 +167,18 @@ class ABCIViewIE(InfoExtractor):
                 'ext': 'vtt',
             }]
 
-        is_live = video_params.get('livestream') == '1'
-        if is_live:
-            title = self._live_title(title)
-
         return {
             'id': video_id,
             'title': title,
-            'description': video_params.get('description'),
-            'thumbnail': video_params.get('thumbnail'),
+            'description': self._html_search_meta(['og:description', 'twitter:description'], webpage),
+            'thumbnail': self._html_search_meta(['og:image', 'twitter:image:src'], webpage),
             'duration': int_or_none(video_params.get('eventDuration')),
             'timestamp': parse_iso8601(video_params.get('pubDate'), ' '),
-            'series': unescapeHTML(video_params.get('seriesTitle')),
+            'series': video_params.get('seriesTitle'),
             'series_id': video_params.get('seriesHouseNumber') or video_id[:7],
-            'season_number': int_or_none(self._search_regex(
-                r'\bSeries\s+(\d+)\b', title, 'season number', default=None)),
-            'episode_number': int_or_none(self._search_regex(
-                r'\bEp\s+(\d+)\b', title, 'episode number', default=None)),
-            'episode_id': house_number,
+            'episode_number': int_or_none(self._html_search_meta('episodeNumber', webpage, default=None)),
+            'episode': self._html_search_meta('episode_title', webpage, default=None),
             'uploader_id': video_params.get('channel'),
             'formats': formats,
             'subtitles': subtitles,
-            'is_live': is_live,
         }

youtube_dl/extractor/abcnews.py

@@ -66,7 +66,7 @@ class AbcNewsIE(InfoExtractor):
     _TESTS = [{
         'url': 'http://abcnews.go.com/Blotter/News/dramatic-video-rare-death-job-america/story?id=10498713#.UIhwosWHLjY',
         'info_dict': {
-            'id': '10505354',
+            'id': '10498713',
             'ext': 'flv',
             'display_id': 'dramatic-video-rare-death-job-america',
             'title': 'Occupational Hazards',
@@ -79,7 +79,7 @@ class AbcNewsIE(InfoExtractor):
     }, {
         'url': 'http://abcnews.go.com/Entertainment/justin-timberlake-performs-stop-feeling-eurovision-2016/story?id=39125818',
         'info_dict': {
-            'id': '38897857',
+            'id': '39125818',
             'ext': 'mp4',
             'display_id': 'justin-timberlake-performs-stop-feeling-eurovision-2016',
             'title': 'Justin Timberlake Drops Hints For Secret Single',

youtube_dl/extractor/acast.py

@@ -7,9 +7,7 @@ import functools
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
-    float_or_none,
     int_or_none,
-    try_get,
     unified_timestamp,
     OnDemandPagedList,
 )
@@ -17,89 +15,65 @@ from ..utils import (
 
 class ACastIE(InfoExtractor):
     IE_NAME = 'acast'
-    _VALID_URL = r'''(?x)
-                    https?://
-                        (?:
-                            (?:(?:embed|www)\.)?acast\.com/|
-                            play\.acast\.com/s/
-                        )
-                        (?P<channel>[^/]+)/(?P<id>[^/#?]+)
-                    '''
+    _VALID_URL = r'https?://(?:www\.)?acast\.com/(?P<channel>[^/]+)/(?P<id>[^/#?]+)'
     _TESTS = [{
+        # test with one bling
+        'url': 'https://www.acast.com/condenasttraveler/-where-are-you-taipei-101-taiwan',
+        'md5': 'ada3de5a1e3a2a381327d749854788bb',
+        'info_dict': {
+            'id': '57de3baa-4bb0-487e-9418-2692c1277a34',
+            'ext': 'mp3',
+            'title': '"Where Are You?": Taipei 101, Taiwan',
+            'timestamp': 1196172000,
+            'upload_date': '20071127',
+            'description': 'md5:a0b4ef3634e63866b542e5b1199a1a0e',
+            'duration': 211,
+        }
+    }, {
+        # test with multiple blings
         'url': 'https://www.acast.com/sparpodcast/2.raggarmordet-rosterurdetforflutna',
-        'md5': 'a02393c74f3bdb1801c3ec2695577ce0',
+        'md5': 'e87d5b8516cd04c0d81b6ee1caca28d0',
         'info_dict': {
             'id': '2a92b283-1a75-4ad8-8396-499c641de0d9',
             'ext': 'mp3',
             'title': '2. Raggarmordet - Röster ur det förflutna',
-            'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
             'timestamp': 1477346700,
             'upload_date': '20161024',
-            'duration': 2766.602563,
-            'creator': 'Anton Berg & Martin Johnson',
-            'series': 'Spår',
-            'episode': '2. Raggarmordet - Röster ur det förflutna',
+            'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
+            'duration': 2766,
         }
-    }, {
-        'url': 'http://embed.acast.com/adambuxton/ep.12-adam-joeschristmaspodcast2015',
-        'only_matching': True,
-    }, {
-        'url': 'https://play.acast.com/s/rattegangspodden/s04e09-styckmordet-i-helenelund-del-22',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         channel, display_id = re.match(self._VALID_URL, url).groups()
-        s = self._download_json(
-            'https://play-api.acast.com/stitch/%s/%s' % (channel, display_id),
-            display_id)['result']
-        media_url = s['url']
         cast_data = self._download_json(
-            'https://play-api.acast.com/splash/%s/%s' % (channel, display_id),
-            display_id)['result']
-        e = cast_data['episode']
-        title = e['name']
+            'https://play-api.acast.com/splash/%s/%s' % (channel, display_id), display_id)
+        e = cast_data['result']['episode']
         return {
             'id': compat_str(e['id']),
             'display_id': display_id,
-            'url': media_url,
-            'title': title,
-            'description': e.get('description') or e.get('summary'),
+            'url': e['mediaUrl'],
+            'title': e['name'],
+            'description': e.get('description'),
             'thumbnail': e.get('image'),
             'timestamp': unified_timestamp(e.get('publishingDate')),
-            'duration': float_or_none(s.get('duration') or e.get('duration')),
-            'filesize': int_or_none(e.get('contentLength')),
-            'creator': try_get(cast_data, lambda x: x['show']['author'], compat_str),
-            'series': try_get(cast_data, lambda x: x['show']['name'], compat_str),
-            'season_number': int_or_none(e.get('seasonNumber')),
-            'episode': title,
-            'episode_number': int_or_none(e.get('episodeNumber')),
+            'duration': int_or_none(e.get('duration')),
         }
 
 
 class ACastChannelIE(InfoExtractor):
     IE_NAME = 'acast:channel'
-    _VALID_URL = r'''(?x)
-                    https?://
-                        (?:
-                            (?:www\.)?acast\.com/|
-                            play\.acast\.com/s/
-                        )
-                        (?P<id>[^/#?]+)
-                    '''
-    _TESTS = [{
-        'url': 'https://www.acast.com/todayinfocus',
+    _VALID_URL = r'https?://(?:www\.)?acast\.com/(?P<id>[^/#?]+)'
+    _TEST = {
+        'url': 'https://www.acast.com/condenasttraveler',
         'info_dict': {
-            'id': '4efc5294-5385-4847-98bd-519799ce5786',
-            'title': 'Today in Focus',
-            'description': 'md5:9ba5564de5ce897faeb12963f4537a64',
+            'id': '50544219-29bb-499e-a083-6087f4cb7797',
+            'title': 'Condé Nast Traveler Podcast',
+            'description': 'md5:98646dee22a5b386626ae31866638fbd',
         },
-        'playlist_mincount': 35,
-    }, {
-        'url': 'http://play.acast.com/s/ft-banking-weekly',
-        'only_matching': True,
-    }]
-    _API_BASE_URL = 'https://play.acast.com/api/'
+        'playlist_mincount': 20,
+    }
+    _API_BASE_URL = 'https://www.acast.com/api/'
     _PAGE_SIZE = 10
 
     @classmethod
@@ -112,7 +86,7 @@ class ACastChannelIE(InfoExtractor):
             channel_slug, note='Download page %d of channel data' % page)
         for cast in casts:
             yield self.url_result(
-                'https://play.acast.com/s/%s/%s' % (channel_slug, cast['url']),
+                'https://www.acast.com/%s/%s' % (channel_slug, cast['url']),
                 'ACast', cast['id'])
 
     def _real_extract(self, url):

youtube_dl/extractor/adn.py

@@ -2,25 +2,17 @@
 from __future__ import unicode_literals
 
 import base64
-import binascii
 import json
 import os
-import random
 
 from .common import InfoExtractor
 from ..aes import aes_cbc_decrypt
-from ..compat import (
-    compat_b64decode,
-    compat_ord,
-)
+from ..compat import compat_ord
 from ..utils import (
     bytes_to_intlist,
-    bytes_to_long,
     ExtractorError,
     float_or_none,
     intlist_to_bytes,
-    long_to_bytes,
-    pkcs1pad,
     srt_subtitles_timecode,
     strip_or_none,
     urljoin,
@@ -41,7 +33,6 @@ class ADNIE(InfoExtractor):
         }
     }
     _BASE_URL = 'http://animedigitalnetwork.fr'
-    _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
 
     def _get_subtitles(self, sub_path, video_id):
         if not sub_path:
@@ -49,15 +40,17 @@ class ADNIE(InfoExtractor):
 
         enc_subtitles = self._download_webpage(
             urljoin(self._BASE_URL, sub_path),
-            video_id, fatal=False)
+            video_id, fatal=False, headers={
+                'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0',
+            })
         if not enc_subtitles:
             return None
 
         # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
         dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
-            bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(binascii.unhexlify(self._K + '9032ad7083106400')),
-            bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
+            bytes_to_intlist(base64.b64decode(enc_subtitles[24:])),
+            bytes_to_intlist(b'\x1b\xe0\x29\x61\x38\x94\x24\x00\x12\xbd\xc5\x80\xac\xce\xbe\xb0'),
+            bytes_to_intlist(base64.b64decode(enc_subtitles[:24]))
         ))
         subtitles_json = self._parse_json(
             dec_subtitles[:-compat_ord(dec_subtitles[-1])].decode(),
@@ -112,31 +105,15 @@ class ADNIE(InfoExtractor):
 
         options = player_config.get('options') or {}
         metas = options.get('metas') or {}
+        title = metas.get('title') or video_info['title']
         links = player_config.get('links') or {}
-        sub_path = player_config.get('subtitles')
         error = None
         if not links:
-            links_url = player_config.get('linksurl') or options['videoUrl']
-            token = options['token']
-            self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
-            message = bytes_to_intlist(json.dumps({
-                'k': self._K,
-                'e': 60,
-                't': token,
-            }))
-            padded_message = intlist_to_bytes(pkcs1pad(message, 128))
-            n, e = self._RSA_KEY
-            encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
-            authorization = base64.b64encode(encrypted_message).decode()
-            links_data = self._download_json(
-                urljoin(self._BASE_URL, links_url), video_id, headers={
-                    'Authorization': 'Bearer ' + authorization,
-                })
+            links_url = player_config['linksurl']
+            links_data = self._download_json(urljoin(
+                self._BASE_URL, links_url), video_id)
             links = links_data.get('links') or {}
-            metas = metas or links_data.get('meta') or {}
-            sub_path = (sub_path or links_data.get('subtitles')) + '&token=' + token
             error = links_data.get('error')
-        title = metas.get('title') or video_info['title']
 
         formats = []
         for format_id, qualities in links.items():
@@ -167,7 +144,7 @@ class ADNIE(InfoExtractor):
             'description': strip_or_none(metas.get('summary') or video_info.get('resume')),
             'thumbnail': video_info.get('image'),
             'formats': formats,
-            'subtitles': self.extract_subtitles(sub_path, video_id),
+            'subtitles': self.extract_subtitles(player_config.get('subtitles'), video_id),
             'episode': metas.get('subtitle') or video_info.get('videoTitle'),
             'series': video_info.get('playlistTitle'),
         }

youtube_dl/extractor/adobepass.py

@@ -1325,8 +1325,8 @@ class AdobePassIE(InfoExtractor):
     _DOWNLOADING_LOGIN_PAGE = 'Downloading Provider Login Page'
 
     def _download_webpage_handle(self, *args, **kwargs):
-        headers = self.geo_verification_headers()
-        headers.update(kwargs.get('headers', {}))
+        headers = kwargs.get('headers', {})
+        headers.update(self.geo_verification_headers())
         kwargs['headers'] = headers
         return super(AdobePassIE, self)._download_webpage_handle(
             *args, **compat_kwargs(kwargs))

youtube_dl/extractor/adultswim.py

@@ -7,7 +7,6 @@ from .turner import TurnerBaseIE
 from ..utils import (
     int_or_none,
     strip_or_none,
-    url_or_none,
 )
 
 
@@ -99,7 +98,7 @@ class AdultSwimIE(TurnerBaseIE):
             if not video_id:
                 entries = []
                 for episode in video_data.get('archiveEpisodes', []):
-                    episode_url = url_or_none(episode.get('url'))
+                    episode_url = episode.get('url')
                     if not episode_url:
                         continue
                     entries.append(self.url_result(

youtube_dl/extractor/aenetworks.py

@@ -22,19 +22,18 @@ class AENetworksBaseIE(ThePlatformIE):
 
 class AENetworksIE(AENetworksBaseIE):
     IE_NAME = 'aenetworks'
-    IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault'
+    IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network'
     _VALID_URL = r'''(?x)
                     https?://
                         (?:www\.)?
                         (?P<domain>
-                            (?:history(?:vault)?|aetv|mylifetime|lifetimemovieclub)\.com|
+                            (?:history|aetv|mylifetime|lifetimemovieclub)\.com|
                             fyi\.tv
                         )/
                         (?:
                             shows/(?P<show_path>[^/]+(?:/[^/]+){0,2})|
                             movies/(?P<movie_display_id>[^/]+)(?:/full-movie)?|
-                            specials/(?P<special_display_id>[^/]+)/full-special|
-                            collections/[^/]+/(?P<collection_display_id>[^/]+)
+                            specials/(?P<special_display_id>[^/]+)/full-special
                         )
                     '''
     _TESTS = [{
@@ -81,9 +80,6 @@ class AENetworksIE(AENetworksBaseIE):
     }, {
         'url': 'http://www.history.com/specials/sniper-into-the-kill-zone/full-special',
         'only_matching': True
-    }, {
-        'url': 'https://www.historyvault.com/collections/america-the-story-of-us/westward',
-        'only_matching': True
     }]
     _DOMAIN_TO_REQUESTOR_ID = {
         'history.com': 'HISTORY',
@@ -94,9 +90,9 @@ class AENetworksIE(AENetworksBaseIE):
     }
 
     def _real_extract(self, url):
-        domain, show_path, movie_display_id, special_display_id, collection_display_id = re.match(self._VALID_URL, url).groups()
-        display_id = show_path or movie_display_id or special_display_id or collection_display_id
-        webpage = self._download_webpage(url, display_id, headers=self.geo_verification_headers())
+        domain, show_path, movie_display_id, special_display_id = re.match(self._VALID_URL, url).groups()
+        display_id = show_path or movie_display_id or special_display_id
+        webpage = self._download_webpage(url, display_id)
         if show_path:
             url_parts = show_path.split('/')
             url_parts_len = len(url_parts)
@@ -126,8 +122,7 @@ class AENetworksIE(AENetworksBaseIE):
 
         query = {
             'mbr': 'true',
-            'assetTypes': 'high_video_ak',
-            'switch': 'hls_high_ak',
+            'assetTypes': 'high_video_s3'
         }
         video_id = self._html_search_meta('aetn:VideoID', webpage)
         media_url = self._search_regex(

youtube_dl/extractor/afreecatv.py

@@ -9,8 +9,6 @@ from ..utils import (
     determine_ext,
     ExtractorError,
     int_or_none,
-    url_or_none,
-    urlencode_postdata,
     xpath_text,
 )
 
@@ -30,7 +28,6 @@ class AfreecaTVIE(InfoExtractor):
                         )
                         (?P<id>\d+)
                     '''
-    _NETRC_MACHINE = 'afreecatv'
     _TESTS = [{
         'url': 'http://live.afreecatv.com:8079/app/index.cgi?szType=read_ucc_bbs&szBjId=dailyapril&nStationNo=16711924&nBbsNo=18605867&nTitleNo=36164052&szSkin=',
         'md5': 'f72c89fe7ecc14c1b5ce506c4996046e',
@@ -142,22 +139,22 @@ class AfreecaTVIE(InfoExtractor):
             'skip_download': True,
         },
     }, {
-        # PARTIAL_ADULT
-        'url': 'http://vod.afreecatv.com/PLAYER/STATION/32028439',
+        # adult video
+        'url': 'http://vod.afreecatv.com/PLAYER/STATION/26542731',
         'info_dict': {
-            'id': '20180327_27901457_202289533_1',
+            'id': '20171001_F1AE1711_196617479_1',
             'ext': 'mp4',
-            'title': '[생]빨개요♥ (part 1)',
+            'title': '[생]서아 초심 찾기 방송 (part 1)',
             'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
-            'uploader': '[SA]서아',
+            'uploader': 'BJ서아',
             'uploader_id': 'bjdyrksu',
-            'upload_date': '20180327',
-            'duration': 3601,
+            'upload_date': '20171001',
+            'duration': 3600,
+            'age_limit': 18,
         },
         'params': {
             'skip_download': True,
         },
-        'expected_warnings': ['adult content'],
     }, {
         'url': 'http://www.afreecatv.com/player/Player.swf?szType=szBjId=djleegoon&nStationNo=11273158&nBbsNo=13161095&nTitleNo=36327652',
         'only_matching': True,
@@ -175,107 +172,25 @@ class AfreecaTVIE(InfoExtractor):
             video_key['part'] = int(m.group('part'))
         return video_key
 
-    def _real_initialize(self):
-        self._login()
-
-    def _login(self):
-        username, password = self._get_login_info()
-        if username is None:
-            return
-
-        login_form = {
-            'szWork': 'login',
-            'szType': 'json',
-            'szUid': username,
-            'szPassword': password,
-            'isSaveId': 'false',
-            'szScriptVar': 'oLoginRet',
-            'szAction': '',
-        }
-
-        response = self._download_json(
-            'https://login.afreecatv.com/app/LoginAction.php', None,
-            'Logging in', data=urlencode_postdata(login_form))
-
-        _ERRORS = {
-            -4: 'Your account has been suspended due to a violation of our terms and policies.',
-            -5: 'https://member.afreecatv.com/app/user_delete_progress.php',
-            -6: 'https://login.afreecatv.com/membership/changeMember.php',
-            -8: "Hello! AfreecaTV here.\nThe username you have entered belongs to \n an account that requires a legal guardian's consent. \nIf you wish to use our services without restriction, \nplease make sure to go through the necessary verification process.",
-            -9: 'https://member.afreecatv.com/app/pop_login_block.php',
-            -11: 'https://login.afreecatv.com/afreeca/second_login.php',
-            -12: 'https://member.afreecatv.com/app/user_security.php',
-            0: 'The username does not exist or you have entered the wrong password.',
-            -1: 'The username does not exist or you have entered the wrong password.',
-            -3: 'You have entered your username/password incorrectly.',
-            -7: 'You cannot use your Global AfreecaTV account to access Korean AfreecaTV.',
-            -10: 'Sorry for the inconvenience. \nYour account has been blocked due to an unauthorized access. \nPlease contact our Help Center for assistance.',
-            -32008: 'You have failed to log in. Please contact our Help Center.',
-        }
-
-        result = int_or_none(response.get('RESULT'))
-        if result != 1:
-            error = _ERRORS.get(result, 'You have failed to log in.')
-            raise ExtractorError(
-                'Unable to login: %s said: %s' % (self.IE_NAME, error),
-                expected=True)
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, video_id)
-
-        if re.search(r'alert\(["\']This video has been deleted', webpage):
-            raise ExtractorError(
-                'Video %s has been deleted' % video_id, expected=True)
-
-        station_id = self._search_regex(
-            r'nStationNo\s*=\s*(\d+)', webpage, 'station')
-        bbs_id = self._search_regex(
-            r'nBbsNo\s*=\s*(\d+)', webpage, 'bbs')
-        video_id = self._search_regex(
-            r'nTitleNo\s*=\s*(\d+)', webpage, 'title', default=video_id)
-
-        partial_view = False
-        for _ in range(2):
-            query = {
+        video_xml = self._download_xml(
+            'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
+            video_id, query={
                 'nTitleNo': video_id,
-                'nStationNo': station_id,
-                'nBbsNo': bbs_id,
-            }
-            if partial_view:
-                query['partialView'] = 'SKIP_ADULT'
-            video_xml = self._download_xml(
-                'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
-                video_id, 'Downloading video info XML%s'
-                % (' (skipping adult)' if partial_view else ''),
-                video_id, headers={
-                    'Referer': url,
-                }, query=query)
+                'partialView': 'SKIP_ADULT',
+            })
 
-            flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
-            if flag and flag == 'SUCCEED':
-                break
-            if flag == 'PARTIAL_ADULT':
-                self._downloader.report_warning(
-                    'In accordance with local laws and regulations, underage users are restricted from watching adult content. '
-                    'Only content suitable for all ages will be downloaded. '
-                    'Provide account credentials if you wish to download restricted content.')
-                partial_view = True
-                continue
-            elif flag == 'ADULT':
-                error = 'Only users older than 19 are able to watch this video. Provide account credentials to download this content.'
-            else:
-                error = flag
+        flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
+        if flag and flag != 'SUCCEED':
             raise ExtractorError(
-                '%s said: %s' % (self.IE_NAME, error), expected=True)
-        else:
-            raise ExtractorError('Unable to download video info')
+                '%s said: %s' % (self.IE_NAME, flag), expected=True)
 
-        video_element = video_xml.findall(compat_xpath('./track/video'))[-1]
+        video_element = video_xml.findall(compat_xpath('./track/video'))[1]
         if video_element is None or video_element.text is None:
-            raise ExtractorError(
-                'Video %s video does not exist' % video_id, expected=True)
+            raise ExtractorError('Specified AfreecaTV video does not exist',
+                                 expected=True)
 
         video_url = video_element.text.strip()
 
@@ -305,7 +220,7 @@ class AfreecaTVIE(InfoExtractor):
             file_elements = video_element.findall(compat_xpath('./file'))
             one = len(file_elements) == 1
             for file_num, file_element in enumerate(file_elements, start=1):
-                file_url = url_or_none(file_element.text)
+                file_url = file_element.text
                 if not file_url:
                     continue
                 key = file_element.get('key', '')

youtube_dl/extractor/amcnetworks.py

@@ -11,7 +11,7 @@ from ..utils import (
 
 
 class AMCNetworksIE(ThePlatformIE):
-    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?:movies|shows(?:/[^/]+)+)/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|wetv)\.com/(?:movies|shows(?:/[^/]+)+)/(?P<id>[^/?#]+)'
     _TESTS = [{
         'url': 'http://www.ifc.com/shows/maron/season-04/episode-01/step-1',
         'md5': '',
@@ -51,9 +51,6 @@ class AMCNetworksIE(ThePlatformIE):
     }, {
         'url': 'http://www.wetv.com/shows/la-hair/videos/season-05/episode-09-episode-9-2/episode-9-sneak-peek-3',
         'only_matching': True,
-    }, {
-        'url': 'https://www.sundancetv.com/shows/riviera/full-episodes/season-1/episode-01-episode-1',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):

youtube_dl/extractor/americastestkitchen.py

@@ -43,6 +43,10 @@ class AmericasTestKitchenIE(InfoExtractor):
 
         webpage = self._download_webpage(url, video_id)
 
+        partner_id = self._search_regex(
+            r'src=["\'](?:https?:)?//(?:[^/]+\.)kaltura\.com/(?:[^/]+/)*(?:p|partner_id)/(\d+)',
+            webpage, 'kaltura partner id')
+
         video_data = self._parse_json(
             self._search_regex(
                 r'window\.__INITIAL_STATE__\s*=\s*({.+?})\s*;\s*</script>',
@@ -54,18 +58,7 @@ class AmericasTestKitchenIE(InfoExtractor):
             (lambda x: x['episodeDetail']['content']['data'],
              lambda x: x['videoDetail']['content']['data']), dict)
         ep_meta = ep_data.get('full_video', {})
-
-        zype_id = ep_meta.get('zype_id')
-        if zype_id:
-            embed_url = 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % zype_id
-            ie_key = 'Zype'
-        else:
-            partner_id = self._search_regex(
-                r'src=["\'](?:https?:)?//(?:[^/]+\.)kaltura\.com/(?:[^/]+/)*(?:p|partner_id)/(\d+)',
-                webpage, 'kaltura partner id')
-            external_id = ep_data.get('external_id') or ep_meta['external_id']
-            embed_url = 'kaltura:%s:%s' % (partner_id, external_id)
-            ie_key = 'Kaltura'
+        external_id = ep_data.get('external_id') or ep_meta['external_id']
 
         title = ep_data.get('title') or ep_meta.get('title')
         description = clean_html(ep_meta.get('episode_description') or ep_data.get(
@@ -79,8 +72,8 @@ class AmericasTestKitchenIE(InfoExtractor):
 
         return {
             '_type': 'url_transparent',
-            'url': embed_url,
-            'ie_key': ie_key,
+            'url': 'kaltura:%s:%s' % (partner_id, external_id),
+            'ie_key': 'Kaltura',
             'title': title,
             'description': description,
             'thumbnail': thumbnail,

youtube_dl/extractor/amp.py

@@ -3,12 +3,11 @@ from __future__ import unicode_literals
 
 from .common import InfoExtractor
 from ..utils import (
+    int_or_none,
+    parse_iso8601,
+    mimetype2ext,
     determine_ext,
     ExtractorError,
-    int_or_none,
-    mimetype2ext,
-    parse_iso8601,
-    url_or_none,
 )
 
 
@@ -36,7 +35,7 @@ class AMPIE(InfoExtractor):
                 media_thumbnail = [media_thumbnail]
             for thumbnail_data in media_thumbnail:
                 thumbnail = thumbnail_data.get('@attributes', {})
-                thumbnail_url = url_or_none(thumbnail.get('url'))
+                thumbnail_url = thumbnail.get('url')
                 if not thumbnail_url:
                     continue
                 thumbnails.append({
@@ -52,7 +51,7 @@ class AMPIE(InfoExtractor):
                 media_subtitle = [media_subtitle]
             for subtitle_data in media_subtitle:
                 subtitle = subtitle_data.get('@attributes', {})
-                subtitle_href = url_or_none(subtitle.get('href'))
+                subtitle_href = subtitle.get('href')
                 if not subtitle_href:
                     continue
                 subtitles.setdefault(subtitle.get('lang') or 'en', []).append({
@@ -66,7 +65,7 @@ class AMPIE(InfoExtractor):
             media_content = [media_content]
         for media_data in media_content:
             media = media_data.get('@attributes', {})
-            media_url = url_or_none(media.get('url'))
+            media_url = media.get('url')
             if not media_url:
                 continue
             ext = mimetype2ext(media.get('type')) or determine_ext(media_url)
@@ -80,7 +79,7 @@ class AMPIE(InfoExtractor):
             else:
                 formats.append({
                     'format_id': media_data.get('media-category', {}).get('@attributes', {}).get('label'),
-                    'url': media_url,
+                    'url': media['url'],
                     'tbr': int_or_none(media.get('bitrate')),
                     'filesize': int_or_none(media.get('fileSize')),
                     'ext': ext,

youtube_dl/extractor/animeondemand.py

@@ -8,7 +8,6 @@ from ..utils import (
     determine_ext,
     extract_attributes,
     ExtractorError,
-    url_or_none,
     urlencode_postdata,
     urljoin,
 )
@@ -53,7 +52,7 @@ class AnimeOnDemandIE(InfoExtractor):
     }]
 
     def _login(self):
-        username, password = self._get_login_info()
+        (username, password) = self._get_login_info()
         if username is None:
             return
 
@@ -166,7 +165,7 @@ class AnimeOnDemandIE(InfoExtractor):
                         }, fatal=False)
                     if not playlist:
                         continue
-                    stream_url = url_or_none(playlist.get('streamurl'))
+                    stream_url = playlist.get('streamurl')
                     if stream_url:
                         rtmp = re.search(
                             r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+/))(?P<playpath>mp[34]:.+)',

youtube_dl/extractor/anvato.py

@@ -134,33 +134,9 @@ class AnvatoIE(InfoExtractor):
         'telemundo': 'anvato_mcp_telemundo_web_prod_c5278d51ad46fda4b6ca3d0ea44a7846a054f582'
     }
 
-    _API_KEY = '3hwbSuqqT690uxjNYBktSQpa5ZrpYYR0Iofx7NcJHyA'
-
     _ANVP_RE = r'<script[^>]+\bdata-anvp\s*=\s*(["\'])(?P<anvp>(?:(?!\1).)+)\1'
     _AUTH_KEY = b'\x31\xc2\x42\x84\x9e\x73\xa0\xce'
 
-    _TESTS = [{
-        # from https://www.boston25news.com/news/watch-humpback-whale-breaches-right-next-to-fishing-boat-near-nh/817484874
-        'url': 'anvato:8v9BEynrwx8EFLYpgfOWcG1qJqyXKlRM:4465496',
-        'info_dict': {
-            'id': '4465496',
-            'ext': 'mp4',
-            'title': 'VIDEO: Humpback whale breaches right next to NH boat',
-            'description': 'VIDEO: Humpback whale breaches right next to NH boat. Footage courtesy: Zach Fahey.',
-            'duration': 22,
-            'timestamp': 1534855680,
-            'upload_date': '20180821',
-            'uploader': 'ANV',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        # from https://sanfrancisco.cbslocal.com/2016/06/17/source-oakland-cop-on-leave-for-having-girlfriend-help-with-police-reports/
-        'url': 'anvato:DVzl9QRzox3ZZsP9bNu5Li3X7obQOnqP:3417601',
-        'only_matching': True,
-    }]
-
     def __init__(self, *args, **kwargs):
         super(AnvatoIE, self).__init__(*args, **kwargs)
         self.__server_time = None
@@ -193,8 +169,7 @@ class AnvatoIE(InfoExtractor):
             'api': {
                 'anvrid': anvrid,
                 'anvstk': md5_text('%s|%s|%d|%s' % (
-                    access_key, anvrid, server_time,
-                    self._ANVACK_TABLE.get(access_key, self._API_KEY))),
+                    access_key, anvrid, server_time, self._ANVACK_TABLE[access_key])),
                 'anvts': server_time,
             },
         }
@@ -302,13 +277,10 @@ class AnvatoIE(InfoExtractor):
 
     def _real_extract(self, url):
         url, smuggled_data = unsmuggle_url(url, {})
-        self._initialize_geo_bypass({
-            'countries': smuggled_data.get('geo_countries'),
-        })
+        self._initialize_geo_bypass(smuggled_data.get('geo_countries'))
 
         mobj = re.match(self._VALID_URL, url)
         access_key, video_id = mobj.group('access_key_or_mcp', 'id')
         if access_key not in self._ANVACK_TABLE:
-            access_key = self._MCP_TO_ACCESS_KEY_TABLE.get(
-                access_key) or access_key
+            access_key = self._MCP_TO_ACCESS_KEY_TABLE[access_key]
         return self._get_anvato_videos(access_key, video_id)

youtube_dl/extractor/aol.py

@@ -7,7 +7,6 @@ from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
     int_or_none,
-    url_or_none,
 )
 
 
@@ -78,7 +77,7 @@ class AolIE(InfoExtractor):
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
         for rendition in video_data.get('renditions', []):
-            video_url = url_or_none(rendition.get('url'))
+            video_url = rendition.get('url')
             if not video_url:
                 continue
             ext = rendition.get('format')

youtube_dl/extractor/apa.py

@@ -1,94 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import (
-    determine_ext,
-    js_to_json,
-    url_or_none,
-)
-
-
-class APAIE(InfoExtractor):
-    _VALID_URL = r'https?://[^/]+\.apa\.at/embed/(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
-    _TESTS = [{
-        'url': 'http://uvp.apa.at/embed/293f6d17-692a-44e3-9fd5-7b178f3a1029',
-        'md5': '2b12292faeb0a7d930c778c7a5b4759b',
-        'info_dict': {
-            'id': 'jjv85FdZ',
-            'ext': 'mp4',
-            'title': '"Blau ist mysteriös": Die Blue Man Group im Interview',
-            'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 254,
-            'timestamp': 1519211149,
-            'upload_date': '20180221',
-        },
-    }, {
-        'url': 'https://uvp-apapublisher.sf.apa.at/embed/2f94e9e6-d945-4db2-9548-f9a41ebf7b78',
-        'only_matching': True,
-    }, {
-        'url': 'http://uvp-rma.sf.apa.at/embed/70404cca-2f47-4855-bbb8-20b1fae58f76',
-        'only_matching': True,
-    }, {
-        'url': 'http://uvp-kleinezeitung.sf.apa.at/embed/f1c44979-dba2-4ebf-b021-e4cf2cac3c81',
-        'only_matching': True,
-    }]
-
-    @staticmethod
-    def _extract_urls(webpage):
-        return [
-            mobj.group('url')
-            for mobj in re.finditer(
-                r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//[^/]+\.apa\.at/embed/[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12}.*?)\1',
-                webpage)]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, video_id)
-
-        jwplatform_id = self._search_regex(
-            r'media[iI]d\s*:\s*["\'](?P<id>[a-zA-Z0-9]{8})', webpage,
-            'jwplatform id', default=None)
-
-        if jwplatform_id:
-            return self.url_result(
-                'jwplatform:' + jwplatform_id, ie='JWPlatform',
-                video_id=video_id)
-
-        sources = self._parse_json(
-            self._search_regex(
-                r'sources\s*=\s*(\[.+?\])\s*;', webpage, 'sources'),
-            video_id, transform_source=js_to_json)
-
-        formats = []
-        for source in sources:
-            if not isinstance(source, dict):
-                continue
-            source_url = url_or_none(source.get('file'))
-            if not source_url:
-                continue
-            ext = determine_ext(source_url)
-            if ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
-                    source_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                    m3u8_id='hls', fatal=False))
-            else:
-                formats.append({
-                    'url': source_url,
-                })
-        self._sort_formats(formats)
-
-        thumbnail = self._search_regex(
-            r'image\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage,
-            'thumbnail', fatal=False, group='url')
-
-        return {
-            'id': video_id,
-            'title': video_id,
-            'thumbnail': thumbnail,
-            'formats': formats,
-        }

youtube_dl/extractor/aparat.py

@@ -4,92 +4,66 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
     int_or_none,
-    merge_dicts,
     mimetype2ext,
-    url_or_none,
 )
 
 
 class AparatIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?aparat\.com/(?:v/|video/video/embed/videohash/)(?P<id>[a-zA-Z0-9]+)'
 
-    _TESTS = [{
+    _TEST = {
         'url': 'http://www.aparat.com/v/wP8On',
         'md5': '131aca2e14fe7c4dcb3c4877ba300c89',
         'info_dict': {
             'id': 'wP8On',
             'ext': 'mp4',
             'title': 'تیم گلکسی 11 - زومیت',
-            'description': 'md5:096bdabcdcc4569f2b8a5e903a3b3028',
-            'duration': 231,
-            'timestamp': 1387394859,
-            'upload_date': '20131218',
-            'view_count': int,
+            'age_limit': 0,
         },
-    }, {
-        # multiple formats
-        'url': 'https://www.aparat.com/v/8dflw/',
-        'only_matching': True,
-    }]
+        # 'skip': 'Extremely unreliable',
+    }
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        # Provides more metadata
-        webpage = self._download_webpage(url, video_id, fatal=False)
-
-        if not webpage:
-            # Note: There is an easier-to-parse configuration at
-            # http://www.aparat.com/video/video/config/videohash/%video_id
-            # but the URL in there does not work
-            webpage = self._download_webpage(
-                'http://www.aparat.com/video/video/embed/vt/frame/showvideo/yes/videohash/' + video_id,
-                video_id)
-
-        options = self._parse_json(
-            self._search_regex(
-                r'options\s*=\s*JSON\.parse\(\s*(["\'])(?P<value>(?:(?!\1).)+)\1\s*\)',
-                webpage, 'options', group='value'),
+        # Note: There is an easier-to-parse configuration at
+        # http://www.aparat.com/video/video/config/videohash/%video_id
+        # but the URL in there does not work
+        webpage = self._download_webpage(
+            'http://www.aparat.com/video/video/embed/vt/frame/showvideo/yes/videohash/' + video_id,
             video_id)
 
-        player = options['plugins']['sabaPlayerPlugin']
+        title = self._search_regex(r'\s+title:\s*"([^"]+)"', webpage, 'title')
+
+        file_list = self._parse_json(
+            self._search_regex(
+                r'fileList\s*=\s*JSON\.parse\(\'([^\']+)\'\)', webpage,
+                'file list'),
+            video_id)
 
         formats = []
-        for sources in player['multiSRC']:
-            for item in sources:
-                if not isinstance(item, dict):
-                    continue
-                file_url = url_or_none(item.get('src'))
-                if not file_url:
-                    continue
-                item_type = item.get('type')
-                if item_type == 'application/vnd.apple.mpegurl':
-                    formats.extend(self._extract_m3u8_formats(
-                        file_url, video_id, 'mp4',
-                        entry_protocol='m3u8_native', m3u8_id='hls',
-                        fatal=False))
-                else:
-                    ext = mimetype2ext(item.get('type'))
-                    label = item.get('label')
-                    formats.append({
-                        'url': file_url,
-                        'ext': ext,
-                        'format_id': 'http-%s' % (label or ext),
-                        'height': int_or_none(self._search_regex(
-                            r'(\d+)[pP]', label or '', 'height',
-                            default=None)),
-                    })
-        self._sort_formats(
-            formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+        for item in file_list[0]:
+            file_url = item.get('file')
+            if not file_url:
+                continue
+            ext = mimetype2ext(item.get('type'))
+            label = item.get('label')
+            formats.append({
+                'url': file_url,
+                'ext': ext,
+                'format_id': label or ext,
+                'height': int_or_none(self._search_regex(
+                    r'(\d+)[pP]', label or '', 'height', default=None)),
+            })
+        self._sort_formats(formats)
 
-        info = self._search_json_ld(webpage, video_id, default={})
+        thumbnail = self._search_regex(
+            r'image:\s*"([^"]+)"', webpage, 'thumbnail', fatal=False)
 
-        if not info.get('title'):
-            info['title'] = player['title']
-
-        return merge_dicts(info, {
+        return {
             'id': video_id,
-            'thumbnail': url_or_none(options.get('poster')),
-            'duration': int_or_none(player.get('duration')),
+            'title': title,
+            'thumbnail': thumbnail,
+            'age_limit': self._family_friendly_search(webpage),
             'formats': formats,
-        })
+        }

youtube_dl/extractor/archiveorg.py

@@ -41,7 +41,7 @@ class ArchiveOrgIE(InfoExtractor):
         webpage = self._download_webpage(
             'http://archive.org/embed/' + video_id, video_id)
         jwplayer_playlist = self._parse_json(self._search_regex(
-            r"(?s)Play\('[^']+'\s*,\s*(\[.+\])\s*,\s*{.*?}\)",
+            r"(?s)Play\('[^']+'\s*,\s*(\[.+\])\s*,\s*{.*?}\);",
             webpage, 'jwplayer playlist'), video_id)
         info = self._parse_jwplayer_data(
             {'playlist': jwplayer_playlist}, video_id, base_url=url)

youtube_dl/extractor/ard.py

@@ -5,64 +5,78 @@ import re
 
 from .common import InfoExtractor
 from .generic import GenericIE
+from ..compat import compat_str
 from ..utils import (
     determine_ext,
     ExtractorError,
+    qualities,
     int_or_none,
     parse_duration,
-    qualities,
-    str_or_none,
-    try_get,
     unified_strdate,
-    unified_timestamp,
-    update_url_query,
-    url_or_none,
     xpath_text,
+    update_url_query,
 )
 from ..compat import compat_etree_fromstring
 
 
 class ARDMediathekIE(InfoExtractor):
     IE_NAME = 'ARD:mediathek'
-    _VALID_URL = r'^https?://(?:(?:(?:www|classic)\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de|one\.ard\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
+    _VALID_URL = r'^https?://(?:(?:www\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
 
     _TESTS = [{
-        # available till 26.07.2022
-        'url': 'http://www.ardmediathek.de/tv/S%C3%9CDLICHT/Was-ist-die-Kunst-der-Zukunft-liebe-Ann/BR-Fernsehen/Video?bcastId=34633636&documentId=44726822',
+        'url': 'http://www.ardmediathek.de/tv/Dokumentation-und-Reportage/Ich-liebe-das-Leben-trotzdem/rbb-Fernsehen/Video?documentId=29582122&bcastId=3822114',
         'info_dict': {
-            'id': '44726822',
+            'id': '29582122',
             'ext': 'mp4',
-            'title': 'Was ist die Kunst der Zukunft, liebe Anna McCarthy?',
-            'description': 'md5:4ada28b3e3b5df01647310e41f3a62f5',
-            'duration': 1740,
+            'title': 'Ich liebe das Leben trotzdem',
+            'description': 'md5:45e4c225c72b27993314b31a84a5261c',
+            'duration': 4557,
         },
         'params': {
             # m3u8 download
             'skip_download': True,
-        }
+        },
+        'skip': 'HTTP Error 404: Not Found',
     }, {
-        'url': 'https://one.ard.de/tv/Mord-mit-Aussicht/Mord-mit-Aussicht-6-39-T%C3%B6dliche-Nach/ONE/Video?bcastId=46384294&documentId=55586872',
-        'only_matching': True,
+        'url': 'http://www.ardmediathek.de/tv/Tatort/Tatort-Scheinwelten-H%C3%B6rfassung-Video/Das-Erste/Video?documentId=29522730&bcastId=602916',
+        'md5': 'f4d98b10759ac06c0072bbcd1f0b9e3e',
+        'info_dict': {
+            'id': '29522730',
+            'ext': 'mp4',
+            'title': 'Tatort: Scheinwelten - Hörfassung (Video tgl. ab 20 Uhr)',
+            'description': 'md5:196392e79876d0ac94c94e8cdb2875f1',
+            'duration': 5252,
+        },
+        'skip': 'HTTP Error 404: Not Found',
     }, {
         # audio
         'url': 'http://www.ardmediathek.de/tv/WDR-H%C3%B6rspiel-Speicher/Tod-eines-Fu%C3%9Fballers/WDR-3/Audio-Podcast?documentId=28488308&bcastId=23074086',
-        'only_matching': True,
+        'md5': '219d94d8980b4f538c7fcb0865eb7f2c',
+        'info_dict': {
+            'id': '28488308',
+            'ext': 'mp3',
+            'title': 'Tod eines Fußballers',
+            'description': 'md5:f6e39f3461f0e1f54bfa48c8875c86ef',
+            'duration': 3240,
+        },
+        'skip': 'HTTP Error 404: Not Found',
     }, {
         'url': 'http://mediathek.daserste.de/sendungen_a-z/328454_anne-will/22429276_vertrauen-ist-gut-spionieren-ist-besser-geht',
         'only_matching': True,
     }, {
         # audio
         'url': 'http://mediathek.rbb-online.de/radio/Hörspiel/Vor-dem-Fest/kulturradio/Audio?documentId=30796318&topRessort=radio&bcastId=9839158',
-        'only_matching': True,
-    }, {
-        'url': 'https://classic.ardmediathek.de/tv/Panda-Gorilla-Co/Panda-Gorilla-Co-Folge-274/Das-Erste/Video?bcastId=16355486&documentId=58234698',
-        'only_matching': True,
+        'md5': '4e8f00631aac0395fee17368ac0e9867',
+        'info_dict': {
+            'id': '30796318',
+            'ext': 'mp3',
+            'title': 'Vor dem Fest',
+            'description': 'md5:c0c1c8048514deaed2a73b3a60eecacb',
+            'duration': 3287,
+        },
+        'skip': 'Video is no longer available',
     }]
 
-    @classmethod
-    def suitable(cls, url):
-        return False if ARDBetaMediathekIE.suitable(url) else super(ARDMediathekIE, cls).suitable(url)
-
     def _extract_media_info(self, media_info_url, webpage, video_id):
         media_info = self._download_json(
             media_info_url, video_id, 'Downloading media JSON')
@@ -113,7 +127,7 @@ class ARDMediathekIE(InfoExtractor):
                 quality = stream.get('_quality')
                 server = stream.get('_server')
                 for stream_url in stream_urls:
-                    if not url_or_none(stream_url):
+                    if not isinstance(stream_url, compat_str) or '//' not in stream_url:
                         continue
                     ext = determine_ext(stream_url)
                     if quality != 'auto' and ext in ('f4m', 'm3u8'):
@@ -183,18 +197,13 @@ class ARDMediathekIE(InfoExtractor):
         title = self._html_search_regex(
             [r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
              r'<meta name="dcterms\.title" content="(.*?)"/>',
-             r'<h4 class="headline">(.*?)</h4>',
-             r'<title[^>]*>(.*?)</title>'],
+             r'<h4 class="headline">(.*?)</h4>'],
             webpage, 'title')
         description = self._html_search_meta(
             'dcterms.abstract', webpage, 'description', default=None)
         if description is None:
             description = self._html_search_meta(
-                'description', webpage, 'meta description', default=None)
-        if description is None:
-            description = self._html_search_regex(
-                r'<p\s+class="teasertext">(.+?)</p>',
-                webpage, 'teaser text', default=None)
+                'description', webpage, 'meta description')
 
         # Thumbnail is sometimes not present.
         # It is in the mobile version, but that seems to use a different URL
@@ -243,23 +252,20 @@ class ARDMediathekIE(InfoExtractor):
 
 class ARDIE(InfoExtractor):
     _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
-    _TESTS = [{
-        # available till 14.02.2019
-        'url': 'http://www.daserste.de/information/talk/maischberger/videos/das-groko-drama-zerlegen-sich-die-volksparteien-video-102.html',
-        'md5': '8e4ec85f31be7c7fc08a26cdbc5a1f49',
+    _TEST = {
+        'url': 'http://www.daserste.de/information/reportage-dokumentation/dokus/videos/die-story-im-ersten-mission-unter-falscher-flagge-100.html',
+        'md5': 'd216c3a86493f9322545e045ddc3eb35',
         'info_dict': {
-            'display_id': 'das-groko-drama-zerlegen-sich-die-volksparteien-video',
-            'id': '102',
+            'display_id': 'die-story-im-ersten-mission-unter-falscher-flagge',
+            'id': '100',
             'ext': 'mp4',
-            'duration': 4435.0,
-            'title': 'Das GroKo-Drama: Zerlegen sich die Volksparteien?',
-            'upload_date': '20180214',
+            'duration': 2600,
+            'title': 'Die Story im Ersten: Mission unter falscher Flagge',
+            'upload_date': '20140804',
             'thumbnail': r're:^https?://.*\.jpg$',
         },
-    }, {
-        'url': 'http://www.daserste.de/information/reportage-dokumentation/dokus/videos/die-story-im-ersten-mission-unter-falscher-flagge-100.html',
-        'only_matching': True,
-    }]
+        'skip': 'HTTP Error 404: Not Found',
+    }
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
@@ -300,101 +306,3 @@ class ARDIE(InfoExtractor):
             'upload_date': upload_date,
             'thumbnail': thumbnail,
         }
-
-
-class ARDBetaMediathekIE(InfoExtractor):
-    _VALID_URL = r'https://(?:beta|www)\.ardmediathek\.de/[^/]+/(?:player|live)/(?P<video_id>[a-zA-Z0-9]+)(?:/(?P<display_id>[^/?#]+))?'
-    _TESTS = [{
-        'url': 'https://beta.ardmediathek.de/ard/player/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE/die-robuste-roswita',
-        'md5': '2d02d996156ea3c397cfc5036b5d7f8f',
-        'info_dict': {
-            'display_id': 'die-robuste-roswita',
-            'id': 'Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
-            'title': 'Tatort: Die robuste Roswita',
-            'description': r're:^Der Mord.*trüber ist als die Ilm.',
-            'duration': 5316,
-            'thumbnail': 'https://img.ardmediathek.de/standard/00/55/43/59/34/-1774185891/16x9/960?mandant=ard',
-            'upload_date': '20180826',
-            'ext': 'mp4',
-        },
-    }, {
-        'url': 'https://www.ardmediathek.de/ard/player/Y3JpZDovL3N3ci5kZS9hZXgvbzEwNzE5MTU/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.ardmediathek.de/swr/live/Y3JpZDovL3N3ci5kZS8xMzQ4MTA0Mg',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('video_id')
-        display_id = mobj.group('display_id') or video_id
-
-        webpage = self._download_webpage(url, display_id)
-        data_json = self._search_regex(r'window\.__APOLLO_STATE__\s*=\s*(\{.*);\n', webpage, 'json')
-        data = self._parse_json(data_json, display_id)
-
-        res = {
-            'id': video_id,
-            'display_id': display_id,
-        }
-        formats = []
-        subtitles = {}
-        geoblocked = False
-        for widget in data.values():
-            if widget.get('_geoblocked') is True:
-                geoblocked = True
-            if '_duration' in widget:
-                res['duration'] = int_or_none(widget['_duration'])
-            if 'clipTitle' in widget:
-                res['title'] = widget['clipTitle']
-            if '_previewImage' in widget:
-                res['thumbnail'] = widget['_previewImage']
-            if 'broadcastedOn' in widget:
-                res['timestamp'] = unified_timestamp(widget['broadcastedOn'])
-            if 'synopsis' in widget:
-                res['description'] = widget['synopsis']
-            subtitle_url = url_or_none(widget.get('_subtitleUrl'))
-            if subtitle_url:
-                subtitles.setdefault('de', []).append({
-                    'ext': 'ttml',
-                    'url': subtitle_url,
-                })
-            if '_quality' in widget:
-                format_url = url_or_none(try_get(
-                    widget, lambda x: x['_stream']['json'][0]))
-                if not format_url:
-                    continue
-                ext = determine_ext(format_url)
-                if ext == 'f4m':
-                    formats.extend(self._extract_f4m_formats(
-                        format_url + '?hdcore=3.11.0',
-                        video_id, f4m_id='hds', fatal=False))
-                elif ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        format_url, video_id, 'mp4', m3u8_id='hls',
-                        fatal=False))
-                else:
-                    # HTTP formats are not available when geoblocked is True,
-                    # other formats are fine though
-                    if geoblocked:
-                        continue
-                    quality = str_or_none(widget.get('_quality'))
-                    formats.append({
-                        'format_id': ('http-' + quality) if quality else 'http',
-                        'url': format_url,
-                        'preference': 10,  # Plain HTTP, that's nice
-                    })
-
-        if not formats and geoblocked:
-            self.raise_geo_restricted(
-                msg='This video is not available due to geoblocking',
-                countries=['DE'])
-
-        self._sort_formats(formats)
-        res.update({
-            'subtitles': subtitles,
-            'formats': formats,
-        })
-
-        return res

youtube_dl/extractor/asiancrush.py

@@ -8,6 +8,7 @@ from .kaltura import KalturaIE
 from ..utils import (
     extract_attributes,
     remove_end,
+    urlencode_postdata,
 )
 
 
@@ -33,40 +34,19 @@ class AsianCrushIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, video_id)
+        data = self._download_json(
+            'https://www.asiancrush.com/wp-admin/admin-ajax.php', video_id,
+            data=urlencode_postdata({
+                'postid': video_id,
+                'action': 'get_channel_kaltura_vars',
+            }))
 
-        entry_id, partner_id, title = [None] * 3
-
-        vars = self._parse_json(
-            self._search_regex(
-                r'iEmbedVars\s*=\s*({.+?})', webpage, 'embed vars',
-                default='{}'), video_id, fatal=False)
-        if vars:
-            entry_id = vars.get('entry_id')
-            partner_id = vars.get('partner_id')
-            title = vars.get('vid_label')
-
-        if not entry_id:
-            entry_id = self._search_regex(
-                r'\bentry_id["\']\s*:\s*["\'](\d+)', webpage, 'entry id')
-
-        player = self._download_webpage(
-            'https://api.asiancrush.com/embeddedVideoPlayer', video_id,
-            query={'id': entry_id})
-
-        kaltura_id = self._search_regex(
-            r'entry_id["\']\s*:\s*(["\'])(?P<id>(?:(?!\1).)+)\1', player,
-            'kaltura id', group='id')
-
-        if not partner_id:
-            partner_id = self._search_regex(
-                r'/p(?:artner_id)?/(\d+)', player, 'partner id',
-                default='513551')
+        entry_id = data['entry_id']
 
         return self.url_result(
-            'kaltura:%s:%s' % (partner_id, kaltura_id),
-            ie=KalturaIE.ie_key(), video_id=kaltura_id,
-            video_title=title)
+            'kaltura:%s:%s' % (data['partner_id'], entry_id),
+            ie=KalturaIE.ie_key(), video_id=entry_id,
+            video_title=data.get('vid_label'))
 
 
 class AsianCrushPlaylistIE(InfoExtractor):

youtube_dl/extractor/atresplayer.py

@@ -74,7 +74,7 @@ class AtresPlayerIE(InfoExtractor):
         self._login()
 
     def _login(self):
-        username, password = self._get_login_info()
+        (username, password) = self._get_login_info()
         if username is None:
             return
 

youtube_dl/extractor/atvat.py

@@ -28,10 +28,8 @@ class ATVAtIE(InfoExtractor):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
         video_data = self._parse_json(unescapeHTML(self._search_regex(
-            [r'flashPlayerOptions\s*=\s*(["\'])(?P<json>(?:(?!\1).)+)\1',
-             r'class="[^"]*jsb_video/FlashPlayer[^"]*"[^>]+data-jsb="(?P<json>[^"]+)"'],
-            webpage, 'player data', group='json')),
-            display_id)['config']['initial_video']
+            r'class="[^"]*jsb_video/FlashPlayer[^"]*"[^>]+data-jsb="([^"]+)"',
+            webpage, 'player data')), display_id)['config']['initial_video']
 
         video_id = video_data['id']
         video_title = video_data['title']

youtube_dl/extractor/audimedia.py

@@ -5,12 +5,13 @@ from .common import InfoExtractor
 from ..utils import (
     int_or_none,
     parse_iso8601,
+    sanitized_Request,
 )
 
 
 class AudiMediaIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?audi-mediacenter\.com/(?:en|de)/audimediatv/(?:video/)?(?P<id>[^/?#]+)'
-    _TESTS = [{
+    _VALID_URL = r'https?://(?:www\.)?audi-mediacenter\.com/(?:en|de)/audimediatv/(?P<id>[^/?#]+)'
+    _TEST = {
         'url': 'https://www.audi-mediacenter.com/en/audimediatv/60-seconds-of-audi-sport-104-2015-wec-bahrain-rookie-test-1467',
         'md5': '79a8b71c46d49042609795ab59779b66',
         'info_dict': {
@@ -23,46 +24,41 @@ class AudiMediaIE(InfoExtractor):
             'duration': 74022,
             'view_count': int,
         }
-    }, {
-        'url': 'https://www.audi-mediacenter.com/en/audimediatv/video/60-seconds-of-audi-sport-104-2015-wec-bahrain-rookie-test-2991',
-        'only_matching': True,
-    }]
+    }
+    # extracted from https://audimedia.tv/assets/embed/embedded-player.js (dataSourceAuthToken)
+    _AUTH_TOKEN = 'e25b42847dba18c6c8816d5d8ce94c326e06823ebf0859ed164b3ba169be97f2'
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
 
         raw_payload = self._search_regex([
-            r'class="amtv-embed"[^>]+id="([0-9a-z-]+)"',
-            r'id="([0-9a-z-]+)"[^>]+class="amtv-embed"',
-            r'class=\\"amtv-embed\\"[^>]+id=\\"([0-9a-z-]+)\\"',
-            r'id=\\"([0-9a-z-]+)\\"[^>]+class=\\"amtv-embed\\"',
-            r'id=(?:\\)?"(amtve-[a-z]-\d+-[a-z]{2})',
+            r'class="amtv-embed"[^>]+id="([^"]+)"',
+            r'class=\\"amtv-embed\\"[^>]+id=\\"([^"]+)\\"',
         ], webpage, 'raw payload')
-        _, stage_mode, video_id, _ = raw_payload.split('-')
+        _, stage_mode, video_id, lang = raw_payload.split('-')
 
         # TODO: handle s and e stage_mode (live streams and ended live streams)
         if stage_mode not in ('s', 'e'):
-            video_data = self._download_json(
-                'https://www.audimedia.tv/api/video/v1/videos/' + video_id,
-                video_id, query={
-                    'embed[]': ['video_versions', 'thumbnail_image'],
-                })['results']
+            request = sanitized_Request(
+                'https://audimedia.tv/api/video/v1/videos/%s?embed[]=video_versions&embed[]=thumbnail_image&where[content_language_iso]=%s' % (video_id, lang),
+                headers={'X-Auth-Token': self._AUTH_TOKEN})
+            json_data = self._download_json(request, video_id)['results']
             formats = []
 
-            stream_url_hls = video_data.get('stream_url_hls')
+            stream_url_hls = json_data.get('stream_url_hls')
             if stream_url_hls:
                 formats.extend(self._extract_m3u8_formats(
                     stream_url_hls, video_id, 'mp4',
                     entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
 
-            stream_url_hds = video_data.get('stream_url_hds')
+            stream_url_hds = json_data.get('stream_url_hds')
             if stream_url_hds:
                 formats.extend(self._extract_f4m_formats(
                     stream_url_hds + '?hdcore=3.4.0',
                     video_id, f4m_id='hds', fatal=False))
 
-            for video_version in video_data.get('video_versions', []):
+            for video_version in json_data.get('video_versions'):
                 video_version_url = video_version.get('download_url') or video_version.get('stream_url')
                 if not video_version_url:
                     continue
@@ -83,11 +79,11 @@ class AudiMediaIE(InfoExtractor):
 
             return {
                 'id': video_id,
-                'title': video_data['title'],
-                'description': video_data.get('subtitle'),
-                'thumbnail': video_data.get('thumbnail_image', {}).get('file'),
-                'timestamp': parse_iso8601(video_data.get('publication_date')),
-                'duration': int_or_none(video_data.get('duration')),
-                'view_count': int_or_none(video_data.get('view_count')),
+                'title': json_data['title'],
+                'description': json_data.get('subtitle'),
+                'thumbnail': json_data.get('thumbnail_image', {}).get('file'),
+                'timestamp': parse_iso8601(json_data.get('publication_date')),
+                'duration': int_or_none(json_data.get('duration')),
+                'view_count': int_or_none(json_data.get('view_count')),
                 'formats': formats,
             }

youtube_dl/extractor/audiomack.py

@@ -62,10 +62,10 @@ class AudiomackIE(InfoExtractor):
         # Audiomack wraps a lot of soundcloud tracks in their branded wrapper
         # if so, pass the work off to the soundcloud extractor
         if SoundcloudIE.suitable(api_response['url']):
-            return self.url_result(api_response['url'], SoundcloudIE.ie_key())
+            return {'_type': 'url', 'url': api_response['url'], 'ie_key': 'Soundcloud'}
 
         return {
-            'id': compat_str(api_response.get('id', album_url_tag)),
+            'id': api_response.get('id', album_url_tag),
             'uploader': api_response.get('artist'),
             'title': api_response.get('title'),
             'url': api_response['url'],

youtube_dl/extractor/azmedien.py

@@ -1,86 +1,213 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import json
 import re
 
 from .common import InfoExtractor
 from .kaltura import KalturaIE
+from ..utils import (
+    get_element_by_class,
+    get_element_by_id,
+    strip_or_none,
+    urljoin,
+)
 
 
-class AZMedienIE(InfoExtractor):
+class AZMedienBaseIE(InfoExtractor):
+    def _kaltura_video(self, partner_id, entry_id):
+        return self.url_result(
+            'kaltura:%s:%s' % (partner_id, entry_id), ie=KalturaIE.ie_key(),
+            video_id=entry_id)
+
+
+class AZMedienIE(AZMedienBaseIE):
     IE_DESC = 'AZ Medien videos'
     _VALID_URL = r'''(?x)
                     https?://
                         (?:www\.)?
-                        (?P<host>
+                        (?:
                             telezueri\.ch|
                             telebaern\.tv|
                             telem1\.ch
                         )/
-                        [^/]+/
-                        (?P<id>
-                            [^/]+-(?P<article_id>\d+)
-                        )
+                        [0-9]+-show-[^/\#]+
                         (?:
-                            \#video=
-                            (?P<kaltura_id>
-                                [_0-9a-z]+
-                            )
-                        )?
+                            /[0-9]+-episode-[^/\#]+
+                            (?:
+                                /[0-9]+-segment-(?:[^/\#]+\#)?|
+                                \#
+                            )|
+                            \#
+                        )
+                        (?P<id>[^\#]+)
                     '''
 
     _TESTS = [{
-        'url': 'https://www.telezueri.ch/sonntalk/bundesrats-vakanzen-eu-rahmenabkommen-133214569',
+        # URL with 'segment'
+        'url': 'http://www.telezueri.ch/62-show-zuerinews/13772-episode-sonntag-18-dezember-2016/32419-segment-massenabweisungen-beim-hiltl-club-wegen-pelzboom',
         'info_dict': {
-            'id': '1_anruz3wy',
+            'id': '1_2444peh4',
             'ext': 'mp4',
-            'title': 'Bundesrats-Vakanzen / EU-Rahmenabkommen',
-            'uploader_id': 'TVOnline',
-            'upload_date': '20180930',
-            'timestamp': 1538328802,
+            'title': 'Massenabweisungen beim Hiltl Club wegen Pelzboom',
+            'description': 'md5:9ea9dd1b159ad65b36ddcf7f0d7c76a8',
+            'uploader_id': 'TeleZ?ri',
+            'upload_date': '20161218',
+            'timestamp': 1482084490,
         },
         'params': {
             'skip_download': True,
         },
     }, {
-        'url': 'https://www.telebaern.tv/telebaern-news/montag-1-oktober-2018-ganze-sendung-133531189#video=0_7xjo9lf1',
+        # URL with 'segment' and fragment:
+        'url': 'http://www.telebaern.tv/118-show-news/14240-episode-dienstag-17-januar-2017/33666-segment-achtung-gefahr#zu-wenig-pflegerinnen-und-pfleger',
+        'only_matching': True
+    }, {
+        # URL with 'episode' and fragment:
+        'url': 'http://www.telem1.ch/47-show-sonntalk/13986-episode-soldaten-fuer-grenzschutz-energiestrategie-obama-bilanz#soldaten-fuer-grenzschutz-energiestrategie-obama-bilanz',
+        'only_matching': True
+    }, {
+        # URL with 'show' and fragment:
+        'url': 'http://www.telezueri.ch/66-show-sonntalk#burka-plakate-trump-putin-china-besuch',
         'only_matching': True
     }]
 
-    _PARTNER_ID = '1719221'
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, video_id)
+
+        partner_id = self._search_regex(
+            r'<script[^>]+src=["\'](?:https?:)?//(?:[^/]+\.)?kaltura\.com(?:/[^/]+)*/(?:p|partner_id)/([0-9]+)',
+            webpage, 'kaltura partner id')
+        entry_id = self._html_search_regex(
+            r'<a[^>]+data-id=(["\'])(?P<id>(?:(?!\1).)+)\1[^>]+data-slug=["\']%s'
+            % re.escape(video_id), webpage, 'kaltura entry id', group='id')
+
+        return self._kaltura_video(partner_id, entry_id)
+
+
+class AZMedienPlaylistIE(AZMedienBaseIE):
+    IE_DESC = 'AZ Medien playlists'
+    _VALID_URL = r'''(?x)
+                    https?://
+                        (?:www\.)?
+                        (?:
+                            telezueri\.ch|
+                            telebaern\.tv|
+                            telem1\.ch
+                        )/
+                        (?P<id>[0-9]+-
+                            (?:
+                                show|
+                                topic|
+                                themen
+                            )-[^/\#]+
+                            (?:
+                                /[0-9]+-episode-[^/\#]+
+                            )?
+                        )$
+                    '''
+
+    _TESTS = [{
+        # URL with 'episode'
+        'url': 'http://www.telebaern.tv/118-show-news/13735-episode-donnerstag-15-dezember-2016',
+        'info_dict': {
+            'id': '118-show-news/13735-episode-donnerstag-15-dezember-2016',
+            'title': 'News - Donnerstag, 15. Dezember 2016',
+        },
+        'playlist_count': 9,
+    }, {
+        # URL with 'themen'
+        'url': 'http://www.telem1.ch/258-themen-tele-m1-classics',
+        'info_dict': {
+            'id': '258-themen-tele-m1-classics',
+            'title': 'Tele M1 Classics',
+        },
+        'playlist_mincount': 15,
+    }, {
+        # URL with 'topic', contains nested playlists
+        'url': 'http://www.telezueri.ch/219-topic-aera-trump-hat-offiziell-begonnen',
+        'only_matching': True,
+    }, {
+        # URL with 'show' only
+        'url': 'http://www.telezueri.ch/86-show-talktaeglich',
+        'only_matching': True
+    }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        host = mobj.group('host')
-        video_id = mobj.group('id')
-        entry_id = mobj.group('kaltura_id')
+        show_id = self._match_id(url)
+        webpage = self._download_webpage(url, show_id)
 
-        if not entry_id:
-            api_url = 'https://www.%s/api/pub/gql/%s' % (host, host.split('.')[0])
-            payload = {
-                'query': '''query VideoContext($articleId: ID!) {
-                    article: node(id: $articleId) {
-                      ... on Article {
-                        mainAssetRelation {
-                          asset {
-                            ... on VideoAsset {
-                              kalturaId
-                            }
-                          }
-                        }
-                      }
-                    }
-                  }''',
-                'variables': {'articleId': 'Article:%s' % mobj.group('article_id')},
-            }
-            json_data = self._download_json(
-                api_url, video_id, headers={
-                    'Content-Type': 'application/json',
-                },
-                data=json.dumps(payload).encode())
-            entry_id = json_data['data']['article']['mainAssetRelation']['asset']['kalturaId']
+        entries = []
 
-        return self.url_result(
-            'kaltura:%s:%s' % (self._PARTNER_ID, entry_id),
-            ie=KalturaIE.ie_key(), video_id=entry_id)
+        partner_id = self._search_regex(
+            r'src=["\'](?:https?:)?//(?:[^/]+\.)kaltura\.com/(?:[^/]+/)*(?:p|partner_id)/(\d+)',
+            webpage, 'kaltura partner id', default=None)
+
+        if partner_id:
+            entries = [
+                self._kaltura_video(partner_id, m.group('id'))
+                for m in re.finditer(
+                    r'data-id=(["\'])(?P<id>(?:(?!\1).)+)\1', webpage)]
+
+        if not entries:
+            entries = [
+                self.url_result(m.group('url'), ie=AZMedienIE.ie_key())
+                for m in re.finditer(
+                    r'<a[^>]+data-real=(["\'])(?P<url>http.+?)\1', webpage)]
+
+        if not entries:
+            entries = [
+                # May contain nested playlists (e.g. [1]) thus no explicit
+                # ie_key
+                # 1. http://www.telezueri.ch/219-topic-aera-trump-hat-offiziell-begonnen)
+                self.url_result(urljoin(url, m.group('url')))
+                for m in re.finditer(
+                    r'<a[^>]+name=[^>]+href=(["\'])(?P<url>/.+?)\1', webpage)]
+
+        title = self._search_regex(
+            r'episodeShareTitle\s*=\s*(["\'])(?P<title>(?:(?!\1).)+)\1',
+            webpage, 'title',
+            default=strip_or_none(get_element_by_id(
+                'video-title', webpage)), group='title')
+
+        return self.playlist_result(entries, show_id, title)
+
+
+class AZMedienShowPlaylistIE(AZMedienBaseIE):
+    IE_DESC = 'AZ Medien show playlists'
+    _VALID_URL = r'''(?x)
+                    https?://
+                        (?:www\.)?
+                        (?:
+                            telezueri\.ch|
+                            telebaern\.tv|
+                            telem1\.ch
+                        )/
+                        (?:
+                            all-episodes|
+                            alle-episoden
+                        )/
+                        (?P<id>[^/?#&]+)
+                    '''
+
+    _TEST = {
+        'url': 'http://www.telezueri.ch/all-episodes/astrotalk',
+        'info_dict': {
+            'id': 'astrotalk',
+            'title': 'TeleZüri: AstroTalk - alle episoden',
+            'description': 'md5:4c0f7e7d741d906004266e295ceb4a26',
+        },
+        'playlist_mincount': 13,
+    }
+
+    def _real_extract(self, url):
+        playlist_id = self._match_id(url)
+        webpage = self._download_webpage(url, playlist_id)
+        episodes = get_element_by_class('search-mobile-box', webpage)
+        entries = [self.url_result(
+            urljoin(url, m.group('url'))) for m in re.finditer(
+                r'<a[^>]+href=(["\'])(?P<url>(?:(?!\1).)+)\1', episodes)]
+        title = self._og_search_title(webpage, fatal=False)
+        description = self._og_search_description(webpage)
+        return self.playlist_result(entries, playlist_id, title, description)

youtube_dl/extractor/bambuser.py

@@ -44,7 +44,7 @@ class BambuserIE(InfoExtractor):
     }
 
     def _login(self):
-        username, password = self._get_login_info()
+        (username, password) = self._get_login_info()
         if username is None:
             return
 

youtube_dl/extractor/bandcamp.py

@@ -1,5 +1,6 @@
 from __future__ import unicode_literals
 
+import json
 import random
 import re
 import time
@@ -15,18 +16,14 @@ from ..utils import (
     int_or_none,
     KNOWN_EXTENSIONS,
     parse_filesize,
-    str_or_none,
-    try_get,
     unescapeHTML,
     update_url_query,
     unified_strdate,
-    unified_timestamp,
-    url_or_none,
 )
 
 
 class BandcampIE(InfoExtractor):
-    _VALID_URL = r'https?://[^/]+\.bandcamp\.com/track/(?P<title>[^/?#&]+)'
+    _VALID_URL = r'https?://.*?\.bandcamp\.com/track/(?P<title>[^/?#&]+)'
     _TESTS = [{
         'url': 'http://youtube-dl.bandcamp.com/track/youtube-dl-test-song',
         'md5': 'c557841d5e50261777a6585648adf439',
@@ -38,44 +35,13 @@ class BandcampIE(InfoExtractor):
         },
         '_skip': 'There is a limit of 200 free downloads / month for the test song'
     }, {
-        # free download
         'url': 'http://benprunty.bandcamp.com/track/lanius-battle',
-        'md5': '853e35bf34aa1d6fe2615ae612564b36',
+        'md5': '0369ace6b939f0927e62c67a1a8d9fa7',
         'info_dict': {
             'id': '2650410135',
             'ext': 'aiff',
             'title': 'Ben Prunty - Lanius (Battle)',
-            'thumbnail': r're:^https?://.*\.jpg$',
             'uploader': 'Ben Prunty',
-            'timestamp': 1396508491,
-            'upload_date': '20140403',
-            'release_date': '20140403',
-            'duration': 260.877,
-            'track': 'Lanius (Battle)',
-            'track_number': 1,
-            'track_id': '2650410135',
-            'artist': 'Ben Prunty',
-            'album': 'FTL: Advanced Edition Soundtrack',
-        },
-    }, {
-        # no free download, mp3 128
-        'url': 'https://relapsealumni.bandcamp.com/track/hail-to-fire',
-        'md5': 'fec12ff55e804bb7f7ebeb77a800c8b7',
-        'info_dict': {
-            'id': '2584466013',
-            'ext': 'mp3',
-            'title': 'Mastodon - Hail to Fire',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'uploader': 'Mastodon',
-            'timestamp': 1322005399,
-            'upload_date': '20111122',
-            'release_date': '20040207',
-            'duration': 120.79,
-            'track': 'Hail to Fire',
-            'track_number': 5,
-            'track_id': '2584466013',
-            'artist': 'Mastodon',
-            'album': 'Call of the Mastodon',
         },
     }]
 
@@ -84,23 +50,19 @@ class BandcampIE(InfoExtractor):
         title = mobj.group('title')
         webpage = self._download_webpage(url, title)
         thumbnail = self._html_search_meta('og:image', webpage, default=None)
+        m_download = re.search(r'freeDownloadPage: "(.*?)"', webpage)
+        if not m_download:
+            m_trackinfo = re.search(r'trackinfo: (.+),\s*?\n', webpage)
+            if m_trackinfo:
+                json_code = m_trackinfo.group(1)
+                data = json.loads(json_code)[0]
+                track_id = compat_str(data['id'])
 
-        track_id = None
-        track = None
-        track_number = None
-        duration = None
+                if not data.get('file'):
+                    raise ExtractorError('Not streamable', video_id=track_id, expected=True)
 
-        formats = []
-        track_info = self._parse_json(
-            self._search_regex(
-                r'trackinfo\s*:\s*\[\s*({.+?})\s*\]\s*,\s*?\n',
-                webpage, 'track info', default='{}'), title)
-        if track_info:
-            file_ = track_info.get('file')
-            if isinstance(file_, dict):
-                for format_id, format_url in file_.items():
-                    if not url_or_none(format_url):
-                        continue
+                formats = []
+                for format_id, format_url in data['file'].items():
                     ext, abr_str = format_id.split('-', 1)
                     formats.append({
                         'format_id': format_id,
@@ -110,110 +72,85 @@ class BandcampIE(InfoExtractor):
                         'acodec': ext,
                         'abr': int_or_none(abr_str),
                     })
-            track = track_info.get('title')
-            track_id = str_or_none(track_info.get('track_id') or track_info.get('id'))
-            track_number = int_or_none(track_info.get('track_num'))
-            duration = float_or_none(track_info.get('duration'))
 
-        def extract(key):
-            return self._search_regex(
-                r'\b%s\s*["\']?\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1' % key,
-                webpage, key, default=None, group='value')
+                self._sort_formats(formats)
 
-        artist = extract('artist')
-        album = extract('album_title')
-        timestamp = unified_timestamp(
-            extract('publish_date') or extract('album_publish_date'))
-        release_date = unified_strdate(extract('album_release_date'))
+                return {
+                    'id': track_id,
+                    'title': data['title'],
+                    'thumbnail': thumbnail,
+                    'formats': formats,
+                    'duration': float_or_none(data.get('duration')),
+                }
+            else:
+                raise ExtractorError('No free songs found')
 
-        download_link = self._search_regex(
-            r'freeDownloadPage\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage,
-            'download link', default=None, group='url')
-        if download_link:
-            track_id = self._search_regex(
-                r'(?ms)var TralbumData = .*?[{,]\s*id: (?P<id>\d+),?$',
-                webpage, 'track id')
+        download_link = m_download.group(1)
+        video_id = self._search_regex(
+            r'(?ms)var TralbumData = .*?[{,]\s*id: (?P<id>\d+),?$',
+            webpage, 'video id')
 
-            download_webpage = self._download_webpage(
-                download_link, track_id, 'Downloading free downloads page')
+        download_webpage = self._download_webpage(
+            download_link, video_id, 'Downloading free downloads page')
 
-            blob = self._parse_json(
-                self._search_regex(
-                    r'data-blob=(["\'])(?P<blob>{.+?})\1', download_webpage,
-                    'blob', group='blob'),
-                track_id, transform_source=unescapeHTML)
+        blob = self._parse_json(
+            self._search_regex(
+                r'data-blob=(["\'])(?P<blob>{.+?})\1', download_webpage,
+                'blob', group='blob'),
+            video_id, transform_source=unescapeHTML)
 
-            info = try_get(
-                blob, (lambda x: x['digital_items'][0],
-                       lambda x: x['download_items'][0]), dict)
-            if info:
-                downloads = info.get('downloads')
-                if isinstance(downloads, dict):
-                    if not track:
-                        track = info.get('title')
-                    if not artist:
-                        artist = info.get('artist')
-                    if not thumbnail:
-                        thumbnail = info.get('thumb_url')
+        info = blob['digital_items'][0]
 
-                    download_formats = {}
-                    download_formats_list = blob.get('download_formats')
-                    if isinstance(download_formats_list, list):
-                        for f in blob['download_formats']:
-                            name, ext = f.get('name'), f.get('file_extension')
-                            if all(isinstance(x, compat_str) for x in (name, ext)):
-                                download_formats[name] = ext.strip('.')
-
-                    for format_id, f in downloads.items():
-                        format_url = f.get('url')
-                        if not format_url:
-                            continue
-                        # Stat URL generation algorithm is reverse engineered from
-                        # download_*_bundle_*.js
-                        stat_url = update_url_query(
-                            format_url.replace('/download/', '/statdownload/'), {
-                                '.rand': int(time.time() * 1000 * random.random()),
-                            })
-                        format_id = f.get('encoding_name') or format_id
-                        stat = self._download_json(
-                            stat_url, track_id, 'Downloading %s JSON' % format_id,
-                            transform_source=lambda s: s[s.index('{'):s.rindex('}') + 1],
-                            fatal=False)
-                        if not stat:
-                            continue
-                        retry_url = url_or_none(stat.get('retry_url'))
-                        if not retry_url:
-                            continue
-                        formats.append({
-                            'url': self._proto_relative_url(retry_url, 'http:'),
-                            'ext': download_formats.get(format_id),
-                            'format_id': format_id,
-                            'format_note': f.get('description'),
-                            'filesize': parse_filesize(f.get('size_mb')),
-                            'vcodec': 'none',
-                        })
-
-        self._sort_formats(formats)
+        downloads = info['downloads']
+        track = info['title']
 
+        artist = info.get('artist')
         title = '%s - %s' % (artist, track) if artist else track
 
-        if not duration:
-            duration = float_or_none(self._html_search_meta(
-                'duration', webpage, default=None))
+        download_formats = {}
+        for f in blob['download_formats']:
+            name, ext = f.get('name'), f.get('file_extension')
+            if all(isinstance(x, compat_str) for x in (name, ext)):
+                download_formats[name] = ext.strip('.')
+
+        formats = []
+        for format_id, f in downloads.items():
+            format_url = f.get('url')
+            if not format_url:
+                continue
+            # Stat URL generation algorithm is reverse engineered from
+            # download_*_bundle_*.js
+            stat_url = update_url_query(
+                format_url.replace('/download/', '/statdownload/'), {
+                    '.rand': int(time.time() * 1000 * random.random()),
+                })
+            format_id = f.get('encoding_name') or format_id
+            stat = self._download_json(
+                stat_url, video_id, 'Downloading %s JSON' % format_id,
+                transform_source=lambda s: s[s.index('{'):s.rindex('}') + 1],
+                fatal=False)
+            if not stat:
+                continue
+            retry_url = stat.get('retry_url')
+            if not isinstance(retry_url, compat_str):
+                continue
+            formats.append({
+                'url': self._proto_relative_url(retry_url, 'http:'),
+                'ext': download_formats.get(format_id),
+                'format_id': format_id,
+                'format_note': f.get('description'),
+                'filesize': parse_filesize(f.get('size_mb')),
+                'vcodec': 'none',
+            })
+        self._sort_formats(formats)
 
         return {
-            'id': track_id,
+            'id': video_id,
             'title': title,
-            'thumbnail': thumbnail,
-            'uploader': artist,
-            'timestamp': timestamp,
-            'release_date': release_date,
-            'duration': duration,
-            'track': track,
-            'track_number': track_number,
-            'track_id': track_id,
+            'thumbnail': info.get('thumb_url') or thumbnail,
+            'uploader': info.get('artist'),
             'artist': artist,
-            'album': album,
+            'track': track,
             'formats': formats,
         }
 
@@ -369,7 +306,7 @@ class BandcampWeeklyIE(InfoExtractor):
 
         formats = []
         for format_id, format_url in show['audio_stream'].items():
-            if not url_or_none(format_url):
+            if not isinstance(format_url, compat_str):
                 continue
             for known_ext in KNOWN_EXTENSIONS:
                 if known_ext in format_id:

youtube_dl/extractor/bbc.py

@@ -12,7 +12,6 @@ from ..utils import (
     float_or_none,
     get_element_by_class,
     int_or_none,
-    js_to_json,
     parse_duration,
     parse_iso8601,
     try_get,
@@ -21,6 +20,7 @@ from ..utils import (
     urljoin,
 )
 from ..compat import (
+    compat_etree_fromstring,
     compat_HTTPError,
     compat_urlparse,
 )
@@ -29,7 +29,7 @@ from ..compat import (
 class BBCCoUkIE(InfoExtractor):
     IE_NAME = 'bbc.co.uk'
     IE_DESC = 'BBC iPlayer'
-    _ID_REGEX = r'(?:[pbm][\da-z]{7}|w[\da-z]{7,14})'
+    _ID_REGEX = r'[pbw][\da-z]{7}'
     _VALID_URL = r'''(?x)
                     https?://
                         (?:www\.)?bbc\.co\.uk/
@@ -236,12 +236,6 @@ class BBCCoUkIE(InfoExtractor):
         }, {
             'url': 'http://www.bbc.co.uk/programmes/w3csv1y9',
             'only_matching': True,
-        }, {
-            'url': 'https://www.bbc.co.uk/programmes/m00005xn',
-            'only_matching': True,
-        }, {
-            'url': 'https://www.bbc.co.uk/programmes/w172w4dww1jqt5s',
-            'only_matching': True,
         }]
 
     _USP_RE = r'/([^/]+?)\.ism(?:\.hlsv2\.ism)?/[^/]+\.m3u8'
@@ -339,9 +333,14 @@ class BBCCoUkIE(InfoExtractor):
         self._raise_extractor_error(last_exception)
 
     def _download_media_selector_url(self, url, programme_id=None):
-        media_selection = self._download_xml(
-            url, programme_id, 'Downloading media selection XML',
-            expected_status=(403, 404))
+        try:
+            media_selection = self._download_xml(
+                url, programme_id, 'Downloading media selection XML')
+        except ExtractorError as ee:
+            if isinstance(ee.cause, compat_HTTPError) and ee.cause.code in (403, 404):
+                media_selection = compat_etree_fromstring(ee.cause.read().decode('utf-8'))
+            else:
+                raise
         return self._process_media_selector(media_selection, programme_id)
 
     def _process_media_selector(self, media_selection, programme_id):
@@ -773,37 +772,6 @@ class BBCIE(BBCCoUkIE):
         # single video article embedded with data-media-vpid
         'url': 'http://www.bbc.co.uk/sport/rowing/35908187',
         'only_matching': True,
-    }, {
-        'url': 'https://www.bbc.co.uk/bbcthree/clip/73d0bbd0-abc3-4cea-b3c0-cdae21905eb1',
-        'info_dict': {
-            'id': 'p06556y7',
-            'ext': 'mp4',
-            'title': 'Transfers: Cristiano Ronaldo to Man Utd, Arsenal to spend?',
-            'description': 'md5:4b7dfd063d5a789a1512e99662be3ddd',
-        },
-        'params': {
-            'skip_download': True,
-        }
-    }, {
-        # window.__PRELOADED_STATE__
-        'url': 'https://www.bbc.co.uk/radio/play/b0b9z4yl',
-        'info_dict': {
-            'id': 'b0b9z4vz',
-            'ext': 'mp4',
-            'title': 'Prom 6: An American in Paris and Turangalila',
-            'description': 'md5:51cf7d6f5c8553f197e58203bc78dff8',
-            'uploader': 'Radio 3',
-            'uploader_id': 'bbc_radio_three',
-        },
-    }, {
-        'url': 'http://www.bbc.co.uk/learningenglish/chinese/features/lingohack/ep-181227',
-        'info_dict': {
-            'id': 'p06w9tws',
-            'ext': 'mp4',
-            'title': 'md5:2fabf12a726603193a2879a055f72514',
-            'description': 'Learn English words and phrases from this story',
-        },
-        'add_ie': [BBCCoUkIE.ie_key()],
     }]
 
     @classmethod
@@ -954,15 +922,6 @@ class BBCIE(BBCCoUkIE):
         if entries:
             return self.playlist_result(entries, playlist_id, playlist_title, playlist_description)
 
-        # http://www.bbc.co.uk/learningenglish/chinese/features/lingohack/ep-181227
-        group_id = self._search_regex(
-            r'<div[^>]+\bclass=["\']video["\'][^>]+\bdata-pid=["\'](%s)' % self._ID_REGEX,
-            webpage, 'group id', default=None)
-        if playlist_id:
-            return self.url_result(
-                'https://www.bbc.co.uk/programmes/%s' % group_id,
-                ie=BBCCoUkIE.ie_key())
-
         # single video story (e.g. http://www.bbc.com/travel/story/20150625-sri-lankas-spicy-secret)
         programme_id = self._search_regex(
             [r'data-(?:video-player|media)-vpid="(%s)"' % self._ID_REGEX,
@@ -1035,66 +994,6 @@ class BBCIE(BBCCoUkIE):
                     'subtitles': subtitles,
                 }
 
-        preload_state = self._parse_json(self._search_regex(
-            r'window\.__PRELOADED_STATE__\s*=\s*({.+?});', webpage,
-            'preload state', default='{}'), playlist_id, fatal=False)
-        if preload_state:
-            current_programme = preload_state.get('programmes', {}).get('current') or {}
-            programme_id = current_programme.get('id')
-            if current_programme and programme_id and current_programme.get('type') == 'playable_item':
-                title = current_programme.get('titles', {}).get('tertiary') or playlist_title
-                formats, subtitles = self._download_media_selector(programme_id)
-                self._sort_formats(formats)
-                synopses = current_programme.get('synopses') or {}
-                network = current_programme.get('network') or {}
-                duration = int_or_none(
-                    current_programme.get('duration', {}).get('value'))
-                thumbnail = None
-                image_url = current_programme.get('image_url')
-                if image_url:
-                    thumbnail = image_url.replace('{recipe}', '1920x1920')
-                return {
-                    'id': programme_id,
-                    'title': title,
-                    'description': dict_get(synopses, ('long', 'medium', 'short')),
-                    'thumbnail': thumbnail,
-                    'duration': duration,
-                    'uploader': network.get('short_title'),
-                    'uploader_id': network.get('id'),
-                    'formats': formats,
-                    'subtitles': subtitles,
-                }
-
-        bbc3_config = self._parse_json(
-            self._search_regex(
-                r'(?s)bbcthreeConfig\s*=\s*({.+?})\s*;\s*<', webpage,
-                'bbcthree config', default='{}'),
-            playlist_id, transform_source=js_to_json, fatal=False)
-        if bbc3_config:
-            bbc3_playlist = try_get(
-                bbc3_config, lambda x: x['payload']['content']['bbcMedia']['playlist'],
-                dict)
-            if bbc3_playlist:
-                playlist_title = bbc3_playlist.get('title') or playlist_title
-                thumbnail = bbc3_playlist.get('holdingImageURL')
-                entries = []
-                for bbc3_item in bbc3_playlist['items']:
-                    programme_id = bbc3_item.get('versionID')
-                    if not programme_id:
-                        continue
-                    formats, subtitles = self._download_media_selector(programme_id)
-                    self._sort_formats(formats)
-                    entries.append({
-                        'id': programme_id,
-                        'title': playlist_title,
-                        'thumbnail': thumbnail,
-                        'timestamp': timestamp,
-                        'formats': formats,
-                        'subtitles': subtitles,
-                    })
-                return self.playlist_result(
-                    entries, playlist_id, playlist_title, playlist_description)
-
         def extract_all(pattern):
             return list(filter(None, map(
                 lambda s: self._parse_json(s, playlist_id, fatal=False),

youtube_dl/extractor/beeg.py

@@ -1,10 +1,15 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
+    compat_chr,
+    compat_ord,
+    compat_urllib_parse_unquote,
+)
 from ..utils import (
     int_or_none,
-    unified_timestamp,
+    parse_iso8601,
+    urljoin,
 )
 
 
@@ -31,9 +36,29 @@ class BeegIE(InfoExtractor):
 
         webpage = self._download_webpage(url, video_id)
 
-        beeg_version = self._search_regex(
-            r'beeg_version\s*=\s*([\da-zA-Z_-]+)', webpage, 'beeg version',
-            default='1546225636701')
+        cpl_url = self._search_regex(
+            r'<script[^>]+src=(["\'])(?P<url>(?:/static|(?:https?:)?//static\.beeg\.com)/cpl/\d+\.js.*?)\1',
+            webpage, 'cpl', default=None, group='url')
+
+        cpl_url = urljoin(url, cpl_url)
+
+        beeg_version, beeg_salt = [None] * 2
+
+        if cpl_url:
+            cpl = self._download_webpage(
+                self._proto_relative_url(cpl_url), video_id,
+                'Downloading cpl JS', fatal=False)
+            if cpl:
+                beeg_version = int_or_none(self._search_regex(
+                    r'beeg_version\s*=\s*([^\b]+)', cpl,
+                    'beeg version', default=None)) or self._search_regex(
+                    r'/(\d+)\.js', cpl_url, 'beeg version', default=None)
+                beeg_salt = self._search_regex(
+                    r'beeg_salt\s*=\s*(["\'])(?P<beeg_salt>.+?)\1', cpl, 'beeg salt',
+                    default=None, group='beeg_salt')
+
+        beeg_version = beeg_version or '2185'
+        beeg_salt = beeg_salt or 'pmweAkq8lAYKdfWcFCUj0yoVgoPlinamH5UE1CB3H'
 
         for api_path in ('', 'api.'):
             video = self._download_json(
@@ -43,6 +68,37 @@ class BeegIE(InfoExtractor):
             if video:
                 break
 
+        def split(o, e):
+            def cut(s, x):
+                n.append(s[:x])
+                return s[x:]
+            n = []
+            r = len(o) % e
+            if r > 0:
+                o = cut(o, r)
+            while len(o) > e:
+                o = cut(o, e)
+            n.append(o)
+            return n
+
+        def decrypt_key(key):
+            # Reverse engineered from http://static.beeg.com/cpl/1738.js
+            a = beeg_salt
+            e = compat_urllib_parse_unquote(key)
+            o = ''.join([
+                compat_chr(compat_ord(e[n]) - compat_ord(a[n % len(a)]) % 21)
+                for n in range(len(e))])
+            return ''.join(split(o, 3)[::-1])
+
+        def decrypt_url(encrypted_url):
+            encrypted_url = self._proto_relative_url(
+                encrypted_url.replace('{DATA_MARKERS}', ''), 'https:')
+            key = self._search_regex(
+                r'/key=(.*?)%2Cend=', encrypted_url, 'key', default=None)
+            if not key:
+                return encrypted_url
+            return encrypted_url.replace(key, decrypt_key(key))
+
         formats = []
         for format_id, video_url in video.items():
             if not video_url:
@@ -52,20 +108,18 @@ class BeegIE(InfoExtractor):
             if not height:
                 continue
             formats.append({
-                'url': self._proto_relative_url(
-                    video_url.replace('{DATA_MARKERS}', 'data=pc_XX__%s_0' % beeg_version), 'https:'),
+                'url': decrypt_url(video_url),
                 'format_id': format_id,
                 'height': int(height),
             })
         self._sort_formats(formats)
 
         title = video['title']
-        video_id = compat_str(video.get('id') or video_id)
+        video_id = video.get('id') or video_id
         display_id = video.get('code')
         description = video.get('desc')
-        series = video.get('ps_name')
 
-        timestamp = unified_timestamp(video.get('date'))
+        timestamp = parse_iso8601(video.get('date'), ' ')
         duration = int_or_none(video.get('duration'))
 
         tags = [tag.strip() for tag in video['tags'].split(',')] if video.get('tags') else None
@@ -75,7 +129,6 @@ class BeegIE(InfoExtractor):
             'display_id': display_id,
             'title': title,
             'description': description,
-            'series': series,
             'timestamp': timestamp,
             'duration': duration,
             'tags': tags,

youtube_dl/extractor/bellmedia.py

@@ -12,7 +12,7 @@ class BellMediaIE(InfoExtractor):
             (?:
                 ctv|
                 tsn|
-                bnn(?:bloomberg)?|
+                bnn|
                 thecomedynetwork|
                 discovery|
                 discoveryvelocity|
@@ -27,16 +27,17 @@ class BellMediaIE(InfoExtractor):
             much\.com
         )/.*?(?:\bvid(?:eoid)?=|-vid|~|%7E|/(?:episode)?)(?P<id>[0-9]{6,})'''
     _TESTS = [{
-        'url': 'https://www.bnnbloomberg.ca/video/david-cockfield-s-top-picks~1403070',
-        'md5': '36d3ef559cfe8af8efe15922cd3ce950',
+        'url': 'http://www.ctv.ca/video/player?vid=706966',
+        'md5': 'ff2ebbeae0aa2dcc32a830c3fd69b7b0',
         'info_dict': {
-            'id': '1403070',
-            'ext': 'flv',
-            'title': 'David Cockfield\'s Top Picks',
-            'description': 'md5:810f7f8c6a83ad5b48677c3f8e5bb2c3',
-            'upload_date': '20180525',
-            'timestamp': 1527288600,
+            'id': '706966',
+            'ext': 'mp4',
+            'title': 'Larry Day and Richard Jutras on the TIFF red carpet of \'Stonewall\'',
+            'description': 'etalk catches up with Larry Day and Richard Jutras on the TIFF red carpet of "Stonewall”.',
+            'upload_date': '20150919',
+            'timestamp': 1442624700,
         },
+        'expected_warnings': ['HTTP Error 404'],
     }, {
         'url': 'http://www.thecomedynetwork.ca/video/player?vid=923582',
         'only_matching': True,
@@ -69,7 +70,6 @@ class BellMediaIE(InfoExtractor):
         'investigationdiscovery': 'invdisc',
         'animalplanet': 'aniplan',
         'etalk': 'ctv',
-        'bnnbloomberg': 'bnn',
     }
 
     def _real_extract(self, url):

youtube_dl/extractor/bigflix.py

@@ -1,13 +1,11 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import base64
 import re
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_b64decode,
-    compat_urllib_parse_unquote,
-)
+from ..compat import compat_urllib_parse_unquote
 
 
 class BigflixIE(InfoExtractor):
@@ -41,8 +39,8 @@ class BigflixIE(InfoExtractor):
             webpage, 'title')
 
         def decode_url(quoted_b64_url):
-            return compat_b64decode(compat_urllib_parse_unquote(
-                quoted_b64_url)).decode('utf-8')
+            return base64.b64decode(compat_urllib_parse_unquote(
+                quoted_b64_url).encode('ascii')).decode('utf-8')
 
         formats = []
         for height, encoded_url in re.findall(

youtube_dl/extractor/bilibili.py

@@ -27,14 +27,14 @@ class BiliBiliIE(InfoExtractor):
 
     _TESTS = [{
         'url': 'http://www.bilibili.tv/video/av1074402/',
-        'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
+        'md5': '9fa226fe2b8a9a4d5a69b4c6a183417e',
         'info_dict': {
             'id': '1074402',
-            'ext': 'flv',
+            'ext': 'mp4',
             'title': '【金坷垃】金泡沫',
             'description': 'md5:ce18c2a2d2193f0df2917d270f2e5923',
-            'duration': 308.067,
-            'timestamp': 1398012678,
+            'duration': 308.315,
+            'timestamp': 1398012660,
             'upload_date': '20140420',
             'thumbnail': r're:^https?://.+\.jpg',
             'uploader': '菊子桑',
@@ -59,42 +59,21 @@ class BiliBiliIE(InfoExtractor):
         'url': 'http://www.bilibili.com/video/av8903802/',
         'info_dict': {
             'id': '8903802',
+            'ext': 'mp4',
             'title': '阿滴英文｜英文歌分享#6 "Closer',
             'description': '滴妹今天唱Closer給你聽! 有史以来，被推最多次也是最久的歌曲，其实歌词跟我原本想像差蛮多的，不过还是好听！ 微博@阿滴英文',
+            'uploader': '阿滴英文',
+            'uploader_id': '65880958',
+            'timestamp': 1488382620,
+            'upload_date': '20170301',
+        },
+        'params': {
+            'skip_download': True,  # Test metadata only
         },
-        'playlist': [{
-            'info_dict': {
-                'id': '8903802_part1',
-                'ext': 'flv',
-                'title': '阿滴英文｜英文歌分享#6 "Closer',
-                'description': 'md5:3b1b9e25b78da4ef87e9b548b88ee76a',
-                'uploader': '阿滴英文',
-                'uploader_id': '65880958',
-                'timestamp': 1488382634,
-                'upload_date': '20170301',
-            },
-            'params': {
-                'skip_download': True,  # Test metadata only
-            },
-        }, {
-            'info_dict': {
-                'id': '8903802_part2',
-                'ext': 'flv',
-                'title': '阿滴英文｜英文歌分享#6 "Closer',
-                'description': 'md5:3b1b9e25b78da4ef87e9b548b88ee76a',
-                'uploader': '阿滴英文',
-                'uploader_id': '65880958',
-                'timestamp': 1488382634,
-                'upload_date': '20170301',
-            },
-            'params': {
-                'skip_download': True,  # Test metadata only
-            },
-        }]
     }]
 
-    _APP_KEY = 'iVGUTjsxvpLeuDCf'
-    _BILIBILI_KEY = 'aHRmhWMLkdeMuILqORnYZocwMBpMEOdt'
+    _APP_KEY = '84956560bc028eb7'
+    _BILIBILI_KEY = '94aba54af9065f71de72f5508f1cd42e'
 
     def _report_error(self, result):
         if 'message' in result:
@@ -113,12 +92,8 @@ class BiliBiliIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
 
         if 'anime/' not in url:
-            cid = self._search_regex(
-                r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
-                default=None
-            ) or compat_parse_qs(self._search_regex(
+            cid = compat_parse_qs(self._search_regex(
                 [r'EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
-                 r'EmbedPlayer\([^)]+,\s*\\"([^"]+)\\"\)',
                  r'<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
                 webpage, 'player parameters'))['cid'][0]
         else:
@@ -127,7 +102,6 @@ class BiliBiliIE(InfoExtractor):
                     video_id, anime_id, compat_urlparse.urljoin(url, '//bangumi.bilibili.com/anime/%s' % anime_id)))
             headers = {
                 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
-                'Referer': url
             }
             headers.update(self.geo_verification_headers())
 
@@ -139,66 +113,48 @@ class BiliBiliIE(InfoExtractor):
                 self._report_error(js)
             cid = js['result']['cid']
 
-        headers = {
-            'Referer': url
-        }
-        headers.update(self.geo_verification_headers())
+        payload = 'appkey=%s&cid=%s&otype=json&quality=2&type=mp4' % (self._APP_KEY, cid)
+        sign = hashlib.md5((payload + self._BILIBILI_KEY).encode('utf-8')).hexdigest()
+
+        video_info = self._download_json(
+            'http://interface.bilibili.com/playurl?%s&sign=%s' % (payload, sign),
+            video_id, note='Downloading video info page',
+            headers=self.geo_verification_headers())
+
+        if 'durl' not in video_info:
+            self._report_error(video_info)
 
         entries = []
 
-        RENDITIONS = ('qn=80&quality=80&type=', 'quality=2&type=mp4')
-        for num, rendition in enumerate(RENDITIONS, start=1):
-            payload = 'appkey=%s&cid=%s&otype=json&%s' % (self._APP_KEY, cid, rendition)
-            sign = hashlib.md5((payload + self._BILIBILI_KEY).encode('utf-8')).hexdigest()
-
-            video_info = self._download_json(
-                'http://interface.bilibili.com/v2/playurl?%s&sign=%s' % (payload, sign),
-                video_id, note='Downloading video info page',
-                headers=headers, fatal=num == len(RENDITIONS))
-
-            if not video_info:
-                continue
-
-            if 'durl' not in video_info:
-                if num < len(RENDITIONS):
-                    continue
-                self._report_error(video_info)
-
-            for idx, durl in enumerate(video_info['durl']):
-                formats = [{
-                    'url': durl['url'],
-                    'filesize': int_or_none(durl['size']),
-                }]
-                for backup_url in durl.get('backup_url', []):
-                    formats.append({
-                        'url': backup_url,
-                        # backup URLs have lower priorities
-                        'preference': -2 if 'hd.mp4' in backup_url else -3,
-                    })
-
-                for a_format in formats:
-                    a_format.setdefault('http_headers', {}).update({
-                        'Referer': url,
-                    })
-
-                self._sort_formats(formats)
-
-                entries.append({
-                    'id': '%s_part%s' % (video_id, idx),
-                    'duration': float_or_none(durl.get('length'), 1000),
-                    'formats': formats,
+        for idx, durl in enumerate(video_info['durl']):
+            formats = [{
+                'url': durl['url'],
+                'filesize': int_or_none(durl['size']),
+            }]
+            for backup_url in durl.get('backup_url', []):
+                formats.append({
+                    'url': backup_url,
+                    # backup URLs have lower priorities
+                    'preference': -2 if 'hd.mp4' in backup_url else -3,
                 })
-            break
 
-        title = self._html_search_regex(
-            ('<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
-             '(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
-            group='title')
+            for a_format in formats:
+                a_format.setdefault('http_headers', {}).update({
+                    'Referer': url,
+                })
+
+            self._sort_formats(formats)
+
+            entries.append({
+                'id': '%s_part%s' % (video_id, idx),
+                'duration': float_or_none(durl.get('length'), 1000),
+                'formats': formats,
+            })
+
+        title = self._html_search_regex('<h1[^>]*>([^<]+)</h1>', webpage, 'title')
         description = self._html_search_meta('description', webpage)
         timestamp = unified_timestamp(self._html_search_regex(
-            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time',
-            default=None) or self._html_search_meta(
-            'uploadDate', webpage, 'timestamp', default=None))
+            r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', default=None))
         thumbnail = self._html_search_meta(['og:image', 'thumbnailUrl'], webpage)
 
         # TODO 'view_count' requires deobfuscating Javascript
@@ -212,16 +168,13 @@ class BiliBiliIE(InfoExtractor):
         }
 
         uploader_mobj = re.search(
-            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]*>(?P<name>[^<]+)',
+            r'<a[^>]+href="(?:https?:)?//space\.bilibili\.com/(?P<id>\d+)"[^>]+title="(?P<name>[^"]+)"',
             webpage)
         if uploader_mobj:
             info.update({
                 'uploader': uploader_mobj.group('name'),
                 'uploader_id': uploader_mobj.group('id'),
             })
-        if not info.get('uploader'):
-            info['uploader'] = self._html_search_meta(
-                'author', webpage, 'uploader', default=None)
 
         for entry in entries:
             entry.update(info)

youtube_dl/extractor/bitchute.py

@@ -1,129 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import itertools
-import re
-
-from .common import InfoExtractor
-from ..utils import (
-    orderedSet,
-    urlencode_postdata,
-)
-
-
-class BitChuteIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?bitchute\.com/(?:video|embed|torrent/[^/]+)/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://www.bitchute.com/video/szoMrox2JEI/',
-        'md5': '66c4a70e6bfc40dcb6be3eb1d74939eb',
-        'info_dict': {
-            'id': 'szoMrox2JEI',
-            'ext': 'mp4',
-            'title': 'Fuck bitches get money',
-            'description': 'md5:3f21f6fb5b1d17c3dee9cf6b5fe60b3a',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'uploader': 'Victoria X Rave',
-        },
-    }, {
-        'url': 'https://www.bitchute.com/embed/lbb5G1hjPhw/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.bitchute.com/torrent/Zee5BE49045h/szoMrox2JEI.webtorrent',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        webpage = self._download_webpage(
-            'https://www.bitchute.com/video/%s' % video_id, video_id, headers={
-                'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.57 Safari/537.36',
-            })
-
-        title = self._html_search_regex(
-            (r'<[^>]+\bid=["\']video-title[^>]+>([^<]+)', r'<title>([^<]+)'),
-            webpage, 'title', default=None) or self._html_search_meta(
-            'description', webpage, 'title',
-            default=None) or self._og_search_description(webpage)
-
-        format_urls = []
-        for mobj in re.finditer(
-                r'addWebSeed\s*\(\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage):
-            format_urls.append(mobj.group('url'))
-        format_urls.extend(re.findall(r'as=(https?://[^&"\']+)', webpage))
-
-        formats = [
-            {'url': format_url}
-            for format_url in orderedSet(format_urls)]
-        self._check_formats(formats, video_id)
-        self._sort_formats(formats)
-
-        description = self._html_search_regex(
-            r'(?s)<div\b[^>]+\bclass=["\']full hidden[^>]+>(.+?)</div>',
-            webpage, 'description', fatal=False)
-        thumbnail = self._og_search_thumbnail(
-            webpage, default=None) or self._html_search_meta(
-            'twitter:image:src', webpage, 'thumbnail')
-        uploader = self._html_search_regex(
-            r'(?s)<p\b[^>]+\bclass=["\']video-author[^>]+>(.+?)</p>', webpage,
-            'uploader', fatal=False)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': description,
-            'thumbnail': thumbnail,
-            'uploader': uploader,
-            'formats': formats,
-        }
-
-
-class BitChuteChannelIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?bitchute\.com/channel/(?P<id>[^/?#&]+)'
-    _TEST = {
-        'url': 'https://www.bitchute.com/channel/victoriaxrave/',
-        'playlist_mincount': 185,
-        'info_dict': {
-            'id': 'victoriaxrave',
-        },
-    }
-
-    _TOKEN = 'zyG6tQcGPE5swyAEFLqKUwMuMMuF6IO2DZ6ZDQjGfsL0e4dcTLwqkTTul05Jdve7'
-
-    def _entries(self, channel_id):
-        channel_url = 'https://www.bitchute.com/channel/%s/' % channel_id
-        offset = 0
-        for page_num in itertools.count(1):
-            data = self._download_json(
-                '%sextend/' % channel_url, channel_id,
-                'Downloading channel page %d' % page_num,
-                data=urlencode_postdata({
-                    'csrfmiddlewaretoken': self._TOKEN,
-                    'name': '',
-                    'offset': offset,
-                }), headers={
-                    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
-                    'Referer': channel_url,
-                    'X-Requested-With': 'XMLHttpRequest',
-                    'Cookie': 'csrftoken=%s' % self._TOKEN,
-                })
-            if data.get('success') is False:
-                break
-            html = data.get('html')
-            if not html:
-                break
-            video_ids = re.findall(
-                r'class=["\']channel-videos-image-container[^>]+>\s*<a\b[^>]+\bhref=["\']/video/([^"\'/]+)',
-                html)
-            if not video_ids:
-                break
-            offset += len(video_ids)
-            for video_id in video_ids:
-                yield self.url_result(
-                    'https://www.bitchute.com/video/%s' % video_id,
-                    ie=BitChuteIE.ie_key(), video_id=video_id)
-
-    def _real_extract(self, url):
-        channel_id = self._match_id(url)
-        return self.playlist_result(
-            self._entries(channel_id), playlist_id=channel_id)

youtube_dl/extractor/breakcom.py

@@ -3,15 +3,15 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from .youtube import YoutubeIE
+from ..compat import compat_str
 from ..utils import (
     int_or_none,
-    url_or_none,
+    parse_age_limit,
 )
 
 
 class BreakIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?break\.com/video/(?P<display_id>[^/]+?)(?:-(?P<id>\d+))?(?:[/?#&]|$)'
+    _VALID_URL = r'https?://(?:www\.)?(?P<site>break|screenjunkies)\.com/video/(?P<display_id>[^/]+?)(?:-(?P<id>\d+))?(?:[/?#&]|$)'
     _TESTS = [{
         'url': 'http://www.break.com/video/when-girls-act-like-guys-2468056',
         'info_dict': {
@@ -19,73 +19,125 @@ class BreakIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'When Girls Act Like D-Bags',
             'age_limit': 13,
+        }
+    }, {
+        'url': 'http://www.screenjunkies.com/video/best-quentin-tarantino-movie-2841915',
+        'md5': '5c2b686bec3d43de42bde9ec047536b0',
+        'info_dict': {
+            'id': '2841915',
+            'display_id': 'best-quentin-tarantino-movie',
+            'ext': 'mp4',
+            'title': 'Best Quentin Tarantino Movie',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 3671,
+            'age_limit': 13,
+            'tags': list,
         },
     }, {
-        # youtube embed
-        'url': 'http://www.break.com/video/someone-forgot-boat-brakes-work',
+        'url': 'http://www.screenjunkies.com/video/honest-trailers-the-dark-knight',
         'info_dict': {
-            'id': 'RrrDLdeL2HQ',
+            'id': '2348808',
+            'display_id': 'honest-trailers-the-dark-knight',
             'ext': 'mp4',
-            'title': 'Whale Watching Boat Crashing Into San Diego Dock',
-            'description': 'md5:afc1b2772f0a8468be51dd80eb021069',
-            'upload_date': '20160331',
-            'uploader': 'Steve Holden',
-            'uploader_id': 'sdholden07',
+            'title': 'Honest Trailers - The Dark Knight',
+            'thumbnail': r're:^https?://.*\.(?:jpg|png)',
+            'age_limit': 10,
+            'tags': list,
+        },
+    }, {
+        # requires subscription but worked around
+        'url': 'http://www.screenjunkies.com/video/knocking-dead-ep-1-the-show-so-far-3003285',
+        'info_dict': {
+            'id': '3003285',
+            'display_id': 'knocking-dead-ep-1-the-show-so-far',
+            'ext': 'mp4',
+            'title': 'State of The Dead Recap: Knocking Dead Pilot',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 3307,
+            'age_limit': 13,
+            'tags': list,
         },
-        'params': {
-            'skip_download': True,
-        }
     }, {
         'url': 'http://www.break.com/video/ugc/baby-flex-2773063',
         'only_matching': True,
     }]
 
+    _DEFAULT_BITRATES = (48, 150, 320, 496, 864, 2240, 3264)
+
     def _real_extract(self, url):
-        display_id, video_id = re.match(self._VALID_URL, url).groups()
+        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
 
-        webpage = self._download_webpage(url, display_id)
+        if not video_id:
+            webpage = self._download_webpage(url, display_id)
+            video_id = self._search_regex(
+                (r'src=["\']/embed/(\d+)', r'data-video-content-id=["\'](\d+)'),
+                webpage, 'video id')
 
-        youtube_url = YoutubeIE._extract_url(webpage)
-        if youtube_url:
-            return self.url_result(youtube_url, ie=YoutubeIE.ie_key())
-
-        content = self._parse_json(
+        webpage = self._download_webpage(
+            'http://www.%s.com/embed/%s' % (site, video_id),
+            display_id, 'Downloading video embed page')
+        embed_vars = self._parse_json(
             self._search_regex(
-                r'(?s)content["\']\s*:\s*(\[.+?\])\s*[,\n]', webpage,
-                'content'),
+                r'(?s)embedVars\s*=\s*({.+?})\s*</script>', webpage, 'embed vars'),
             display_id)
 
+        youtube_id = embed_vars.get('youtubeId')
+        if youtube_id:
+            return self.url_result(youtube_id, 'Youtube')
+
+        title = embed_vars['contentName']
+
         formats = []
-        for video in content:
-            video_url = url_or_none(video.get('url'))
-            if not video_url:
+        bitrates = []
+        for f in embed_vars.get('media', []):
+            if not f.get('uri') or f.get('mediaPurpose') != 'play':
                 continue
-            bitrate = int_or_none(self._search_regex(
-                r'(\d+)_kbps', video_url, 'tbr', default=None))
+            bitrate = int_or_none(f.get('bitRate'))
+            if bitrate:
+                bitrates.append(bitrate)
             formats.append({
-                'url': video_url,
+                'url': f['uri'],
                 'format_id': 'http-%d' % bitrate if bitrate else 'http',
+                'width': int_or_none(f.get('width')),
+                'height': int_or_none(f.get('height')),
                 'tbr': bitrate,
+                'format': 'mp4',
             })
+
+        if not bitrates:
+            # When subscriptionLevel > 0, i.e. plus subscription is required
+            # media list will be empty. However, hds and hls uris are still
+            # available. We can grab them assuming bitrates to be default.
+            bitrates = self._DEFAULT_BITRATES
+
+        auth_token = embed_vars.get('AuthToken')
+
+        def construct_manifest_url(base_url, ext):
+            pieces = [base_url]
+            pieces.extend([compat_str(b) for b in bitrates])
+            pieces.append('_kbps.mp4.%s?%s' % (ext, auth_token))
+            return ','.join(pieces)
+
+        if bitrates and auth_token:
+            hds_url = embed_vars.get('hdsUri')
+            if hds_url:
+                formats.extend(self._extract_f4m_formats(
+                    construct_manifest_url(hds_url, 'f4m'),
+                    display_id, f4m_id='hds', fatal=False))
+            hls_url = embed_vars.get('hlsUri')
+            if hls_url:
+                formats.extend(self._extract_m3u8_formats(
+                    construct_manifest_url(hls_url, 'm3u8'),
+                    display_id, 'mp4', entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
         self._sort_formats(formats)
 
-        title = self._search_regex(
-            (r'title["\']\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1',
-             r'<h1[^>]*>(?P<value>[^<]+)'), webpage, 'title', group='value')
-
-        def get(key, name):
-            return int_or_none(self._search_regex(
-                r'%s["\']\s*:\s*["\'](\d+)' % key, webpage, name,
-                default=None))
-
-        age_limit = get('ratings', 'age limit')
-        video_id = video_id or get('pid', 'video id') or display_id
-
         return {
             'id': video_id,
             'display_id': display_id,
             'title': title,
-            'thumbnail': self._og_search_thumbnail(webpage),
-            'age_limit': age_limit,
+            'thumbnail': embed_vars.get('thumbUri'),
+            'duration': int_or_none(embed_vars.get('videoLengthInSeconds')) or None,
+            'age_limit': parse_age_limit(embed_vars.get('audienceRating')),
+            'tags': embed_vars.get('tags', '').split(','),
             'formats': formats,
         }

youtube_dl/extractor/brightcove.py

@@ -1,10 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import base64
-import json
 import re
-import struct
+import json
 
 from .common import InfoExtractor
 from .adobepass import AdobePassIE
@@ -312,10 +310,6 @@ class BrightcoveLegacyIE(InfoExtractor):
                 'Cannot find playerKey= variable. Did you forget quotes in a shell invocation?',
                 expected=True)
 
-    def _brightcove_new_url_result(self, publisher_id, video_id):
-        brightcove_new_url = 'http://players.brightcove.net/%s/default_default/index.html?videoId=%s' % (publisher_id, video_id)
-        return self.url_result(brightcove_new_url, BrightcoveNewIE.ie_key(), video_id)
-
     def _get_video_info(self, video_id, query, referer=None):
         headers = {}
         linkBase = query.get('linkBaseURL')
@@ -329,28 +323,6 @@ class BrightcoveLegacyIE(InfoExtractor):
             r"<h1>We're sorry.</h1>([\s\n]*<p>.*?</p>)+", webpage,
             'error message', default=None)
         if error_msg is not None:
-            publisher_id = query.get('publisherId')
-            if publisher_id and publisher_id[0].isdigit():
-                publisher_id = publisher_id[0]
-            if not publisher_id:
-                player_key = query.get('playerKey')
-                if player_key and ',' in player_key[0]:
-                    player_key = player_key[0]
-                else:
-                    player_id = query.get('playerID')
-                    if player_id and player_id[0].isdigit():
-                        player_page = self._download_webpage(
-                            'http://link.brightcove.com/services/player/bcpid' + player_id[0],
-                            video_id, headers=headers, fatal=False)
-                        if player_page:
-                            player_key = self._search_regex(
-                                r'<param\s+name="playerKey"\s+value="([\w~,-]+)"',
-                                player_page, 'player key', fatal=False)
-                if player_key:
-                    enc_pub_id = player_key.split(',')[1].replace('~', '=')
-                    publisher_id = struct.unpack('>Q', base64.urlsafe_b64decode(enc_pub_id))[0]
-                if publisher_id:
-                    return self._brightcove_new_url_result(publisher_id, video_id)
             raise ExtractorError(
                 'brightcove said: %s' % error_msg, expected=True)
 
@@ -472,12 +444,8 @@ class BrightcoveLegacyIE(InfoExtractor):
                 else:
                     return ad_info
 
-        if not info.get('url') and not info.get('formats'):
-            uploader_id = info.get('uploader_id')
-            if uploader_id:
-                info.update(self._brightcove_new_url_result(uploader_id, video_id))
-            else:
-                raise ExtractorError('Unable to extract video url for %s' % video_id)
+        if 'url' not in info and not info.get('formats'):
+            raise ExtractorError('Unable to extract video url for %s' % video_id)
         return info
 
 
@@ -596,7 +564,7 @@ class BrightcoveNewIE(AdobePassIE):
 
         return entries
 
-    def _parse_brightcove_metadata(self, json_data, video_id, headers={}):
+    def _parse_brightcove_metadata(self, json_data, video_id):
         title = json_data['name'].strip()
 
         formats = []
@@ -604,8 +572,7 @@ class BrightcoveNewIE(AdobePassIE):
             container = source.get('container')
             ext = mimetype2ext(source.get('type'))
             src = source.get('src')
-            # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
-            if ext == 'ism' or container == 'WVM' or source.get('key_systems'):
+            if ext == 'ism' or container == 'WVM':
                 continue
             elif ext == 'm3u8' or container == 'M2TS':
                 if not src:
@@ -662,14 +629,6 @@ class BrightcoveNewIE(AdobePassIE):
                         'format_id': build_format_id('rtmp'),
                     })
                 formats.append(f)
-        if not formats:
-            # for sonyliv.com DRM protected videos
-            s3_source_url = json_data.get('custom_fields', {}).get('s3sourceurl')
-            if s3_source_url:
-                formats.append({
-                    'url': s3_source_url,
-                    'format_id': 'source',
-                })
 
         errors = json_data.get('errors')
         if not formats and errors:
@@ -679,9 +638,6 @@ class BrightcoveNewIE(AdobePassIE):
 
         self._sort_formats(formats)
 
-        for f in formats:
-            f.setdefault('http_headers', {}).update(headers)
-
         subtitles = {}
         for text_track in json_data.get('text_tracks', []):
             if text_track.get('src'):
@@ -710,10 +666,7 @@ class BrightcoveNewIE(AdobePassIE):
 
     def _real_extract(self, url):
         url, smuggled_data = unsmuggle_url(url, {})
-        self._initialize_geo_bypass({
-            'countries': smuggled_data.get('geo_countries'),
-            'ip_blocks': smuggled_data.get('geo_ip_blocks'),
-        })
+        self._initialize_geo_bypass(smuggled_data.get('geo_countries'))
 
         account_id, player_id, embed, video_id = re.match(self._VALID_URL, url).groups()
 
@@ -737,17 +690,10 @@ class BrightcoveNewIE(AdobePassIE):
                 webpage, 'policy key', group='pk')
 
         api_url = 'https://edge.api.brightcove.com/playback/v1/accounts/%s/videos/%s' % (account_id, video_id)
-        headers = {
-            'Accept': 'application/json;pk=%s' % policy_key,
-        }
-        referrer = smuggled_data.get('referrer')
-        if referrer:
-            headers.update({
-                'Referer': referrer,
-                'Origin': re.search(r'https?://[^/]+', referrer).group(0),
-            })
         try:
-            json_data = self._download_json(api_url, video_id, headers=headers)
+            json_data = self._download_json(api_url, video_id, headers={
+                'Accept': 'application/json;pk=%s' % policy_key
+            })
         except ExtractorError as e:
             if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                 json_data = self._parse_json(e.cause.read().decode(), video_id)[0]
@@ -771,5 +717,4 @@ class BrightcoveNewIE(AdobePassIE):
                     'tveToken': tve_token,
                 })
 
-        return self._parse_brightcove_metadata(
-            json_data, video_id, headers=headers)
+        return self._parse_brightcove_metadata(json_data, video_id)

youtube_dl/extractor/businessinsider.py

@@ -1,42 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from .jwplatform import JWPlatformIE
-
-
-class BusinessInsiderIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:[^/]+\.)?businessinsider\.(?:com|nl)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'http://uk.businessinsider.com/how-much-radiation-youre-exposed-to-in-everyday-life-2016-6',
-        'md5': 'ca237a53a8eb20b6dc5bd60564d4ab3e',
-        'info_dict': {
-            'id': 'hZRllCfw',
-            'ext': 'mp4',
-            'title': "Here's how much radiation you're exposed to in everyday life",
-            'description': 'md5:9a0d6e2c279948aadaa5e84d6d9b99bd',
-            'upload_date': '20170709',
-            'timestamp': 1499606400,
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'https://www.businessinsider.nl/5-scientifically-proven-things-make-you-less-attractive-2017-7/',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.businessinsider.com/excel-index-match-vlookup-video-how-to-2015-2?IR=T',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        jwplatform_id = self._search_regex(
-            (r'data-media-id=["\']([a-zA-Z0-9]{8})',
-             r'id=["\']jwplayer_([a-zA-Z0-9]{8})',
-             r'id["\']?\s*:\s*["\']?([a-zA-Z0-9]{8})'),
-            webpage, 'jwplatform id')
-        return self.url_result(
-            'jwplatform:%s' % jwplatform_id, ie=JWPlatformIE.ie_key(),
-            video_id=video_id)

youtube_dl/extractor/cammodels.py

@@ -1,98 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    ExtractorError,
-    int_or_none,
-    url_or_none,
-)
-
-
-class CamModelsIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?cammodels\.com/cam/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://www.cammodels.com/cam/AutumnKnight/',
-        'only_matching': True,
-        'age_limit': 18
-    }]
-
-    def _real_extract(self, url):
-        user_id = self._match_id(url)
-
-        webpage = self._download_webpage(
-            url, user_id, headers=self.geo_verification_headers())
-
-        manifest_root = self._html_search_regex(
-            r'manifestUrlRoot=([^&\']+)', webpage, 'manifest', default=None)
-
-        if not manifest_root:
-            ERRORS = (
-                ("I'm offline, but let's stay connected", 'This user is currently offline'),
-                ('in a private show', 'This user is in a private show'),
-                ('is currently performing LIVE', 'This model is currently performing live'),
-            )
-            for pattern, message in ERRORS:
-                if pattern in webpage:
-                    error = message
-                    expected = True
-                    break
-            else:
-                error = 'Unable to find manifest URL root'
-                expected = False
-            raise ExtractorError(error, expected=expected)
-
-        manifest = self._download_json(
-            '%s%s.json' % (manifest_root, user_id), user_id)
-
-        formats = []
-        for format_id, format_dict in manifest['formats'].items():
-            if not isinstance(format_dict, dict):
-                continue
-            encodings = format_dict.get('encodings')
-            if not isinstance(encodings, list):
-                continue
-            vcodec = format_dict.get('videoCodec')
-            acodec = format_dict.get('audioCodec')
-            for media in encodings:
-                if not isinstance(media, dict):
-                    continue
-                media_url = url_or_none(media.get('location'))
-                if not media_url:
-                    continue
-
-                format_id_list = [format_id]
-                height = int_or_none(media.get('videoHeight'))
-                if height is not None:
-                    format_id_list.append('%dp' % height)
-                f = {
-                    'url': media_url,
-                    'format_id': '-'.join(format_id_list),
-                    'width': int_or_none(media.get('videoWidth')),
-                    'height': height,
-                    'vbr': int_or_none(media.get('videoKbps')),
-                    'abr': int_or_none(media.get('audioKbps')),
-                    'fps': int_or_none(media.get('fps')),
-                    'vcodec': vcodec,
-                    'acodec': acodec,
-                }
-                if 'rtmp' in format_id:
-                    f['ext'] = 'flv'
-                elif 'hls' in format_id:
-                    f.update({
-                        'ext': 'mp4',
-                        # hls skips fragments, preferring rtmp
-                        'preference': -1,
-                    })
-                else:
-                    continue
-                formats.append(f)
-        self._sort_formats(formats)
-
-        return {
-            'id': user_id,
-            'title': self._live_title(user_id),
-            'is_live': True,
-            'formats': formats,
-            'age_limit': 18
-        }

youtube_dl/extractor/camtube.py

@@ -1,71 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    int_or_none,
-    unified_timestamp,
-)
-
-
-class CamTubeIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:www|api)\.)?camtube\.co/recordings?/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://camtube.co/recording/minafay-030618-1136-chaturbate-female',
-        'info_dict': {
-            'id': '42ad3956-dd5b-445a-8313-803ea6079fac',
-            'display_id': 'minafay-030618-1136-chaturbate-female',
-            'ext': 'mp4',
-            'title': 'minafay-030618-1136-chaturbate-female',
-            'duration': 1274,
-            'timestamp': 1528018608,
-            'upload_date': '20180603',
-            'age_limit': 18
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }]
-
-    _API_BASE = 'https://api.camtube.co'
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-
-        token = self._download_json(
-            '%s/rpc/session/new' % self._API_BASE, display_id,
-            'Downloading session token')['token']
-
-        self._set_cookie('api.camtube.co', 'session', token)
-
-        video = self._download_json(
-            '%s/recordings/%s' % (self._API_BASE, display_id), display_id,
-            headers={'Referer': url})
-
-        video_id = video['uuid']
-        timestamp = unified_timestamp(video.get('createdAt'))
-        duration = int_or_none(video.get('duration'))
-        view_count = int_or_none(video.get('viewCount'))
-        like_count = int_or_none(video.get('likeCount'))
-        creator = video.get('stageName')
-
-        formats = [{
-            'url': '%s/recordings/%s/manifest.m3u8'
-                   % (self._API_BASE, video_id),
-            'format_id': 'hls',
-            'ext': 'mp4',
-            'protocol': 'm3u8_native',
-        }]
-
-        return {
-            'id': video_id,
-            'display_id': display_id,
-            'title': display_id,
-            'timestamp': timestamp,
-            'duration': duration,
-            'view_count': view_count,
-            'like_count': like_count,
-            'creator': creator,
-            'formats': formats,
-            'age_limit': 18
-        }

youtube_dl/extractor/camwithher.py

@@ -25,7 +25,6 @@ class CamWithHerIE(InfoExtractor):
             'comment_count': int,
             'uploader': 'MileenaK',
             'upload_date': '20160322',
-            'age_limit': 18,
         },
         'params': {
             'skip_download': True,
@@ -85,5 +84,4 @@ class CamWithHerIE(InfoExtractor):
             'comment_count': comment_count,
             'uploader': uploader,
             'upload_date': upload_date,
-            'age_limit': 18
         }

youtube_dl/extractor/canalc2.py

@@ -31,10 +31,6 @@ class Canalc2IE(InfoExtractor):
         webpage = self._download_webpage(
             'http://www.canalc2.tv/video/%s' % video_id, video_id)
 
-        title = self._html_search_regex(
-            r'(?s)class="[^"]*col_description[^"]*">.*?<h3>(.+?)</h3>',
-            webpage, 'title')
-
         formats = []
         for _, video_url in re.findall(r'file\s*=\s*(["\'])(.+?)\1', webpage):
             if video_url.startswith('rtmp://'):
@@ -53,21 +49,17 @@ class Canalc2IE(InfoExtractor):
                     'url': video_url,
                     'format_id': 'http',
                 })
+        self._sort_formats(formats)
 
-        if formats:
-            info = {
-                'formats': formats,
-            }
-        else:
-            info = self._parse_html5_media_entries(url, webpage, url)[0]
+        title = self._html_search_regex(
+            r'(?s)class="[^"]*col_description[^"]*">.*?<h3>(.*?)</h3>', webpage, 'title')
+        duration = parse_duration(self._search_regex(
+            r'id=["\']video_duree["\'][^>]*>([^<]+)',
+            webpage, 'duration', fatal=False))
 
-        self._sort_formats(info['formats'])
-
-        info.update({
+        return {
             'id': video_id,
             'title': title,
-            'duration': parse_duration(self._search_regex(
-                r'id=["\']video_duree["\'][^>]*>([^<]+)',
-                webpage, 'duration', fatal=False)),
-        })
-        return info
+            'duration': duration,
+            'formats': formats,
+        }

youtube_dl/extractor/canalplus.py

@@ -4,36 +4,59 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from ..compat import compat_urllib_parse_urlparse
 from ..utils import (
+    dict_get,
     # ExtractorError,
     # HEADRequest,
     int_or_none,
     qualities,
+    remove_end,
     unified_strdate,
 )
 
 
 class CanalplusIE(InfoExtractor):
-    IE_DESC = 'mycanal.fr and piwiplus.fr'
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>mycanal|piwiplus)\.fr/(?:[^/]+/)*(?P<display_id>[^?/]+)(?:\.html\?.*\bvid=|/p/)(?P<id>\d+)'
+    IE_DESC = 'canalplus.fr, piwiplus.fr and d8.tv'
+    _VALID_URL = r'''(?x)
+                        https?://
+                            (?:
+                                (?:
+                                    (?:(?:www|m)\.)?canalplus\.fr|
+                                    (?:www\.)?piwiplus\.fr|
+                                    (?:www\.)?d8\.tv|
+                                    (?:www\.)?c8\.fr|
+                                    (?:www\.)?d17\.tv|
+                                    (?:(?:football|www)\.)?cstar\.fr|
+                                    (?:www\.)?itele\.fr
+                                )/(?:(?:[^/]+/)*(?P<display_id>[^/?#&]+))?(?:\?.*\bvid=(?P<vid>\d+))?|
+                                player\.canalplus\.fr/#/(?P<id>\d+)
+                            )
+
+                    '''
     _VIDEO_INFO_TEMPLATE = 'http://service.canal-plus.com/video/rest/getVideosLiees/%s/%s?format=json'
     _SITE_ID_MAP = {
-        'mycanal': 'cplus',
+        'canalplus': 'cplus',
         'piwiplus': 'teletoon',
+        'd8': 'd8',
+        'c8': 'd8',
+        'd17': 'd17',
+        'cstar': 'd17',
+        'itele': 'itele',
     }
 
     # Only works for direct mp4 URLs
     _GEO_COUNTRIES = ['FR']
 
     _TESTS = [{
-        'url': 'https://www.mycanal.fr/d17-emissions/lolywood/p/1397061',
+        'url': 'http://www.canalplus.fr/c-emissions/pid1830-c-zapping.html?vid=1192814',
         'info_dict': {
-            'id': '1397061',
-            'display_id': 'lolywood',
+            'id': '1405510',
+            'display_id': 'pid1830-c-zapping',
             'ext': 'mp4',
-            'title': 'Euro 2016 : Je préfère te prévenir - Lolywood - Episode 34',
-            'description': 'md5:7d97039d455cb29cdba0d652a0efaa5e',
-            'upload_date': '20160602',
+            'title': 'Zapping - 02/07/2016',
+            'description': 'Le meilleur de toutes les chaînes, tous les jours',
+            'upload_date': '20160702',
         },
     }, {
         # geo restricted, bypassed
@@ -47,12 +70,64 @@ class CanalplusIE(InfoExtractor):
             'upload_date': '20140724',
         },
         'expected_warnings': ['HTTP Error 403: Forbidden'],
+    }, {
+        # geo restricted, bypassed
+        'url': 'http://www.c8.fr/c8-divertissement/ms-touche-pas-a-mon-poste/pid6318-videos-integrales.html?vid=1443684',
+        'md5': 'bb6f9f343296ab7ebd88c97b660ecf8d',
+        'info_dict': {
+            'id': '1443684',
+            'display_id': 'pid6318-videos-integrales',
+            'ext': 'mp4',
+            'title': 'Guess my iep ! - TPMP - 07/04/2017',
+            'description': 'md5:6f005933f6e06760a9236d9b3b5f17fa',
+            'upload_date': '20170407',
+        },
+        'expected_warnings': ['HTTP Error 403: Forbidden'],
+    }, {
+        'url': 'http://www.itele.fr/chroniques/invite-michael-darmon/rachida-dati-nicolas-sarkozy-est-le-plus-en-phase-avec-les-inquietudes-des-francais-171510',
+        'info_dict': {
+            'id': '1420176',
+            'display_id': 'rachida-dati-nicolas-sarkozy-est-le-plus-en-phase-avec-les-inquietudes-des-francais-171510',
+            'ext': 'mp4',
+            'title': 'L\'invité de Michaël Darmon du 14/10/2016 - ',
+            'description': 'Chaque matin du lundi au vendredi, Michaël Darmon reçoit un invité politique à 8h25.',
+            'upload_date': '20161014',
+        },
+    }, {
+        'url': 'http://football.cstar.fr/cstar-minisite-foot/pid7566-feminines-videos.html?vid=1416769',
+        'info_dict': {
+            'id': '1416769',
+            'display_id': 'pid7566-feminines-videos',
+            'ext': 'mp4',
+            'title': 'France - Albanie : les temps forts de la soirée - 20/09/2016',
+            'description': 'md5:c3f30f2aaac294c1c969b3294de6904e',
+            'upload_date': '20160921',
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }, {
+        'url': 'http://m.canalplus.fr/?vid=1398231',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.d17.tv/emissions/pid8303-lolywood.html?vid=1397061',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
 
-        site_id = self._SITE_ID_MAP[site]
+        site_id = self._SITE_ID_MAP[compat_urllib_parse_urlparse(url).netloc.rsplit('.', 2)[-2]]
+
+        # Beware, some subclasses do not define an id group
+        display_id = remove_end(dict_get(mobj.groupdict(), ('display_id', 'id', 'vid')), '.html')
+
+        webpage = self._download_webpage(url, display_id)
+        video_id = self._search_regex(
+            [r'<canal:player[^>]+?videoId=(["\'])(?P<id>\d+)',
+             r'id=["\']canal_video_player(?P<id>\d+)',
+             r'data-video=["\'](?P<id>\d+)'],
+            webpage, 'video id', default=mobj.group('vid'), group='id')
 
         info_url = self._VIDEO_INFO_TEMPLATE % (site_id, video_id)
         video_data = self._download_json(info_url, video_id, 'Downloading video JSON')
@@ -86,7 +161,7 @@ class CanalplusIE(InfoExtractor):
                     format_url + '?hdcore=2.11.3', video_id, f4m_id=format_id, fatal=False))
             else:
                 formats.append({
-                    # the secret extracted from ya function in http://player.canalplus.fr/common/js/canalPlayer.js
+                    # the secret extracted ya function in http://player.canalplus.fr/common/js/canalPlayer.js
                     'url': format_url + '?secret=pqzerjlsmdkjfoiuerhsdlfknaes',
                     'format_id': format_id,
                     'preference': preference(format_id),

youtube_dl/extractor/canvas.py

@@ -11,7 +11,6 @@ from ..utils import (
     strip_or_none,
     float_or_none,
     int_or_none,
-    merge_dicts,
     parse_iso8601,
 )
 
@@ -247,15 +246,11 @@ class VrtNUIE(GigyaBaseIE):
     def _real_extract(self, url):
         display_id = self._match_id(url)
 
-        webpage, urlh = self._download_webpage_handle(url, display_id)
+        webpage = self._download_webpage(url, display_id)
 
-        info = self._search_json_ld(webpage, display_id, default={})
-
-        # title is optional here since it may be extracted by extractor
-        # that is delegated from here
-        title = strip_or_none(self._html_search_regex(
+        title = self._html_search_regex(
             r'(?ms)<h1 class="content__heading">(.+?)</h1>',
-            webpage, 'title', default=None))
+            webpage, 'title').strip()
 
         description = self._html_search_regex(
             r'(?ms)<div class="content__description">(.+?)</div>',
@@ -281,7 +276,7 @@ class VrtNUIE(GigyaBaseIE):
             webpage, 'release_date', default=None))
 
         # If there's a ? or a # in the URL, remove them and everything after
-        clean_url = urlh.geturl().split('?')[0].split('#')[0].strip('/')
+        clean_url = url.split('?')[0].split('#')[0].strip('/')
         securevideo_url = clean_url + '.mssecurevideo.json'
 
         try:
@@ -300,7 +295,7 @@ class VrtNUIE(GigyaBaseIE):
         # the first one
         video_id = list(video.values())[0].get('videoid')
 
-        return merge_dicts(info, {
+        return {
             '_type': 'url_transparent',
             'url': 'https://mediazone.vrt.be/api/v1/vrtvideo/assets/%s' % video_id,
             'ie_key': CanvasIE.ie_key(),
@@ -312,4 +307,4 @@ class VrtNUIE(GigyaBaseIE):
             'season_number': season_number,
             'episode_number': episode_number,
             'release_date': release_date,
-        })
+        }

youtube_dl/extractor/carambatv.py

@@ -82,12 +82,6 @@ class CarambaTVPageIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
 
         videomore_url = VideomoreIE._extract_url(webpage)
-        if not videomore_url:
-            videomore_id = self._search_regex(
-                r'getVMCode\s*\(\s*["\']?(\d+)', webpage, 'videomore id',
-                default=None)
-            if videomore_id:
-                videomore_url = 'videomore:%s' % videomore_id
         if videomore_url:
             title = self._og_search_title(webpage)
             return {

youtube_dl/extractor/cartoonnetwork.py

@@ -1,19 +1,20 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import re
+
 from .turner import TurnerBaseIE
-from ..utils import int_or_none
 
 
 class CartoonNetworkIE(TurnerBaseIE):
     _VALID_URL = r'https?://(?:www\.)?cartoonnetwork\.com/video/(?:[^/]+/)+(?P<id>[^/?#]+)-(?:clip|episode)\.html'
     _TEST = {
-        'url': 'https://www.cartoonnetwork.com/video/ben-10/how-to-draw-upgrade-episode.html',
+        'url': 'http://www.cartoonnetwork.com/video/teen-titans-go/starfire-the-cat-lady-clip.html',
         'info_dict': {
-            'id': '6e3375097f63874ebccec7ef677c1c3845fa850e',
+            'id': '8a250ab04ed07e6c014ef3f1e2f9016c',
             'ext': 'mp4',
-            'title': 'How to Draw Upgrade',
-            'description': 'md5:2061d83776db7e8be4879684eefe8c0f',
+            'title': 'Starfire the Cat Lady',
+            'description': 'Robin decides to become a cat so that Starfire will finally love him.',
         },
         'params': {
             # m3u8 download
@@ -24,39 +25,18 @@ class CartoonNetworkIE(TurnerBaseIE):
     def _real_extract(self, url):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
-
-        def find_field(global_re, name, content_re=None, value_re='[^"]+', fatal=False):
-            metadata_re = ''
-            if content_re:
-                metadata_re = r'|video_metadata\.content_' + content_re
-            return self._search_regex(
-                r'(?:_cnglobal\.currentVideo\.%s%s)\s*=\s*"(%s)";' % (global_re, metadata_re, value_re),
-                webpage, name, fatal=fatal)
-
-        media_id = find_field('mediaId', 'media id', 'id', '[0-9a-f]{40}', True)
-        title = find_field('episodeTitle', 'title', '(?:episodeName|name)', fatal=True)
-
-        info = self._extract_ngtv_info(
-            media_id, {'networkId': 'cartoonnetwork'}, {
+        id_type, video_id = re.search(r"_cnglobal\.cvp(Video|Title)Id\s*=\s*'([^']+)';", webpage).groups()
+        query = ('id' if id_type == 'Video' else 'titleId') + '=' + video_id
+        return self._extract_cvp_info(
+            'http://www.cartoonnetwork.com/video-seo-svc/episodeservices/getCvpPlaylist?networkName=CN2&' + query, video_id, {
+                'secure': {
+                    'media_src': 'http://androidhls-secure.cdn.turner.com/toon/big',
+                    'tokenizer_src': 'https://token.vgtf.net/token/token_mobile',
+                },
+            }, {
                 'url': url,
                 'site_name': 'CartoonNetwork',
-                'auth_required': find_field('authType', 'auth type') != 'unauth',
+                'auth_required': self._search_regex(
+                    r'_cnglobal\.cvpFullOrPreviewAuth\s*=\s*(true|false);',
+                    webpage, 'auth required', default='false') == 'true',
             })
-
-        series = find_field(
-            'propertyName', 'series', 'showName') or self._html_search_meta('partOfSeries', webpage)
-        info.update({
-            'id': media_id,
-            'display_id': display_id,
-            'title': title,
-            'description': self._html_search_meta('description', webpage),
-            'series': series,
-            'episode': title,
-        })
-
-        for field in ('season', 'episode'):
-            field_name = field + 'Number'
-            info[field + '_number'] = int_or_none(find_field(
-                field_name, field + ' number', value_re=r'\d+') or self._html_search_meta(field_name, webpage))
-
-        return info

youtube_dl/extractor/cbc.py

@@ -1,14 +1,10 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import json
 import re
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_str,
-    compat_HTTPError,
-)
+from ..compat import compat_str
 from ..utils import (
     js_to_json,
     smuggle_url,
@@ -17,11 +13,8 @@ from ..utils import (
     xpath_element,
     xpath_with_ns,
     find_xpath_attr,
-    orderedSet,
-    parse_duration,
     parse_iso8601,
     parse_age_limit,
-    strip_or_none,
     int_or_none,
     ExtractorError,
 )
@@ -131,23 +124,15 @@ class CBCIE(InfoExtractor):
     def _real_extract(self, url):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
-        title = self._og_search_title(webpage, default=None) or self._html_search_meta(
-            'twitter:title', webpage, 'title', default=None) or self._html_search_regex(
-                r'<title>([^<]+)</title>', webpage, 'title', fatal=False)
         entries = [
             self._extract_player_init(player_init, display_id)
             for player_init in re.findall(r'CBC\.APP\.Caffeine\.initInstance\(({.+?})\);', webpage)]
-        media_ids = []
-        for media_id_re in (
-                r'<iframe[^>]+src="[^"]+?mediaId=(\d+)"',
-                r'<div[^>]+\bid=["\']player-(\d+)',
-                r'guid["\']\s*:\s*["\'](\d+)'):
-            media_ids.extend(re.findall(media_id_re, webpage))
         entries.extend([
             self.url_result('cbcplayer:%s' % media_id, 'CBCPlayer', media_id)
-            for media_id in orderedSet(media_ids)])
+            for media_id in re.findall(r'<iframe[^>]+src="[^"]+?mediaId=(\d+)"', webpage)])
         return self.playlist_result(
-            entries, display_id, strip_or_none(title),
+            entries, display_id,
+            self._og_search_title(webpage, fatal=False),
             self._og_search_description(webpage))
 
 
@@ -219,48 +204,30 @@ class CBCWatchBaseIE(InfoExtractor):
 
     def _call_api(self, path, video_id):
         url = path if path.startswith('http') else self._API_BASE_URL + path
-        for _ in range(2):
-            try:
-                result = self._download_xml(url, video_id, headers={
-                    'X-Clearleap-DeviceId': self._device_id,
-                    'X-Clearleap-DeviceToken': self._device_token,
-                })
-            except ExtractorError as e:
-                if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                    # Device token has expired, re-acquiring device token
-                    self._register_device()
-                    continue
-                raise
+        result = self._download_xml(url, video_id, headers={
+            'X-Clearleap-DeviceId': self._device_id,
+            'X-Clearleap-DeviceToken': self._device_token,
+        })
         error_message = xpath_text(result, 'userMessage') or xpath_text(result, 'systemMessage')
         if error_message:
             raise ExtractorError('%s said: %s' % (self.IE_NAME, error_message))
         return result
 
     def _real_initialize(self):
-        if self._valid_device_token():
-            return
-        device = self._downloader.cache.load('cbcwatch', 'device') or {}
-        self._device_id, self._device_token = device.get('id'), device.get('token')
-        if self._valid_device_token():
-            return
-        self._register_device()
-
-    def _valid_device_token(self):
-        return self._device_id and self._device_token
-
-    def _register_device(self):
-        self._device_id = self._device_token = None
-        result = self._download_xml(
-            self._API_BASE_URL + 'device/register',
-            None, 'Acquiring device token',
-            data=b'<device><type>web</type></device>')
-        self._device_id = xpath_text(result, 'deviceId', fatal=True)
-        self._device_token = xpath_text(result, 'deviceToken', fatal=True)
-        self._downloader.cache.store(
-            'cbcwatch', 'device', {
-                'id': self._device_id,
-                'token': self._device_token,
-            })
+        if not self._device_id or not self._device_token:
+            device = self._downloader.cache.load('cbcwatch', 'device') or {}
+            self._device_id, self._device_token = device.get('id'), device.get('token')
+            if not self._device_id or not self._device_token:
+                result = self._download_xml(
+                    self._API_BASE_URL + 'device/register',
+                    None, data=b'<device><type>web</type></device>')
+                self._device_id = xpath_text(result, 'deviceId', fatal=True)
+                self._device_token = xpath_text(result, 'deviceToken', fatal=True)
+                self._downloader.cache.store(
+                    'cbcwatch', 'device', {
+                        'id': self._device_id,
+                        'token': self._device_token,
+                    })
 
     def _parse_rss_feed(self, rss):
         channel = xpath_element(rss, 'channel', fatal=True)
@@ -392,63 +359,3 @@ class CBCWatchIE(CBCWatchBaseIE):
         video_id = self._match_id(url)
         rss = self._call_api('web/browse/' + video_id, video_id)
         return self._parse_rss_feed(rss)
-
-
-class CBCOlympicsIE(InfoExtractor):
-    IE_NAME = 'cbc.ca:olympics'
-    _VALID_URL = r'https?://olympics\.cbc\.ca/video/[^/]+/(?P<id>[^/?#]+)'
-    _TESTS = [{
-        'url': 'https://olympics.cbc.ca/video/whats-on-tv/olympic-morning-featuring-the-opening-ceremony/',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        video_id = self._hidden_inputs(webpage)['videoId']
-        video_doc = self._download_xml(
-            'https://olympics.cbc.ca/videodata/%s.xml' % video_id, video_id)
-        title = xpath_text(video_doc, 'title', fatal=True)
-        is_live = xpath_text(video_doc, 'kind') == 'Live'
-        if is_live:
-            title = self._live_title(title)
-
-        formats = []
-        for video_source in video_doc.findall('videoSources/videoSource'):
-            uri = xpath_text(video_source, 'uri')
-            if not uri:
-                continue
-            tokenize = self._download_json(
-                'https://olympics.cbc.ca/api/api-akamai/tokenize',
-                video_id, data=json.dumps({
-                    'VideoSource': uri,
-                }).encode(), headers={
-                    'Content-Type': 'application/json',
-                    'Referer': url,
-                    # d3.VideoPlayer._init in https://olympics.cbc.ca/components/script/base.js
-                    'Cookie': '_dvp=TK:C0ObxjerU',  # AKAMAI CDN cookie
-                }, fatal=False)
-            if not tokenize:
-                continue
-            content_url = tokenize['ContentUrl']
-            video_source_format = video_source.get('format')
-            if video_source_format == 'IIS':
-                formats.extend(self._extract_ism_formats(
-                    content_url, video_id, ism_id=video_source_format, fatal=False))
-            else:
-                formats.extend(self._extract_m3u8_formats(
-                    content_url, video_id, 'mp4',
-                    'm3u8' if is_live else 'm3u8_native',
-                    m3u8_id=video_source_format, fatal=False))
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'display_id': display_id,
-            'title': title,
-            'description': xpath_text(video_doc, 'description'),
-            'thumbnail': xpath_text(video_doc, 'thumbnailUrl'),
-            'duration': parse_duration(xpath_text(video_doc, 'duration')),
-            'formats': formats,
-            'is_live': is_live,
-        }

youtube_dl/extractor/cbs.py

@@ -2,7 +2,6 @@ from __future__ import unicode_literals
 
 from .theplatform import ThePlatformFeedIE
 from ..utils import (
-    ExtractorError,
     int_or_none,
     find_xpath_attr,
     xpath_element,
@@ -62,10 +61,9 @@ class CBSIE(CBSBaseIE):
         asset_types = []
         subtitles = {}
         formats = []
-        last_e = None
         for item in items_data.findall('.//item'):
             asset_type = xpath_text(item, 'assetType')
-            if not asset_type or asset_type in asset_types or asset_type in ('HLS_FPS', 'DASH_CENC'):
+            if not asset_type or asset_type in asset_types:
                 continue
             asset_types.append(asset_type)
             query = {
@@ -76,17 +74,11 @@ class CBSIE(CBSBaseIE):
                 query['formats'] = 'MPEG4,M3U'
             elif asset_type in ('RTMP', 'WIFI', '3G'):
                 query['formats'] = 'MPEG4,FLV'
-            try:
-                tp_formats, tp_subtitles = self._extract_theplatform_smil(
-                    update_url_query(tp_release_url, query), content_id,
-                    'Downloading %s SMIL data' % asset_type)
-            except ExtractorError as e:
-                last_e = e
-                continue
+            tp_formats, tp_subtitles = self._extract_theplatform_smil(
+                update_url_query(tp_release_url, query), content_id,
+                'Downloading %s SMIL data' % asset_type)
             formats.extend(tp_formats)
             subtitles = self._merge_subtitles(subtitles, tp_subtitles)
-        if last_e and not formats:
-            raise last_e
         self._sort_formats(formats)
 
         info = self._extract_theplatform_metadata(tp_path, content_id)

youtube_dl/extractor/cbsinteractive.py

@@ -75,10 +75,10 @@ class CBSInteractiveIE(CBSIE):
         webpage = self._download_webpage(url, display_id)
 
         data_json = self._html_search_regex(
-            r"data(?:-(?:cnet|zdnet))?-video(?:-(?:uvp(?:js)?|player))?-options='([^']+)'",
+            r"data-(?:cnet|zdnet)-video(?:-uvp(?:js)?)?-options='([^']+)'",
             webpage, 'data json')
         data = self._parse_json(data_json, display_id)
-        vdata = data.get('video') or (data.get('videos') or data.get('playlist'))[0]
+        vdata = data.get('video') or data['videos'][0]
 
         video_id = vdata['mpxRefId']
 

youtube_dl/extractor/cbssports.py

@@ -4,35 +4,28 @@ from .cbs import CBSBaseIE
 
 
 class CBSSportsIE(CBSBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/(?:video|news)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/video/player/[^/]+/(?P<id>\d+)'
 
     _TESTS = [{
-        'url': 'https://www.cbssports.com/nba/video/donovan-mitchell-flashes-star-potential-in-game-2-victory-over-thunder/',
+        'url': 'http://www.cbssports.com/video/player/videos/708337219968/0/ben-simmons-the-next-lebron?-not-so-fast',
         'info_dict': {
-            'id': '1214315075735',
+            'id': '708337219968',
             'ext': 'mp4',
-            'title': 'Donovan Mitchell flashes star potential in Game 2 victory over Thunder',
-            'description': 'md5:df6f48622612c2d6bd2e295ddef58def',
-            'timestamp': 1524111457,
-            'upload_date': '20180419',
+            'title': 'Ben Simmons the next LeBron? Not so fast',
+            'description': 'md5:854294f627921baba1f4b9a990d87197',
+            'timestamp': 1466293740,
+            'upload_date': '20160618',
             'uploader': 'CBSI-NEW',
         },
         'params': {
             # m3u8 download
             'skip_download': True,
         }
-    }, {
-        'url': 'https://www.cbssports.com/nba/news/nba-playoffs-2018-watch-76ers-vs-heat-game-3-series-schedule-tv-channel-online-stream/',
-        'only_matching': True,
     }]
 
     def _extract_video_info(self, filter_query, video_id):
         return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(
-            [r'(?:=|%26)pcid%3D(\d+)', r'embedVideo(?:Container)?_(\d+)'],
-            webpage, 'video id')
+        video_id = self._match_id(url)
         return self._extract_video_info('byId=%s' % video_id, video_id)

youtube_dl/extractor/ccma.py

@@ -5,12 +5,10 @@ import re
 
 from .common import InfoExtractor
 from ..utils import (
-    clean_html,
     int_or_none,
     parse_duration,
     parse_iso8601,
-    parse_resolution,
-    url_or_none,
+    clean_html,
 )
 
 
@@ -42,42 +40,34 @@ class CCMAIE(InfoExtractor):
 
     def _real_extract(self, url):
         media_type, media_id = re.match(self._VALID_URL, url).groups()
-
-        media = self._download_json(
-            'http://dinamics.ccma.cat/pvideo/media.jsp', media_id, query={
+        media_data = {}
+        formats = []
+        profiles = ['pc'] if media_type == 'audio' else ['mobil', 'pc']
+        for i, profile in enumerate(profiles):
+            md = self._download_json('http://dinamics.ccma.cat/pvideo/media.jsp', media_id, query={
                 'media': media_type,
                 'idint': media_id,
-            })
-
-        formats = []
-        media_url = media['media']['url']
-        if isinstance(media_url, list):
-            for format_ in media_url:
-                format_url = url_or_none(format_.get('file'))
-                if not format_url:
-                    continue
-                label = format_.get('label')
-                f = parse_resolution(label)
-                f.update({
-                    'url': format_url,
-                    'format_id': label,
-                })
-                formats.append(f)
-        else:
-            formats.append({
-                'url': media_url,
-                'vcodec': 'none' if media_type == 'audio' else None,
-            })
+                'profile': profile,
+            }, fatal=False)
+            if md:
+                media_data = md
+                media_url = media_data.get('media', {}).get('url')
+                if media_url:
+                    formats.append({
+                        'format_id': profile,
+                        'url': media_url,
+                        'quality': i,
+                    })
         self._sort_formats(formats)
 
-        informacio = media['informacio']
+        informacio = media_data['informacio']
         title = informacio['titol']
         durada = informacio.get('durada', {})
         duration = int_or_none(durada.get('milisegons'), 1000) or parse_duration(durada.get('text'))
         timestamp = parse_iso8601(informacio.get('data_emissio', {}).get('utc'))
 
         subtitles = {}
-        subtitols = media.get('subtitols', {})
+        subtitols = media_data.get('subtitols', {})
         if subtitols:
             sub_url = subtitols.get('url')
             if sub_url:
@@ -87,7 +77,7 @@ class CCMAIE(InfoExtractor):
                     })
 
         thumbnails = []
-        imatges = media.get('imatges', {})
+        imatges = media_data.get('imatges', {})
         if imatges:
             thumbnail_url = imatges.get('url')
             if thumbnail_url:

youtube_dl/extractor/ceskatelevize.py

@@ -13,7 +13,6 @@ from ..utils import (
     float_or_none,
     sanitized_Request,
     unescapeHTML,
-    update_url_query,
     urlencode_postdata,
     USER_AGENTS,
 )
@@ -108,7 +107,7 @@ class CeskaTelevizeIE(InfoExtractor):
 
         for user_agent in (None, USER_AGENTS['Safari']):
             req = sanitized_Request(
-                'https://www.ceskatelevize.cz/ivysilani/ajax/get-client-playlist',
+                'http://www.ceskatelevize.cz/ivysilani/ajax/get-client-playlist',
                 data=urlencode_postdata(data))
 
             req.add_header('Content-type', 'application/x-www-form-urlencoded')
@@ -266,10 +265,6 @@ class CeskaTelevizePoradyIE(InfoExtractor):
             # m3u8 download
             'skip_download': True,
         },
-    }, {
-        # iframe embed
-        'url': 'http://www.ceskatelevize.cz/porady/10614999031-neviditelni/21251212048/',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -277,11 +272,8 @@ class CeskaTelevizePoradyIE(InfoExtractor):
 
         webpage = self._download_webpage(url, video_id)
 
-        data_url = update_url_query(unescapeHTML(self._search_regex(
-            (r'<span[^>]*\bdata-url=(["\'])(?P<url>(?:(?!\1).)+)\1',
-             r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//(?:www\.)?ceskatelevize\.cz/ivysilani/embed/iFramePlayer\.php.*?)\1'),
-            webpage, 'iframe player url', group='url')), query={
-                'autoStart': 'true',
-        })
+        data_url = unescapeHTML(self._search_regex(
+            r'<span[^>]*\bdata-url=(["\'])(?P<url>(?:(?!\1).)+)\1',
+            webpage, 'iframe player url', group='url'))
 
         return self.url_result(data_url, ie=CeskaTelevizeIE.ie_key())

youtube_dl/extractor/chaturbate.py

@@ -31,8 +31,7 @@ class ChaturbateIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(
-            url, video_id, headers=self.geo_verification_headers())
+        webpage = self._download_webpage(url, video_id)
 
         m3u8_urls = []
 

youtube_dl/extractor/chilloutzone.py

@@ -1,11 +1,11 @@
 from __future__ import unicode_literals
 
 import re
+import base64
 import json
 
 from .common import InfoExtractor
 from .youtube import YoutubeIE
-from ..compat import compat_b64decode
 from ..utils import (
     clean_html,
     ExtractorError
@@ -58,7 +58,7 @@ class ChilloutzoneIE(InfoExtractor):
 
         base64_video_info = self._html_search_regex(
             r'var cozVidData = "(.+?)";', webpage, 'video data')
-        decoded_video_info = compat_b64decode(base64_video_info).decode('utf-8')
+        decoded_video_info = base64.b64decode(base64_video_info.encode('utf-8')).decode('utf-8')
         video_info_dict = json.loads(decoded_video_info)
 
         # get video information from dict

youtube_dl/extractor/chirbit.py

@@ -1,10 +1,10 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import base64
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_b64decode
 from ..utils import parse_duration
 
 
@@ -44,7 +44,8 @@ class ChirbitIE(InfoExtractor):
 
         # Reverse engineered from https://chirb.it/js/chirbit.player.js (look
         # for soundURL)
-        audio_url = compat_b64decode(data_fd[::-1]).decode('utf-8')
+        audio_url = base64.b64decode(
+            data_fd[::-1].encode('ascii')).decode('utf-8')
 
         title = self._search_regex(
             r'class=["\']chirbit-title["\'][^>]*>([^<]+)', webpage, 'title')

youtube_dl/extractor/ciscolive.py

@@ -1,142 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import itertools
-
-from .common import InfoExtractor
-from ..compat import (
-    compat_parse_qs,
-    compat_urllib_parse_urlparse,
-)
-from ..utils import (
-    clean_html,
-    float_or_none,
-    int_or_none,
-    try_get,
-    urlencode_postdata,
-)
-
-
-class CiscoLiveBaseIE(InfoExtractor):
-    # These appear to be constant across all Cisco Live presentations
-    # and are not tied to any user session or event
-    RAINFOCUS_API_URL = 'https://events.rainfocus.com/api/%s'
-    RAINFOCUS_API_PROFILE_ID = 'Na3vqYdAlJFSxhYTYQGuMbpafMqftalz'
-    RAINFOCUS_WIDGET_ID = 'n6l4Lo05R8fiy3RpUBm447dZN8uNWoye'
-    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/5647924234001/SyK2FdqjM_default/index.html?videoId=%s'
-
-    HEADERS = {
-        'Origin': 'https://ciscolive.cisco.com',
-        'rfApiProfileId': RAINFOCUS_API_PROFILE_ID,
-        'rfWidgetId': RAINFOCUS_WIDGET_ID,
-    }
-
-    def _call_api(self, ep, rf_id, query, referrer, note=None):
-        headers = self.HEADERS.copy()
-        headers['Referer'] = referrer
-        return self._download_json(
-            self.RAINFOCUS_API_URL % ep, rf_id, note=note,
-            data=urlencode_postdata(query), headers=headers)
-
-    def _parse_rf_item(self, rf_item):
-        event_name = rf_item.get('eventName')
-        title = rf_item['title']
-        description = clean_html(rf_item.get('abstract'))
-        presenter_name = try_get(rf_item, lambda x: x['participants'][0]['fullName'])
-        bc_id = rf_item['videos'][0]['url']
-        bc_url = self.BRIGHTCOVE_URL_TEMPLATE % bc_id
-        duration = float_or_none(try_get(rf_item, lambda x: x['times'][0]['length']))
-        location = try_get(rf_item, lambda x: x['times'][0]['room'])
-
-        if duration:
-            duration = duration * 60
-
-        return {
-            '_type': 'url_transparent',
-            'url': bc_url,
-            'ie_key': 'BrightcoveNew',
-            'title': title,
-            'description': description,
-            'duration': duration,
-            'creator': presenter_name,
-            'location': location,
-            'series': event_name,
-        }
-
-
-class CiscoLiveSessionIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/\??[^#]*#/session/(?P<id>[^/?&]+)'
-    _TEST = {
-        'url': 'https://ciscolive.cisco.com/on-demand-library/?#/session/1423353499155001FoSs',
-        'md5': 'c98acf395ed9c9f766941c70f5352e22',
-        'info_dict': {
-            'id': '5803694304001',
-            'ext': 'mp4',
-            'title': '13 Smart Automations to Monitor Your Cisco IOS Network',
-            'description': 'md5:ec4a436019e09a918dec17714803f7cc',
-            'timestamp': 1530305395,
-            'upload_date': '20180629',
-            'uploader_id': '5647924234001',
-            'location': '16B Mezz.',
-        },
-    }
-
-    def _real_extract(self, url):
-        rf_id = self._match_id(url)
-        rf_result = self._call_api('session', rf_id, {'id': rf_id}, url)
-        return self._parse_rf_item(rf_result['items'][0])
-
-
-class CiscoLiveSearchIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/'
-    _TESTS = [{
-        'url': 'https://ciscolive.cisco.com/on-demand-library/?search.event=ciscoliveus2018&search.technicallevel=scpsSkillLevel_aintroductory&search.focus=scpsSessionFocus_designAndDeployment#/',
-        'info_dict': {
-            'title': 'Search query',
-        },
-        'playlist_count': 5,
-    }, {
-        'url': 'https://ciscolive.cisco.com/on-demand-library/?search.technology=scpsTechnology_applicationDevelopment&search.technology=scpsTechnology_ipv6&search.focus=scpsSessionFocus_troubleshootingTroubleshooting#/',
-        'only_matching': True,
-    }]
-
-    @classmethod
-    def suitable(cls, url):
-        return False if CiscoLiveSessionIE.suitable(url) else super(CiscoLiveSearchIE, cls).suitable(url)
-
-    @staticmethod
-    def _check_bc_id_exists(rf_item):
-        return int_or_none(try_get(rf_item, lambda x: x['videos'][0]['url'])) is not None
-
-    def _entries(self, query, url):
-        query['size'] = 50
-        query['from'] = 0
-        for page_num in itertools.count(1):
-            results = self._call_api(
-                'search', None, query, url,
-                'Downloading search JSON page %d' % page_num)
-            sl = try_get(results, lambda x: x['sectionList'][0], dict)
-            if sl:
-                results = sl
-            items = results.get('items')
-            if not items or not isinstance(items, list):
-                break
-            for item in items:
-                if not isinstance(item, dict):
-                    continue
-                if not self._check_bc_id_exists(item):
-                    continue
-                yield self._parse_rf_item(item)
-            size = int_or_none(results.get('size'))
-            if size is not None:
-                query['size'] = size
-            total = int_or_none(results.get('total'))
-            if total is not None and query['from'] + query['size'] > total:
-                break
-            query['from'] += query['size']
-
-    def _real_extract(self, url):
-        query = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
-        query['type'] = 'session'
-        return self.playlist_result(
-            self._entries(query, url), playlist_title='Search query')

youtube_dl/extractor/cliphunter.py

@@ -1,10 +1,19 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..utils import (
-    int_or_none,
-    url_or_none,
-)
+from ..utils import int_or_none
+
+
+_translation_table = {
+    'a': 'h', 'd': 'e', 'e': 'v', 'f': 'o', 'g': 'f', 'i': 'd', 'l': 'n',
+    'm': 'a', 'n': 'm', 'p': 'u', 'q': 't', 'r': 's', 'v': 'p', 'x': 'r',
+    'y': 'l', 'z': 'i',
+    '$': ':', '&': '.', '(': '=', '^': '&', '=': '/',
+}
+
+
+def _decode(s):
+    return ''.join(_translation_table.get(c, c) for c in s)
 
 
 class CliphunterIE(InfoExtractor):
@@ -51,14 +60,14 @@ class CliphunterIE(InfoExtractor):
 
         formats = []
         for format_id, f in gexo_files.items():
-            video_url = url_or_none(f.get('url'))
+            video_url = f.get('url')
             if not video_url:
                 continue
             fmt = f.get('fmt')
             height = f.get('h')
             format_id = '%s_%sp' % (fmt, height) if fmt and height else format_id
             formats.append({
-                'url': video_url,
+                'url': _decode(video_url),
                 'format_id': format_id,
                 'width': int_or_none(f.get('w')),
                 'height': int_or_none(height),

youtube_dl/extractor/cloudflarestream.py

@@ -1,60 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-
-
-class CloudflareStreamIE(InfoExtractor):
-    _VALID_URL = r'''(?x)
-                    https?://
-                        (?:
-                            (?:watch\.)?cloudflarestream\.com/|
-                            embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=
-                        )
-                        (?P<id>[\da-f]+)
-                    '''
-    _TESTS = [{
-        'url': 'https://embed.cloudflarestream.com/embed/we4g.fla9.latest.js?video=31c9291ab41fac05471db4e73aa11717',
-        'info_dict': {
-            'id': '31c9291ab41fac05471db4e73aa11717',
-            'ext': 'mp4',
-            'title': '31c9291ab41fac05471db4e73aa11717',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }, {
-        'url': 'https://watch.cloudflarestream.com/9df17203414fd1db3e3ed74abbe936c1',
-        'only_matching': True,
-    }, {
-        'url': 'https://cloudflarestream.com/31c9291ab41fac05471db4e73aa11717/manifest/video.mpd',
-        'only_matching': True,
-    }]
-
-    @staticmethod
-    def _extract_urls(webpage):
-        return [
-            mobj.group('url')
-            for mobj in re.finditer(
-                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=[\da-f]+?.*?)\1',
-                webpage)]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        formats = self._extract_m3u8_formats(
-            'https://cloudflarestream.com/%s/manifest/video.m3u8' % video_id,
-            video_id, 'mp4', entry_protocol='m3u8_native', m3u8_id='hls',
-            fatal=False)
-        formats.extend(self._extract_mpd_formats(
-            'https://cloudflarestream.com/%s/manifest/video.mpd' % video_id,
-            video_id, mpd_id='dash', fatal=False))
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'title': video_id,
-            'formats': formats,
-        }

youtube_dl/extractor/clyp.py

@@ -1,19 +1,15 @@
 from __future__ import unicode_literals
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_parse_qs,
-    compat_urllib_parse_urlparse,
-)
 from ..utils import (
     float_or_none,
-    unified_timestamp,
+    parse_iso8601,
 )
 
 
 class ClypIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?clyp\.it/(?P<id>[a-z0-9]+)'
-    _TESTS = [{
+    _TEST = {
         'url': 'https://clyp.it/ojz2wfah',
         'md5': '1d4961036c41247ecfdcc439c0cddcbb',
         'info_dict': {
@@ -25,34 +21,13 @@ class ClypIE(InfoExtractor):
             'timestamp': 1443515251,
             'upload_date': '20150929',
         },
-    }, {
-        'url': 'https://clyp.it/b04p1odi?token=b0078e077e15835845c528a44417719d',
-        'info_dict': {
-            'id': 'b04p1odi',
-            'ext': 'mp3',
-            'title': 'GJ! (Reward Edit)',
-            'description': 'Metal Resistance (THE ONE edition)',
-            'duration': 177.789,
-            'timestamp': 1528241278,
-            'upload_date': '20180605',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }]
+    }
 
     def _real_extract(self, url):
         audio_id = self._match_id(url)
 
-        qs = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
-        token = qs.get('token', [None])[0]
-
-        query = {}
-        if token:
-            query['token'] = token
-
         metadata = self._download_json(
-            'https://api.clyp.it/%s' % audio_id, audio_id, query=query)
+            'https://api.clyp.it/%s' % audio_id, audio_id)
 
         formats = []
         for secure in ('', 'Secure'):
@@ -70,7 +45,7 @@ class ClypIE(InfoExtractor):
         title = metadata['Title']
         description = metadata.get('Description')
         duration = float_or_none(metadata.get('Duration'))
-        timestamp = unified_timestamp(metadata.get('DateCreated'))
+        timestamp = parse_iso8601(metadata.get('DateCreated'))
 
         return {
             'id': audio_id,

youtube_dl/extractor/cnbc.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-
 from .common import InfoExtractor
 from ..utils import smuggle_url
 
@@ -35,32 +34,3 @@ class CNBCIE(InfoExtractor):
                 {'force_smil_url': True}),
             'id': video_id,
         }
-
-
-class CNBCVideoIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?cnbc\.com/video/(?:[^/]+/)+(?P<id>[^./?#&]+)'
-    _TEST = {
-        'url': 'https://www.cnbc.com/video/2018/07/19/trump-i-dont-necessarily-agree-with-raising-rates.html',
-        'info_dict': {
-            'id': '7000031301',
-            'ext': 'mp4',
-            'title': "Trump: I don't necessarily agree with raising rates",
-            'description': 'md5:878d8f0b4ebb5bb1dda3514b91b49de3',
-            'timestamp': 1531958400,
-            'upload_date': '20180719',
-            'uploader': 'NBCU-CNBC',
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(
-            r'content_id["\']\s*:\s*["\'](\d+)', webpage, display_id,
-            'video id')
-        return self.url_result(
-            'http://video.cnbc.com/gallery/?video=%s' % video_id,
-            CNBCIE.ie_key())