release 2015.12.13

Update funimation.py

Update funimation.py

Removed unnecessary lines.

Update funimation.py

Added thumbnail and description.

Filename improvement.

fixed TEST.

AUTHORS

@@ -147,3 +147,5 @@ Qijiang Fan
 Rémy Léone
 Marco Ferragina
 reiv
+Muratcan Simsek
+Evan Lu

CONTRIBUTING.md

@@ -34,7 +34,7 @@ For bug reports, this means that your report should contain the *complete* outpu
 
 If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
 
-**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL.
+**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `http://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `http://www.youtube.com/`) is *not* an example URL.
 
 ###  Are you using the latest version?
 
@@ -42,7 +42,7 @@ Before reporting any issue, type `youtube-dl -U`. This should report that you're
 
 ###  Is the issue already documented?
 
-Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or at https://github.com/rg3/youtube-dl/search?type=Issues . If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
+Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/rg3/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
 
 ###  Why are existing options not enough?
 

README.md

@@ -35,7 +35,7 @@ You can also use pip:
 
     sudo pip install youtube-dl
 
-Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see https://rg3.github.io/youtube-dl/download.html .
+Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see the [youtube-dl Download Page](https://rg3.github.io/youtube-dl/download.html).
 
 # DESCRIPTION
 **youtube-dl** is a small command-line program to download videos from
@@ -414,7 +414,7 @@ You can configure youtube-dl by placing any supported command line option to a c
 
 You can use `--ignore-config` if you want to disable the configuration file for a particular youtube-dl run.
 
-### Authentication with `.netrc` file ###
+### Authentication with `.netrc` file
 
 You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every youtube-dl execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](http://stackoverflow.com/tags/.netrc/info) on per extractor basis. For that you will need to create a`.netrc` file in your `$HOME` and restrict permissions to read/write by you only:
 ```
@@ -559,11 +559,11 @@ If you want to play the video on a machine that is not running youtube-dl, you c
 
 YouTube has switched to a new video info format in July 2011 which is not supported by old versions of youtube-dl. See [above](#how-do-i-update-youtube-dl) for how to update youtube-dl.
 
-### ERROR: unable to download video ###
+### ERROR: unable to download video
 
 YouTube requires an additional signature since September 2012 which is not supported by old versions of youtube-dl. See [above](#how-do-i-update-youtube-dl) for how to update youtube-dl.
 
-### Video URL contains an ampersand and I'm getting some strange output `[1] 2839` or `'v' is not recognized as an internal or external command` ###
+### Video URL contains an ampersand and I'm getting some strange output `[1] 2839` or `'v' is not recognized as an internal or external command`
 
 That's actually the output from your shell. Since ampersand is one of the special shell characters it's interpreted by the shell preventing you from passing the whole URL to youtube-dl. To disable your shell from interpreting the ampersands (or any other special characters) you have to either put the whole URL in quotes or escape them with a backslash (which approach will work depends on your shell).
 
@@ -587,7 +587,7 @@ In February 2015, the new YouTube player contained a character sequence in a str
 
 These two error codes indicate that the service is blocking your IP address because of overuse. Contact the service and ask them to unblock your IP address, or - if you have acquired a whitelisted IP address already - use the [`--proxy` or `--source-address` options](#network-options) to select another IP address.
 
-### SyntaxError: Non-ASCII character ###
+### SyntaxError: Non-ASCII character
 
 The error
 
@@ -616,7 +616,7 @@ From then on, after restarting your shell, you will be able to access both youtu
 
 Use the `-o` to specify an [output template](#output-template), for example `-o "/home/user/videos/%(title)s-%(id)s.%(ext)s"`. If you want this for all of your downloads, put the option into your [configuration file](#configuration).
 
-### How do I download a video starting with a `-` ?
+### How do I download a video starting with a `-`?
 
 Either prepend `http://www.youtube.com/watch?v=` or separate the ID from the options with `--`:
 
@@ -798,7 +798,7 @@ with youtube_dl.YoutubeDL(ydl_opts) as ydl:
 
 # BUGS
 
-Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues> . Unless you were prompted so or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the irc channel #youtube-dl on freenode.
+Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues>. Unless you were prompted so or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the IRC channel [#youtube-dl](irc://chat.freenode.net/#youtube-dl) on freenode ([webchat](http://webchat.freenode.net/?randomnick=1&channels=youtube-dl)).
 
 **Please include the full output of youtube-dl when run with `-v`**, i.e. add `-v` flag to your command line, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this:
 ```
@@ -836,7 +836,7 @@ For bug reports, this means that your report should contain the *complete* outpu
 
 If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
 
-**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL.
+**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `http://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `http://www.youtube.com/`) is *not* an example URL.
 
 ###  Are you using the latest version?
 
@@ -844,7 +844,7 @@ Before reporting any issue, type `youtube-dl -U`. This should report that you're
 
 ###  Is the issue already documented?
 
-Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or at https://github.com/rg3/youtube-dl/search?type=Issues . If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
+Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/rg3/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
 
 ###  Why are existing options not enough?
 
@@ -874,4 +874,4 @@ It may sound strange, but some bug reports we receive are completely unrelated t
 
 youtube-dl is released into the public domain by the copyright holders.
 
-This README file was originally written by Daniel Bolton (<https://github.com/dbbolton>) and is likewise released into the public domain.
+This README file was originally written by [Daniel Bolton](https://github.com/dbbolton) and is likewise released into the public domain.

youtube_dl/downloader/f4m.py

@@ -15,6 +15,7 @@ from ..compat import (
 )
 from ..utils import (
     encodeFilename,
+    fix_xml_ampersands,
     sanitize_open,
     struct_pack,
     struct_unpack,
@@ -288,7 +289,10 @@ class F4mFD(FragmentFD):
         self.to_screen('[%s] Downloading f4m manifest' % self.FD_NAME)
         urlh = self.ydl.urlopen(man_url)
         man_url = urlh.geturl()
-        manifest = urlh.read()
+        # Some manifests may be malformed, e.g. prosiebensat1 generated manifests
+        # (see https://github.com/rg3/youtube-dl/issues/6215#issuecomment-121704244
+        # and https://github.com/rg3/youtube-dl/issues/7823)
+        manifest = fix_xml_ampersands(urlh.read().decode('utf-8', 'ignore')).strip()
 
         doc = compat_etree_fromstring(manifest)
         formats = [(int(f.attrib.get('bitrate', -1)), f)

youtube_dl/extractor/__init__.py

@@ -206,6 +206,7 @@ from .francetv import (
 from .freesound import FreesoundIE
 from .freespeech import FreespeechIE
 from .freevideo import FreeVideoIE
+from .funimation import FunimationIE
 from .funnyordie import FunnyOrDieIE
 from .gameinformer import GameInformerIE
 from .gamekings import GamekingsIE

youtube_dl/extractor/beeg.py

@@ -34,15 +34,29 @@ class BeegIE(InfoExtractor):
         video_id = self._match_id(url)
 
         video = self._download_json(
-            'http://beeg.com/api/v4/video/%s' % video_id, video_id)
+            'http://beeg.com/api/v5/video/%s' % video_id, video_id)
+
+        def split(o, e):
+            def cut(s, x):
+                n.append(s[:x])
+                return s[x:]
+            n = []
+            r = len(o) % e
+            if r > 0:
+                o = cut(o, r)
+            while len(o) > e:
+                o = cut(o, e)
+            n.append(o)
+            return n
 
         def decrypt_key(key):
-            # Reverse engineered from http://static.beeg.com/cpl/1067.js
-            a = '8RPUUCS35ZWp3ADnKcSmpH71ZusrROo'
+            # Reverse engineered from http://static.beeg.com/cpl/1105.js
+            a = '5ShMcIQlssOd7zChAIOlmeTZDaUxULbJRnywYaiB'
             e = compat_urllib_parse_unquote(key)
-            return ''.join([
-                compat_chr(compat_ord(e[n]) - compat_ord(a[n % len(a)]) % 25)
+            o = ''.join([
+                compat_chr(compat_ord(e[n]) - compat_ord(a[n % len(a)]) % 21)
                 for n in range(len(e))])
+            return ''.join(split(o, 3)[::-1])
 
         def decrypt_url(encrypted_url):
             encrypted_url = self._proto_relative_url(

youtube_dl/extractor/dailymotion.py

@@ -99,6 +99,11 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
         {
             'url': 'http://www.dailymotion.com/video/xhza0o',
             'only_matching': True,
+        },
+        # with subtitles
+        {
+            'url': 'http://www.dailymotion.com/video/x20su5f_the-power-of-nightmares-1-the-rise-of-the-politics-of-fear-bbc-2004_news',
+            'only_matching': True,
         }
     ]
 
@@ -122,7 +127,9 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
             webpage, 'comment count', fatal=False))
 
         player_v5 = self._search_regex(
-            [r'buildPlayer\(({.+?})\);', r'playerV5\s*=\s*dmp\.create\([^,]+?,\s*({.+?})\);'],
+            [r'buildPlayer\(({.+?})\);\n',  # See https://github.com/rg3/youtube-dl/issues/7826
+             r'playerV5\s*=\s*dmp\.create\([^,]+?,\s*({.+?})\);',
+             r'buildPlayer\(({.+?})\);'],
             webpage, 'player v5', default=None)
         if player_v5:
             player = self._parse_json(player_v5, video_id)
@@ -172,11 +179,13 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
             uploader_id = metadata.get('owner', {}).get('id')
 
             subtitles = {}
-            for subtitle_lang, subtitle in metadata.get('subtitles', {}).get('data', {}).items():
-                subtitles[subtitle_lang] = [{
-                    'ext': determine_ext(subtitle_url),
-                    'url': subtitle_url,
-                } for subtitle_url in subtitle.get('urls', [])]
+            subtitles_data = metadata.get('subtitles', {}).get('data', {})
+            if subtitles_data and isinstance(subtitles_data, dict):
+                for subtitle_lang, subtitle in subtitles_data.items():
+                    subtitles[subtitle_lang] = [{
+                        'ext': determine_ext(subtitle_url),
+                        'url': subtitle_url,
+                    } for subtitle_url in subtitle.get('urls', [])]
 
             return {
                 'id': video_id,

youtube_dl/extractor/funimation.py

@@ -0,0 +1,193 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_html,
+    determine_ext,
+    encode_dict,
+    int_or_none,
+    sanitized_Request,
+    ExtractorError,
+    urlencode_postdata
+)
+
+
+class FunimationIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?funimation\.com/shows/[^/]+/videos/(?:official|promotional)/(?P<id>[^/?#&]+)'
+
+    _NETRC_MACHINE = 'funimation'
+
+    _TESTS = [{
+        'url': 'http://www.funimation.com/shows/air/videos/official/breeze',
+        'info_dict': {
+            'id': '658',
+            'display_id': 'breeze',
+            'ext': 'mp4',
+            'title': 'Air - 1 - Breeze',
+            'description': 'md5:1769f43cd5fc130ace8fd87232207892',
+            'thumbnail': 're:https?://.*\.jpg',
+        },
+    }, {
+        'url': 'http://www.funimation.com/shows/hacksign/videos/official/role-play',
+        'info_dict': {
+            'id': '31128',
+            'display_id': 'role-play',
+            'ext': 'mp4',
+            'title': '.hack//SIGN - 1 - Role Play',
+            'description': 'md5:b602bdc15eef4c9bbb201bb6e6a4a2dd',
+            'thumbnail': 're:https?://.*\.jpg',
+        },
+    }, {
+        'url': 'http://www.funimation.com/shows/attack-on-titan-junior-high/videos/promotional/broadcast-dub-preview',
+        'info_dict': {
+            'id': '9635',
+            'display_id': 'broadcast-dub-preview',
+            'ext': 'mp4',
+            'title': 'Attack on Titan: Junior High - Broadcast Dub Preview',
+            'description': 'md5:f8ec49c0aff702a7832cd81b8a44f803',
+            'thumbnail': 're:https?://.*\.(?:jpg|png)',
+        },
+    }]
+
+    def _login(self):
+        (username, password) = self._get_login_info()
+        if username is None:
+            return
+        data = urlencode_postdata(encode_dict({
+            'email_field': username,
+            'password_field': password,
+        }))
+        login_request = sanitized_Request('http://www.funimation.com/login', data, headers={
+            'User-Agent': 'Mozilla/5.0 (Windows NT 5.2; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0',
+            'Content-Type': 'application/x-www-form-urlencoded'
+        })
+        login_page = self._download_webpage(
+            login_request, None, 'Logging in as %s' % username)
+        if any(p in login_page for p in ('funimation.com/logout', '>Log Out<')):
+            return
+        error = self._html_search_regex(
+            r'(?s)<div[^>]+id=["\']errorMessages["\'][^>]*>(.+?)</div>',
+            login_page, 'error messages', default=None)
+        if error:
+            raise ExtractorError('Unable to login: %s' % error, expected=True)
+        raise ExtractorError('Unable to log in')
+
+    def _real_initialize(self):
+        self._login()
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+
+        errors = []
+        formats = []
+
+        ERRORS_MAP = {
+            'ERROR_MATURE_CONTENT_LOGGED_IN': 'matureContentLoggedIn',
+            'ERROR_MATURE_CONTENT_LOGGED_OUT': 'matureContentLoggedOut',
+            'ERROR_SUBSCRIPTION_LOGGED_OUT': 'subscriptionLoggedOut',
+            'ERROR_VIDEO_EXPIRED': 'videoExpired',
+            'ERROR_TERRITORY_UNAVAILABLE': 'territoryUnavailable',
+            'SVODBASIC_SUBSCRIPTION_IN_PLAYER': 'basicSubscription',
+            'SVODNON_SUBSCRIPTION_IN_PLAYER': 'nonSubscription',
+            'ERROR_PLAYER_NOT_RESPONDING': 'playerNotResponding',
+            'ERROR_UNABLE_TO_CONNECT_TO_CDN': 'unableToConnectToCDN',
+            'ERROR_STREAM_NOT_FOUND': 'streamNotFound',
+        }
+
+        USER_AGENTS = (
+            # PC UA is served with m3u8 that provides some bonus lower quality formats
+            ('pc', 'Mozilla/5.0 (Windows NT 5.2; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0'),
+            # Mobile UA allows to extract direct links and also does not fail when
+            # PC UA fails with hulu error (e.g.
+            # http://www.funimation.com/shows/hacksign/videos/official/role-play)
+            ('mobile', 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36'),
+        )
+
+        for kind, user_agent in USER_AGENTS:
+            request = sanitized_Request(url)
+            request.add_header('User-Agent', user_agent)
+            webpage = self._download_webpage(
+                request, display_id, 'Downloading %s webpage' % kind)
+
+            playlist = self._parse_json(
+                self._search_regex(
+                    r'var\s+playersData\s*=\s*(\[.+?\]);\n',
+                    webpage, 'players data'),
+                display_id)[0]['playlist']
+
+            items = next(item['items'] for item in playlist if item.get('items'))
+            item = next(item for item in items if item.get('itemAK') == display_id)
+
+            error_messages = {}
+            video_error_messages = self._search_regex(
+                r'var\s+videoErrorMessages\s*=\s*({.+?});\n',
+                webpage, 'error messages', default=None)
+            if video_error_messages:
+                error_messages_json = self._parse_json(video_error_messages, display_id, fatal=False)
+                if error_messages_json:
+                    for _, error in error_messages_json.items():
+                        type_ = error.get('type')
+                        description = error.get('description')
+                        content = error.get('content')
+                        if type_ == 'text' and description and content:
+                            error_message = ERRORS_MAP.get(description)
+                            if error_message:
+                                error_messages[error_message] = content
+
+            for video in item.get('videoSet', []):
+                auth_token = video.get('authToken')
+                if not auth_token:
+                    continue
+                funimation_id = video.get('FUNImationID') or video.get('videoId')
+                preference = 1 if video.get('languageMode') == 'dub' else 0
+                if not auth_token.startswith('?'):
+                    auth_token = '?%s' % auth_token
+                for quality, height in (('sd', 480), ('hd', 720), ('hd1080', 1080)):
+                    format_url = video.get('%sUrl' % quality)
+                    if not format_url:
+                        continue
+                    if not format_url.startswith(('http', '//')):
+                        errors.append(format_url)
+                        continue
+                    if determine_ext(format_url) == 'm3u8':
+                        m3u8_formats = self._extract_m3u8_formats(
+                            format_url + auth_token, display_id, 'mp4', entry_protocol='m3u8_native',
+                            preference=preference, m3u8_id='%s-hls' % funimation_id, fatal=False)
+                        if m3u8_formats:
+                            formats.extend(m3u8_formats)
+                    else:
+                        tbr = int_or_none(self._search_regex(
+                            r'-(\d+)[Kk]', format_url, 'tbr', default=None))
+                        formats.append({
+                            'url': format_url + auth_token,
+                            'format_id': '%s-http-%dp' % (funimation_id, height),
+                            'height': height,
+                            'tbr': tbr,
+                            'preference': preference,
+                        })
+
+        if not formats and errors:
+            raise ExtractorError(
+                '%s returned error: %s'
+                % (self.IE_NAME, clean_html(error_messages.get(errors[0], errors[0]))),
+                expected=True)
+
+        self._sort_formats(formats)
+
+        title = item['title']
+        artist = item.get('artist')
+        if artist:
+            title = '%s - %s' % (artist, title)
+        description = self._og_search_description(webpage) or item.get('description')
+        thumbnail = self._og_search_thumbnail(webpage) or item.get('posterUrl')
+        video_id = item.get('itemId') or display_id
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+            'thumbnail': thumbnail,
+            'formats': formats,
+        }

youtube_dl/extractor/infoq.py

@@ -3,10 +3,7 @@ from __future__ import unicode_literals
 import base64
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_urllib_parse_unquote,
-    compat_urlparse,
-)
+from ..compat import compat_urllib_parse_unquote
 
 
 class InfoQIE(InfoExtractor):
@@ -45,9 +42,11 @@ class InfoQIE(InfoExtractor):
         video_filename = playpath.split('/')[-1]
         video_id, extension = video_filename.split('.')
 
-        http_base = self._search_regex(
-            r'EXPRESSINSTALL_SWF\s*=\s*[^"]*"((?:https?:)?//[^/"]+/)', webpage,
-            'HTTP base URL')
+        http_video_url = self._search_regex(r'P\.s\s*=\s*\'([^\']+)\'', webpage, 'video URL')
+
+        policy = self._search_regex(r'InfoQConstants.scp\s*=\s*\'([^\']+)\'', webpage, 'policy')
+        signature = self._search_regex(r'InfoQConstants.scs\s*=\s*\'([^\']+)\'', webpage, 'signature')
+        key_pair_id = self._search_regex(r'InfoQConstants.sck\s*=\s*\'([^\']+)\'', webpage, 'key-pair-id')
 
         formats = [{
             'format_id': 'rtmp',
@@ -56,7 +55,11 @@ class InfoQIE(InfoExtractor):
             'play_path': playpath,
         }, {
             'format_id': 'http',
-            'url': compat_urlparse.urljoin(url, http_base) + real_id,
+            'url': http_video_url,
+            'http_headers': {
+                'Cookie': 'CloudFront-Policy=%s; CloudFront-Signature=%s; CloudFront-Key-Pair-Id=%s' % (
+                    policy, signature, key_pair_id),
+            },
         }]
         self._sort_formats(formats)
 

youtube_dl/extractor/novamov.py

@@ -23,9 +23,10 @@ class NovaMovIE(InfoExtractor):
     _HOST = 'www.novamov.com'
 
     _FILE_DELETED_REGEX = r'This file no longer exists on our servers!</h2>'
-    _FILEKEY_REGEX = r'flashvars\.filekey="(?P<filekey>[^"]+)";'
+    _FILEKEY_REGEX = r'flashvars\.filekey=(?P<filekey>"?[^"]+"?);'
     _TITLE_REGEX = r'(?s)<div class="v_tab blockborder rounded5" id="v_tab1">\s*<h3>([^<]+)</h3>'
     _DESCRIPTION_REGEX = r'(?s)<div class="v_tab blockborder rounded5" id="v_tab1">\s*<h3>[^<]+</h3><p>([^<]+)</p>'
+    _URL_TEMPLATE = 'http://%s/video/%s'
 
     _TEST = {
         'url': 'http://www.novamov.com/video/4rurhn9x446jj',
@@ -39,20 +40,28 @@ class NovaMovIE(InfoExtractor):
         'skip': '"Invalid token" errors abound (in web interface as well as youtube-dl, there is nothing we can do about it.)'
     }
 
+    def _check_existence(self, webpage, video_id):
+        if re.search(self._FILE_DELETED_REGEX, webpage) is not None:
+            raise ExtractorError('Video %s does not exist' % video_id, expected=True)
+
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        url = 'http://%s/video/%s' % (self._HOST, video_id)
+        url = self._URL_TEMPLATE % (self._HOST, video_id)
 
         webpage = self._download_webpage(
             url, video_id, 'Downloading video page')
 
-        if re.search(self._FILE_DELETED_REGEX, webpage) is not None:
-            raise ExtractorError('Video %s does not exist' % video_id, expected=True)
+        self._check_existence(webpage, video_id)
 
         def extract_filekey(default=NO_DEFAULT):
-            return self._search_regex(
+            filekey = self._search_regex(
                 self._FILEKEY_REGEX, webpage, 'filekey', default=default)
+            if filekey is not default and (filekey[0] != '"' or filekey[-1] != '"'):
+                return self._search_regex(
+                    r'var\s*%s\s*=\s*"([^"]+)"', webpage, 'filekey', default=default)
+            else:
+                return filekey
 
         filekey = extract_filekey(default=None)
 
@@ -69,6 +78,7 @@ class NovaMovIE(InfoExtractor):
             request.add_header('Referer', post_url)
             webpage = self._download_webpage(
                 request, video_id, 'Downloading continue to the video page')
+            self._check_existence(webpage, video_id)
 
         filekey = extract_filekey()
 
@@ -127,7 +137,6 @@ class NowVideoIE(NovaMovIE):
     _HOST = 'www.nowvideo.to'
 
     _FILE_DELETED_REGEX = r'>This file no longer exists on our servers.<'
-    _FILEKEY_REGEX = r'var fkzd="([^"]+)";'
     _TITLE_REGEX = r'<h4>([^<]+)</h4>'
     _DESCRIPTION_REGEX = r'</h4>\s*<p>([^<]+)</p>'
 
@@ -139,7 +148,8 @@ class NowVideoIE(NovaMovIE):
             'ext': 'flv',
             'title': 'youtubedl test video _BaW_jenozKc.mp4',
             'description': 'Description',
-        }
+        },
+        'skip': 'Video 0mw0yow7b6dxa does not exist',
     }
 
 
@@ -153,6 +163,7 @@ class VideoWeedIE(NovaMovIE):
 
     _FILE_DELETED_REGEX = r'>This file no longer exists on our servers.<'
     _TITLE_REGEX = r'<h1 class="text_shadow">([^<]+)</h1>'
+    _URL_TEMPLATE = 'http://%s/file/%s'
 
     _TEST = {
         'url': 'http://www.videoweed.es/file/b42178afbea14',

youtube_dl/extractor/ooyala.py

@@ -27,6 +27,7 @@ class OoyalaBaseIE(InfoExtractor):
             'duration': float_or_none(metadata.get('duration'), 1000),
         }
 
+        urls = []
         formats = []
         for supported_format in ('mp4', 'm3u8', 'hds', 'rtmp'):
             auth_data = self._download_json(
@@ -38,20 +39,28 @@ class OoyalaBaseIE(InfoExtractor):
             if cur_auth_data['authorized']:
                 for stream in cur_auth_data['streams']:
                     url = base64.b64decode(stream['url']['data'].encode('ascii')).decode('utf-8')
+                    if url in urls:
+                        continue
+                    urls.append(url)
                     delivery_type = stream['delivery_type']
-                    if delivery_type == 'remote_asset':
-                        video_info['url'] = url
-                        return video_info
-                    if delivery_type == 'hls':
-                        formats.extend(self._extract_m3u8_formats(url, embed_code, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
-                    elif delivery_type == 'hds':
-                        formats.extend(self._extract_f4m_formats(url, embed_code, -1, 'hds', fatal=False))
+                    if delivery_type == 'hls' or '.m3u8' in url:
+                        m3u8_formats = self._extract_m3u8_formats(url, embed_code, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False)
+                        if m3u8_formats:
+                            formats.extend(m3u8_formats)
+                    elif delivery_type == 'hds' or '.f4m' in url:
+                        f4m_formats = self._extract_f4m_formats(url, embed_code, f4m_id='hds', fatal=False)
+                        if f4m_formats:
+                            formats.extend(f4m_formats)
+                    elif '.smil' in url:
+                        smil_formats = self._extract_smil_formats(url, embed_code, fatal=False)
+                        if smil_formats:
+                            formats.extend(smil_formats)
                     else:
                         formats.append({
                             'url': url,
                             'ext': stream.get('delivery_type'),
                             'vcodec': stream.get('video_codec'),
-                            'format_id': '%s-%s-%sp' % (stream.get('profile'), delivery_type, stream.get('height')),
+                            'format_id': delivery_type,
                             'width': int_or_none(stream.get('width')),
                             'height': int_or_none(stream.get('height')),
                             'abr': int_or_none(stream.get('audio_bitrate')),

youtube_dl/extractor/pbs.py

@@ -16,473 +16,165 @@ from ..utils import (
 
 class PBSIE(InfoExtractor):
     _STATIONS = (
-        ('pbs.org', 'PBS: Public Broadcasting Service'),  # http://www.pbs.org/
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # AL, http://aptv.org/
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # AL, http://www.gpb.org/
-        ('mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # AL, http://www.mpbonline.org
-        ('wnpt.org', 'Nashville Public Television (WNPT)'),  # AL, http://www.wnpt.org
-        ('wfsu.org', 'WFSU-TV (WFSU)'),  # AL, http://wfsu.org/
-        ('wsre.org', 'WSRE (WSRE)'),  # AL, http://www.wsre.org
-        ('wtcitv.org', 'WTCI (WTCI)'),  # AL, http://www.wtcitv.org
-        ('pba.org', 'WPBA/Channel 30 (WPBA)'),  # AL, http://pba.org/
-        ('alaskapublic.org', 'Alaska Public Media (KAKM)'),  # AK, http://alaskapublic.org/kakm
-        ('kuac.org', 'KUAC (KUAC)'),  # AK, http://kuac.org/kuac-tv/
-        ('ktoo.org', '360 North (KTOO)'),  # AK, http://www.ktoo.org/
-        ('azpm.org', 'KUAT 6 (KUAT)'),  # AZ, http://www.azpm.org/
-        ('azpbs.org', 'Arizona PBS (KAET)'),  # AZ, http://www.azpbs.org
-        ('newmexicopbs.org', 'KNME-TV/Channel 5 (KNME)'),  # AZ, http://www.newmexicopbs.org/
-        ('vegaspbs.org', 'Vegas PBS (KLVX)'),  # AZ, http://vegaspbs.org/
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # AR, http://www.aetn.org/
-        ('ket.org', 'KET (WKLE)'),  # AR, http://www.ket.org/
-        ('wkno.org', 'WKNO/Channel 10 (WKNO)'),  # AR, http://www.wkno.org/
-        ('lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # AR, http://www.lpb.org/
-        ('mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # AR, http://www.mpbonline.org
-        ('oeta.tv', 'OETA (KETA)'),  # AR, http://www.oeta.tv
-        ('optv.org', 'Ozarks Public Television (KOZK)'),  # AR, http://www.optv.org/
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # AR, http://www.wsiu.org/
-        ('keet.org', 'KEET TV (KEET)'),  # CA, http://www.keet.org
-        ('kixe.org', 'KIXE/Channel 9 (KIXE)'),  # CA, http://kixe.org/
-        ('kpbs.org', 'KPBS San Diego (KPBS)'),  # CA, http://www.kpbs.org/
-        ('kqed.org', 'KQED (KQED)'),  # CA, http://www.kqed.org
-        ('kvie.org', 'KVIE Public Television (KVIE)'),  # CA, http://www.kvie.org
-        ('pbssocal.org', 'PBS SoCal/KOCE (KOCE)'),  # CA, http://www.pbssocal.org/
-        ('valleypbs.org', 'ValleyPBS (KVPT)'),  # CA, http://www.valleypbs.org/
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # CA, http://aptv.org/
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # CA, http://cptv.org
-        ('azpbs.org', 'Arizona PBS (KAET)'),  # CA, http://www.azpbs.org
-        ('knpb.org', 'KNPB Channel 5 (KNPB)'),  # CA, http://www.knpb.org/
-        ('soptv.org', 'SOPTV (KSYS)'),  # CA, http://www.soptv.org
-        ('klcs.org', 'KLCS/Channel 58 (KLCS)'),  # CA, http://www.klcs.org
-        ('krcb.org', 'KRCB Television & Radio (KRCB)'),  # CA, http://www.krcb.org
-        ('kvcr.org', 'KVCR TV/DT/FM :: Vision for the Future (KVCR)'),  # CA, http://kvcr.org
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # CO, http://www.rmpbs.org
-        ('kenw.org', 'KENW-TV3 (KENW)'),  # CO, http://www.kenw.org
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # CO, http://www.kued.org
-        ('newmexicopbs.org', 'KNME-TV/Channel 5 (KNME)'),  # CO, http://www.newmexicopbs.org/
-        ('wyomingpbs.org', 'Wyoming PBS (KCWC)'),  # CO, http://www.wyomingpbs.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # CO, http://www.cpt12.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # CO, http://www.kbyutv.org/
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # CT, http://cptv.org
-        ('thirteen.org', 'Thirteen/WNET New York (WNET)'),  # CT, http://www.thirteen.org
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # CT, http://wgbh.org
-        ('wgby.org', 'WGBY (WGBY)'),  # CT, http://www.wgby.org
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # CT, http://www.njtvonline.org/
-        ('ripbs.org', 'Rhode Island PBS (WSBE)'),  # CT, http://www.ripbs.org/home/
-        ('wliw.org', 'WLIW21 (WLIW)'),  # CT, http://www.wliw.org/
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # DE, http://www.mpt.org
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # DE, http://www.weta.org
-        ('whyy.org', 'WHYY (WHYY)'),  # DE, http://www.whyy.org
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # DE, http://www.njtvonline.org/
-        ('wlvt.org', 'PBS 39 (WLVT)'),  # DE, http://www.wlvt.org/
-        ('wliw.org', 'WLIW21 (WLIW)'),  # DE, http://www.wliw.org/
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # DC, http://www.weta.org
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # DC, http://www.mpt.org
-        ('wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # DC, http://www.wvpt.net
-        ('whut.org', 'Howard University Television (WHUT)'),  # DC, http://www.whut.org
-        ('wedu.org', 'WEDU PBS (WEDU)'),  # FL, http://www.wedu.org
-        ('wfsu.org', 'WFSU-TV (WFSU)'),  # FL, http://wfsu.org/
-        ('wgcu.org', 'WGCU Public Media (WGCU)'),  # FL, http://www.wgcu.org/
-        ('wjct.org', 'WJCT Public Broadcasting (WJCT)'),  # FL, http://www.wjct.org
-        ('wpbt2.org', 'WPBT2 (WPBT)'),  # FL, http://www.wpbt2.org
-        ('wsre.org', 'WSRE (WSRE)'),  # FL, http://www.wsre.org
-        ('wucftv.org', 'WUCF TV (WUCF)'),  # FL, http://wucftv.org
-        ('wuft.org', 'WUFT/Channel 5 (WUFT)'),  # FL, http://www.wuft.org
-        ('wxel.org', 'WXEL/Channel 42 (WXEL)'),  # FL, http://www.wxel.org/home/
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # FL, http://aptv.org/
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # FL, http://www.gpb.org/
-        ('wlrn.org', 'WLRN/Channel 17 (WLRN)'),  # FL, http://www.wlrn.org/
-        ('wusf.org', 'WUSF Public Broadcasting (WUSF)'),  # FL, http://wusf.org/
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # FL, http://www.njtvonline.org/
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # GA, http://www.gpb.org/
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # GA, http://aptv.org/
-        ('scetv.org', 'ETV (WRLK)'),  # GA, http://www.scetv.org
-        ('unctv.org', 'UNC-TV (WUNC)'),  # GA, http://www.unctv.org/
-        ('wfsu.org', 'WFSU-TV (WFSU)'),  # GA, http://wfsu.org/
-        ('wjct.org', 'WJCT Public Broadcasting (WJCT)'),  # GA, http://www.wjct.org
-        ('wtcitv.org', 'WTCI (WTCI)'),  # GA, http://www.wtcitv.org
-        ('pba.org', 'WPBA/Channel 30 (WPBA)'),  # GA, http://pba.org/
-        ('pbsguam.org', 'PBS Guam (KGTF)'),  # GU, http://www.pbsguam.org/
-        ('pbshawaii.org', 'PBS Hawaii - Oceanic Cable Channel 10 (KHET)'),  # HI, http://www.pbshawaii.org/
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # ID, http://idahoptv.org
-        ('ksps.org', 'KSPS (KSPS)'),  # ID, http://www.ksps.org/home/
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # ID, http://www.kued.org
-        ('opb.org', 'OPB (KOPB)'),  # ID, http://www.opb.org
-        ('soptv.org', 'SOPTV (KSYS)'),  # ID, http://www.soptv.org
-        ('wyomingpbs.org', 'Wyoming PBS (KCWC)'),  # ID, http://www.wyomingpbs.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # ID, http://www.cpt12.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # ID, http://www.kbyutv.org/
-        ('kwsu.org', 'KWSU/Channel 10 & KTNW/Channel 31 (KWSU)'),  # ID, http://www.kwsu.org
-        ('illinois.edu', 'WILL-TV (WILL)'),  # IL, http://will.illinois.edu/
-        ('wsec.tv', 'Network Knowledge - WSEC/Springfield (WSEC)'),  # IL, http://www.wsec.tv
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # IL, http://www.wsiu.org/
-        ('wttw.com', 'WTTW11 (WTTW)'),  # IL, http://www.wttw.com/
-        ('wtvp.org', 'WTVP & WTVP.org, Public Media for Central Illinois (WTVP)'),  # IL, http://www.wtvp.org/
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # IL, http://www.iptv.org/
-        ('ket.org', 'KET (WKLE)'),  # IL, http://www.ket.org/
-        ('ninenet.org', 'Nine Network (KETC)'),  # IL, http://www.ninenet.org
-        ('wfwa.org', 'PBS39 Fort Wayne (WFWA)'),  # IL, http://wfwa.org/
-        ('wfyi.org', 'WFYI Indianapolis (WFYI)'),  # IL, http://www.wfyi.org
-        ('mptv.org', 'Milwaukee Public Television (WMVS)'),  # IL, http://www.mptv.org
-        ('wnin.org', 'WNIN (WNIN)'),  # IL, http://www.wnin.org/
-        ('wnit.org', 'WNIT Public Television (WNIT)'),  # IL, http://www.wnit.org/
-        ('wpt.org', 'WPT (WPNE)'),  # IL, http://www.wpt.org/
-        ('wvut.org', 'WVUT/Channel 22 (WVUT)'),  # IL, http://wvut.org/
-        ('weiu.net', 'WEIU/Channel 51 (WEIU)'),  # IL, http://www.weiu.net
-        ('wqpt.org', 'WQPT-TV (WQPT)'),  # IL, http://www.wqpt.org
-        ('wycc.org', 'WYCC PBS Chicago (WYCC)'),  # IL, http://www.wycc.org
-        ('lakeshorepublicmedia.org', 'Lakeshore Public Television (WYIN)'),  # IL, http://lakeshorepublicmedia.org/
-        ('wfwa.org', 'PBS39 Fort Wayne (WFWA)'),  # IN, http://wfwa.org/
-        ('wfyi.org', 'WFYI Indianapolis (WFYI)'),  # IN, http://www.wfyi.org
-        ('wipb.org', 'WIPB-TV (WIPB)'),  # IN, http://wipb.org
-        ('wnin.org', 'WNIN (WNIN)'),  # IN, http://www.wnin.org/
-        ('wnit.org', 'WNIT Public Television (WNIT)'),  # IN, http://www.wnit.org/
-        ('indianapublicmedia.org', 'WTIU (WTIU)'),  # IN, http://indianapublicmedia.org/tv/
-        ('wvut.org', 'WVUT/Channel 22 (WVUT)'),  # IN, http://wvut.org/
-        ('cetconnect.org', 'CET  (WCET)'),  # IN, http://www.cetconnect.org
-        ('ket.org', 'KET (WKLE)'),  # IN, http://www.ket.org/
-        ('thinktv.org', 'ThinkTVNetwork (WPTD)'),  # IN, http://www.thinktv.org
-        ('wbgu.org', 'WBGU-TV (WBGU)'),  # IN, http://wbgu.org
-        ('wgvu.org', 'WGVU TV (WGVU)'),  # IN, http://www.wgvu.org/
-        ('illinois.edu', 'WILL-TV (WILL)'),  # IN, http://will.illinois.edu/
-        ('wsec.tv', 'Network Knowledge - WSEC/Springfield (WSEC)'),  # IN, http://www.wsec.tv
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # IN, http://www.wsiu.org/
-        ('wttw.com', 'WTTW11 (WTTW)'),  # IN, http://www.wttw.com/
-        ('lakeshorepublicmedia.org', 'Lakeshore Public Television (WYIN)'),  # IN, http://lakeshorepublicmedia.org/
-        ('weiu.net', 'WEIU/Channel 51 (WEIU)'),  # IN, http://www.weiu.net
-        ('wycc.org', 'WYCC PBS Chicago (WYCC)'),  # IN, http://www.wycc.org
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # IA, http://www.iptv.org/
-        ('netnebraska.org', 'NET1 (KUON)'),  # IA, http://netnebraska.org
-        ('pioneer.org', 'Pioneer Public Television (KWCM)'),  # IA, http://www.pioneer.org
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # IA, http://www.sdpb.org
-        ('tpt.org', 'TPT (KTCA)'),  # IA, http://www.tpt.org
-        ('wsec.tv', 'Network Knowledge - WSEC/Springfield (WSEC)'),  # IA, http://www.wsec.tv
-        ('wpt.org', 'WPT (WPNE)'),  # IA, http://www.wpt.org/
-        ('ksmq.org', 'KSMQ (KSMQ)'),  # IA, http://www.ksmq.org/
-        ('wqpt.org', 'WQPT-TV (WQPT)'),  # IA, http://www.wqpt.org
-        ('kpts.org', 'KPTS/Channel 8 (KPTS)'),  # KS, http://www.kpts.org/
-        ('ktwu.org', 'KTWU/Channel 11 (KTWU)'),  # KS, http://ktwu.org
-        ('shptv.org', 'Smoky Hills Public Television (KOOD)'),  # KS, http://www.shptv.org
-        ('kcpt.org', 'KCPT Kansas City Public Television (KCPT)'),  # KS, http://kcpt.org/
-        ('netnebraska.org', 'NET1 (KUON)'),  # KS, http://netnebraska.org
-        ('oeta.tv', 'OETA (KETA)'),  # KS, http://www.oeta.tv
-        ('optv.org', 'Ozarks Public Television (KOZK)'),  # KS, http://www.optv.org/
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # KS, http://www.rmpbs.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # KS, http://www.cpt12.org/
-        ('ket.org', 'KET (WKLE)'),  # KY, http://www.ket.org/
-        ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # KY, http://www.blueridgepbs.org/
-        ('cetconnect.org', 'CET  (WCET)'),  # KY, http://www.cetconnect.org
-        ('easttennesseepbs.org', 'East Tennessee PBS (WSJK)'),  # KY, http://easttennesseepbs.org
-        ('wnpt.org', 'Nashville Public Television (WNPT)'),  # KY, http://www.wnpt.org
-        ('thinktv.org', 'ThinkTVNetwork (WPTD)'),  # KY, http://www.thinktv.org
-        ('wcte.org', 'WCTE-TV (WCTE)'),  # KY, http://www.wcte.org
-        ('wljt.org', 'WLJT, Channel 11 (WLJT)'),  # KY, http://wljt.org/
-        ('wnin.org', 'WNIN (WNIN)'),  # KY, http://www.wnin.org/
-        ('wosu.org', 'WOSU TV (WOSU)'),  # KY, http://wosu.org/
-        ('woub.org', 'WOUB/WOUC (WOUB)'),  # KY, http://woub.org/tv/index.php?section=5
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # KY, http://www.wsiu.org/
-        ('wvpublic.org', 'WVPB (WVPB)'),  # KY, http://wvpublic.org/
-        ('wkyupbs.org', 'WKYU-PBS (WKYU)'),  # KY, http://www.wkyupbs.org
-        ('lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # LA, http://www.lpb.org/
-        ('wyes.org', 'WYES-TV/New Orleans (WYES)'),  # LA, http://www.wyes.org
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # LA, http://www.aetn.org/
-        ('kera.org', 'KERA 13 (KERA)'),  # LA, http://www.kera.org/
-        ('mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # LA, http://www.mpbonline.org
-        ('mpbn.net', 'MPBN (WCBB)'),  # ME, http://www.mpbn.net/
-        ('mountainlake.org', 'Mountain Lake PBS (WCFE)'),  # ME, http://www.mountainlake.org/
-        ('nhptv.org', 'NHPTV (WENH)'),  # ME, http://nhptv.org/
-        ('vpt.org', 'Vermont PBS (WETK)'),  # ME, http://www.vpt.org
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # ME, http://wgbh.org
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # MD, http://www.mpt.org
-        ('thirteen.org', 'Thirteen/WNET New York (WNET)'),  # MD, http://www.thirteen.org
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # MD, http://www.weta.org
-        ('whyy.org', 'WHYY (WHYY)'),  # MD, http://www.whyy.org
-        ('witf.org', 'witf (WITF)'),  # MD, http://www.witf.org
-        ('wqed.org', 'WQED Multimedia (WQED)'),  # MD, http://www.wqed.org/
-        ('wvpublic.org', 'WVPB (WVPB)'),  # MD, http://wvpublic.org/
-        ('wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # MD, http://www.wvpt.net
-        ('whut.org', 'Howard University Television (WHUT)'),  # MD, http://www.whut.org
-        ('wliw.org', 'WLIW21 (WLIW)'),  # MD, http://www.wliw.org/
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # MA, http://wgbh.org
-        ('wgby.org', 'WGBY (WGBY)'),  # MA, http://www.wgby.org
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # MA, http://cptv.org
-        ('nhptv.org', 'NHPTV (WENH)'),  # MA, http://nhptv.org/
-        ('vpt.org', 'Vermont PBS (WETK)'),  # MA, http://www.vpt.org
-        ('wmht.org', 'WMHT Educational Telecommunications (WMHT)'),  # MA, http://www.wmht.org/home/
-        ('ripbs.org', 'Rhode Island PBS (WSBE)'),  # MA, http://www.ripbs.org/home/
-        ('deltabroadcasting.org', 'Q-TV (WDCQ)'),  # MI, http://www.deltabroadcasting.org
-        ('dptv.org', 'WTVS Detroit Public TV (WTVS)'),  # MI, http://www.dptv.org/
-        ('wcmu.org', 'CMU Public Television (WCMU)'),  # MI, http://www.wcmu.org
-        ('wgvu.org', 'WGVU TV (WGVU)'),  # MI, http://www.wgvu.org/
-        ('wkar.org', 'WKAR-TV (WKAR)'),  # MI, http://wkar.org/
-        ('nmu.edu', 'WNMU-TV Public TV 13 (WNMU)'),  # MI, http://wnmutv.nmu.edu
-        ('wdse.org', 'WDSE - WRPT (WDSE)'),  # MI, http://www.wdse.org/
-        ('wbgu.org', 'WBGU-TV (WBGU)'),  # MI, http://wbgu.org
-        ('wgte.org', 'WGTE TV (WGTE)'),  # MI, http://www.wgte.org
-        ('wnit.org', 'WNIT Public Television (WNIT)'),  # MI, http://www.wnit.org/
-        ('wpt.org', 'WPT (WPNE)'),  # MI, http://www.wpt.org/
-        ('wttw.com', 'WTTW11 (WTTW)'),  # MI, http://www.wttw.com/
-        ('wycc.org', 'WYCC PBS Chicago (WYCC)'),  # MI, http://www.wycc.org
-        ('lakelandptv.org', 'Lakeland Public Television (KAWE)'),  # MN, http://www.lakelandptv.org
-        ('wdse.org', 'WDSE - WRPT (WDSE)'),  # MN, http://www.wdse.org/
-        ('pioneer.org', 'Pioneer Public Television (KWCM)'),  # MN, http://www.pioneer.org
-        ('tpt.org', 'TPT (KTCA)'),  # MN, http://www.tpt.org
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # MN, http://www.iptv.org/
-        ('prairiepublic.org', 'PRAIRIE PUBLIC (KFME)'),  # MN, http://www.prairiepublic.org/
-        ('ninenet.org', 'Nine Network (KETC)'),  # MN, http://www.ninenet.org
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # MN, http://www.sdpb.org
-        ('wpt.org', 'WPT (WPNE)'),  # MN, http://www.wpt.org/
-        ('ksmq.org', 'KSMQ (KSMQ)'),  # MN, http://www.ksmq.org/
-        ('mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # MS, http://www.mpbonline.org
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # MS, http://www.aetn.org/
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # MS, http://aptv.org/
-        ('wkno.org', 'WKNO/Channel 10 (WKNO)'),  # MS, http://www.wkno.org/
-        ('lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # MS, http://www.lpb.org/
-        ('wsre.org', 'WSRE (WSRE)'),  # MS, http://www.wsre.org
-        ('wyes.org', 'WYES-TV/New Orleans (WYES)'),  # MS, http://www.wyes.org
-        ('kcpt.org', 'KCPT Kansas City Public Television (KCPT)'),  # MO, http://kcpt.org/
-        ('kmos.org', 'KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS)'),  # MO, http://www.kmos.org/
-        ('ninenet.org', 'Nine Network (KETC)'),  # MO, http://www.ninenet.org
-        ('optv.org', 'Ozarks Public Television (KOZK)'),  # MO, http://www.optv.org/
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # MO, http://www.aetn.org/
-        ('dptv.org', 'WTVS Detroit Public TV (WTVS)'),  # MO, http://www.dptv.org/
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # MO, http://www.iptv.org/
-        ('ket.org', 'KET (WKLE)'),  # MO, http://www.ket.org/
-        ('wkno.org', 'WKNO/Channel 10 (WKNO)'),  # MO, http://www.wkno.org/
-        ('ktwu.org', 'KTWU/Channel 11 (KTWU)'),  # MO, http://ktwu.org
-        ('netnebraska.org', 'NET1 (KUON)'),  # MO, http://netnebraska.org
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # MO, http://www.rmpbs.org
-        ('wsec.tv', 'Network Knowledge - WSEC/Springfield (WSEC)'),  # MO, http://www.wsec.tv
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # MO, http://www.wsiu.org/
-        ('montanapbs.org', 'MontanaPBS (KUSM)'),  # MT, http://montanapbs.org
-        ('azpbs.org', 'Arizona PBS (KAET)'),  # MT, http://www.azpbs.org
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # MT, http://idahoptv.org
-        ('prairiepublic.org', 'PRAIRIE PUBLIC (KFME)'),  # MT, http://www.prairiepublic.org/
-        ('ksps.org', 'KSPS (KSPS)'),  # MT, http://www.ksps.org/home/
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # MT, http://www.rmpbs.org
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # MT, http://www.sdpb.org
-        ('netnebraska.org', 'NET1 (KUON)'),  # NE, http://netnebraska.org
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # NE, http://www.iptv.org/
-        ('ktwu.org', 'KTWU/Channel 11 (KTWU)'),  # NE, http://ktwu.org
-        ('pioneer.org', 'Pioneer Public Television (KWCM)'),  # NE, http://www.pioneer.org
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # NE, http://www.rmpbs.org
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # NE, http://www.sdpb.org
-        ('wyomingpbs.org', 'Wyoming PBS (KCWC)'),  # NE, http://www.wyomingpbs.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # NE, http://www.cpt12.org/
-        ('knpb.org', 'KNPB Channel 5 (KNPB)'),  # NV, http://www.knpb.org/
-        ('vegaspbs.org', 'Vegas PBS (KLVX)'),  # NV, http://vegaspbs.org/
-        ('azpbs.org', 'Arizona PBS (KAET)'),  # NV, http://www.azpbs.org
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # NV, http://www.kued.org
-        ('pbssocal.org', 'PBS SoCal/KOCE (KOCE)'),  # NV, http://www.pbssocal.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # NV, http://www.kbyutv.org/
-        ('nhptv.org', 'NHPTV (WENH)'),  # NH, http://nhptv.org/
-        ('scetv.org', 'ETV (WRLK)'),  # NH, http://www.scetv.org
-        ('mountainlake.org', 'Mountain Lake PBS (WCFE)'),  # NH, http://www.mountainlake.org/
-        ('mpbn.net', 'MPBN (WCBB)'),  # NH, http://www.mpbn.net/
-        ('vpt.org', 'Vermont PBS (WETK)'),  # NH, http://www.vpt.org
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # NH, http://wgbh.org
-        ('wgby.org', 'WGBY (WGBY)'),  # NH, http://www.wgby.org
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # NJ, http://cptv.org
-        ('thirteen.org', 'Thirteen/WNET New York (WNET)'),  # NJ, http://www.thirteen.org
-        ('whyy.org', 'WHYY (WHYY)'),  # NJ, http://www.whyy.org
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # NJ, http://www.njtvonline.org/
-        ('wlvt.org', 'PBS 39 (WLVT)'),  # NJ, http://www.wlvt.org/
-        ('wliw.org', 'WLIW21 (WLIW)'),  # NJ, http://www.wliw.org/
-        ('kenw.org', 'KENW-TV3 (KENW)'),  # NM, http://www.kenw.org
-        ('krwg.org', 'KRWG/Channel 22 (KRWG)'),  # NM, http://www.krwg.org
-        ('newmexicopbs.org', 'KNME-TV/Channel 5 (KNME)'),  # NM, http://www.newmexicopbs.org/
-        ('azpm.org', 'KUAT 6 (KUAT)'),  # NM, http://www.azpm.org/
-        ('panhandlepbs.org', 'KACV (KACV)'),  # NM, http://www.panhandlepbs.org/home/
-        ('kcostv.org', 'KCOS/Channel 13 (KCOS)'),  # NM, www.kcostv.org
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # NM, http://www.kued.org
-        ('oeta.tv', 'OETA (KETA)'),  # NM, http://www.oeta.tv
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # NM, http://www.rmpbs.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # NM, http://www.cpt12.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # NM, http://www.kbyutv.org/
-        ('mountainlake.org', 'Mountain Lake PBS (WCFE)'),  # NY, http://www.mountainlake.org/
-        ('thirteen.org', 'Thirteen/WNET New York (WNET)'),  # NY, http://www.thirteen.org
-        ('wcny.org', 'WCNY/Channel 24 (WCNY)'),  # NY, http://www.wcny.org
-        ('wmht.org', 'WMHT Educational Telecommunications (WMHT)'),  # NY, http://www.wmht.org/home/
-        ('wned.org', 'WNED (WNED)'),  # NY, http://www.wned.org/
-        ('wpbstv.org', 'WPBS (WPBS)'),  # NY, http://www.wpbstv.org
-        ('wskg.org', 'WSKG Public TV (WSKG)'),  # NY, http://wskg.org
-        ('wxxi.org', 'WXXI (WXXI)'),  # NY, http://wxxi.org
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # NY, http://cptv.org
-        ('nhptv.org', 'NHPTV (WENH)'),  # NY, http://nhptv.org/
-        ('vpt.org', 'Vermont PBS (WETK)'),  # NY, http://www.vpt.org
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # NY, http://wgbh.org
-        ('wpsu.org', 'WPSU (WPSU)'),  # NY, http://www.wpsu.org
-        ('wqln.org', 'WQLN/Channel 54 (WQLN)'),  # NY, http://www.wqln.org
-        ('wvia.org', 'WVIA Public Media Studios (WVIA)'),  # NY, http://www.wvia.org/
-        ('wliw.org', 'WLIW21 (WLIW)'),  # NY, http://www.wliw.org/
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # NY, http://www.njtvonline.org/
-        ('unctv.org', 'UNC-TV (WUNC)'),  # NC, http://www.unctv.org/
-        ('wtvi.org', 'WTVI (WTVI)'),  # NC, http://www.wtvi.org/
-        ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # NC, http://www.blueridgepbs.org/
-        ('scetv.org', 'ETV (WRLK)'),  # NC, http://www.scetv.org
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # NC, http://www.gpb.org/
-        ('whro.org', 'WHRO (WHRO)'),  # NC, http://whro.org
-        ('wtcitv.org', 'WTCI (WTCI)'),  # NC, http://www.wtcitv.org
-        ('pba.org', 'WPBA/Channel 30 (WPBA)'),  # NC, http://pba.org/
-        ('prairiepublic.org', 'PRAIRIE PUBLIC (KFME)'),  # ND, http://www.prairiepublic.org/
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # ND, http://www.sdpb.org
-        ('cetconnect.org', 'CET  (WCET)'),  # OH, http://www.cetconnect.org
-        ('thinktv.org', 'ThinkTVNetwork (WPTD)'),  # OH, http://www.thinktv.org
-        ('wbgu.org', 'WBGU-TV (WBGU)'),  # OH, http://wbgu.org
-        ('wgte.org', 'WGTE TV (WGTE)'),  # OH, http://www.wgte.org
-        ('WesternReservePublicMedia.org', 'Western Reserve PBS (WNEO)'),  # OH, http://www.WesternReservePublicMedia.org/
-        ('wosu.org', 'WOSU TV (WOSU)'),  # OH, http://wosu.org/
-        ('woub.org', 'WOUB/WOUC (WOUB)'),  # OH, http://woub.org/tv/index.php?section=5
-        ('wviz.org', 'WVIZ/PBS ideastream (WVIZ)'),  # OH, http://www.wviz.org/
-        ('dptv.org', 'WTVS Detroit Public TV (WTVS)'),  # OH, http://www.dptv.org/
-        ('ket.org', 'KET (WKLE)'),  # OH, http://www.ket.org/
-        ('wfwa.org', 'PBS39 Fort Wayne (WFWA)'),  # OH, http://wfwa.org/
-        ('wipb.org', 'WIPB-TV (WIPB)'),  # OH, http://wipb.org
-        ('wqed.org', 'WQED Multimedia (WQED)'),  # OH, http://www.wqed.org/
-        ('wqln.org', 'WQLN/Channel 54 (WQLN)'),  # OH, http://www.wqln.org
-        ('wvpublic.org', 'WVPB (WVPB)'),  # OH, http://wvpublic.org/
-        ('oeta.tv', 'OETA (KETA)'),  # OK, http://www.oeta.tv
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # OK, http://www.aetn.org/
-        ('panhandlepbs.org', 'KACV (KACV)'),  # OK, http://www.panhandlepbs.org/home/
-        ('kera.org', 'KERA 13 (KERA)'),  # OK, http://www.kera.org/
-        ('lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # OK, http://www.lpb.org/
-        ('optv.org', 'Ozarks Public Television (KOZK)'),  # OK, http://www.optv.org/
-        ('opb.org', 'OPB (KOPB)'),  # OR, http://www.opb.org
-        ('soptv.org', 'SOPTV (KSYS)'),  # OR, http://www.soptv.org
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # OR, http://idahoptv.org
-        ('kcts9.org', 'KCTS 9 (KCTS)'),  # OR, http://kcts9.org/
-        ('keet.org', 'KEET TV (KEET)'),  # OR, http://www.keet.org
-        ('ksps.org', 'KSPS (KSPS)'),  # OR, http://www.ksps.org/home/
-        ('kwsu.org', 'KWSU/Channel 10 & KTNW/Channel 31 (KWSU)'),  # OR, http://www.kwsu.org
-        ('whyy.org', 'WHYY (WHYY)'),  # PA, http://www.whyy.org
-        ('witf.org', 'witf (WITF)'),  # PA, http://www.witf.org
-        ('wpsu.org', 'WPSU (WPSU)'),  # PA, http://www.wpsu.org
-        ('wqed.org', 'WQED Multimedia (WQED)'),  # PA, http://www.wqed.org/
-        ('wqln.org', 'WQLN/Channel 54 (WQLN)'),  # PA, http://www.wqln.org
-        ('wvia.org', 'WVIA Public Media Studios (WVIA)'),  # PA, http://www.wvia.org/
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # PA, http://cptv.org
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # PA, http://www.mpt.org
-        ('thirteen.org', 'Thirteen/WNET New York (WNET)'),  # PA, http://www.thirteen.org
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # PA, http://www.weta.org
-        ('wned.org', 'WNED (WNED)'),  # PA, http://www.wned.org/
-        ('WesternReservePublicMedia.org', 'Western Reserve PBS (WNEO)'),  # PA, http://www.WesternReservePublicMedia.org/
-        ('wskg.org', 'WSKG Public TV (WSKG)'),  # PA, http://wskg.org
-        ('wviz.org', 'WVIZ/PBS ideastream (WVIZ)'),  # PA, http://www.wviz.org/
-        ('wvpublic.org', 'WVPB (WVPB)'),  # PA, http://wvpublic.org/
-        ('wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # PA, http://www.wvpt.net
-        ('wlvt.org', 'PBS 39 (WLVT)'),  # PA, http://www.wlvt.org/
-        ('njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # PA, http://www.njtvonline.org/
-        ('whut.org', 'Howard University Television (WHUT)'),  # PA, http://www.whut.org
-        ('wliw.org', 'WLIW21 (WLIW)'),  # PA, http://www.wliw.org/
-        ('cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # RI, http://cptv.org
-        ('nhptv.org', 'NHPTV (WENH)'),  # RI, http://nhptv.org/
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # RI, http://wgbh.org
-        ('ripbs.org', 'Rhode Island PBS (WSBE)'),  # RI, http://www.ripbs.org/home/
-        ('scetv.org', 'ETV (WRLK)'),  # SC, http://www.scetv.org
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # SC, http://www.gpb.org/
-        ('unctv.org', 'UNC-TV (WUNC)'),  # SC, http://www.unctv.org/
-        ('wtvi.org', 'WTVI (WTVI)'),  # SC, http://www.wtvi.org/
-        ('pba.org', 'WPBA/Channel 30 (WPBA)'),  # SC, http://pba.org/
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # SD, http://www.sdpb.org
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # SD, http://www.iptv.org/
-        ('prairiepublic.org', 'PRAIRIE PUBLIC (KFME)'),  # SD, http://www.prairiepublic.org/
-        ('netnebraska.org', 'NET1 (KUON)'),  # SD, http://netnebraska.org
-        ('pioneer.org', 'Pioneer Public Television (KWCM)'),  # SD, http://www.pioneer.org
-        ('easttennesseepbs.org', 'East Tennessee PBS (WSJK)'),  # TN, http://easttennesseepbs.org
-        ('wkno.org', 'WKNO/Channel 10 (WKNO)'),  # TN, http://www.wkno.org/
-        ('wnpt.org', 'Nashville Public Television (WNPT)'),  # TN, http://www.wnpt.org
-        ('wcte.org', 'WCTE-TV (WCTE)'),  # TN, http://www.wcte.org
-        ('wljt.org', 'WLJT, Channel 11 (WLJT)'),  # TN, http://wljt.org/
-        ('wtcitv.org', 'WTCI (WTCI)'),  # TN, http://www.wtcitv.org
-        ('aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # TN, http://aptv.org/
-        ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # TN, http://www.blueridgepbs.org/
-        ('gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # TN, http://www.gpb.org/
-        ('lakelandptv.org', 'Lakeland Public Television (KAWE)'),  # TN, http://www.lakelandptv.org
-        ('ket.org', 'KET (WKLE)'),  # TN, http://www.ket.org/
-        ('mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # TN, http://www.mpbonline.org
-        ('unctv.org', 'UNC-TV (WUNC)'),  # TN, http://www.unctv.org/
-        ('WesternReservePublicMedia.org', 'Western Reserve PBS (WNEO)'),  # TN, http://www.WesternReservePublicMedia.org/
-        ('wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # TN, http://www.wsiu.org/
-        ('wviz.org', 'WVIZ/PBS ideastream (WVIZ)'),  # TN, http://www.wviz.org/
-        ('wkyupbs.org', 'WKYU-PBS (WKYU)'),  # TN, http://www.wkyupbs.org
-        ('pba.org', 'WPBA/Channel 30 (WPBA)'),  # TN, http://pba.org/
-        ('basinpbs.org', 'Basin PBS (KPBT)'),  # TX, http://www.basinpbs.org
-        ('houstonpublicmedia.org', 'KUHT / Channel 8 (KUHT)'),  # TX, http://www.houstonpublicmedia.org/
-        ('panhandlepbs.org', 'KACV (KACV)'),  # TX, http://www.panhandlepbs.org/home/
-        ('tamu.edu', 'KAMU - TV (KAMU)'),  # TX, http://KAMU.tamu.edu
-        ('kcostv.org', 'KCOS/Channel 13 (KCOS)'),  # TX, www.kcostv.org
-        ('kedt.org', 'KEDT/Channel 16 (KEDT)'),  # TX, http://www.kedt.org
-        ('kera.org', 'KERA 13 (KERA)'),  # TX, http://www.kera.org/
-        ('klrn.org', 'KLRN (KLRN)'),  # TX, http://www.klrn.org
-        ('klru.org', 'KLRU (KLRU)'),  # TX, http://www.klru.org
-        ('kmbh.org', 'KMBH-TV (KMBH)'),  # TX, http://www.kmbh.org
-        ('knct.org', 'KNCT (KNCT)'),  # TX, http://www.knct.org
-        ('ktxt.org', 'KTTZ-TV (KTXT)'),  # TX, http://www.ktxt.org
-        ('aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # TX, http://www.aetn.org/
-        ('kenw.org', 'KENW-TV3 (KENW)'),  # TX, http://www.kenw.org
-        ('krwg.org', 'KRWG/Channel 22 (KRWG)'),  # TX, http://www.krwg.org
-        ('lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # TX, http://www.lpb.org/
-        ('netnebraska.org', 'NET1 (KUON)'),  # TX, http://netnebraska.org
-        ('newmexicopbs.org', 'KNME-TV/Channel 5 (KNME)'),  # TX, http://www.newmexicopbs.org/
-        ('wnpt.org', 'Nashville Public Television (WNPT)'),  # TX, http://www.wnpt.org
-        ('oeta.tv', 'OETA (KETA)'),  # TX, http://www.oeta.tv
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # TX, http://www.rmpbs.org
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # UT, http://www.kued.org
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # UT, http://idahoptv.org
-        ('vegaspbs.org', 'Vegas PBS (KLVX)'),  # UT, http://vegaspbs.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # UT, http://www.kbyutv.org/
-        ('vpt.org', 'Vermont PBS (WETK)'),  # VT, http://www.vpt.org
-        ('mountainlake.org', 'Mountain Lake PBS (WCFE)'),  # VT, http://www.mountainlake.org/
-        ('nhptv.org', 'NHPTV (WENH)'),  # VT, http://nhptv.org/
-        ('wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # VT, http://wgbh.org
-        ('wgby.org', 'WGBY (WGBY)'),  # VT, http://www.wgby.org
-        ('wmht.org', 'WMHT Educational Telecommunications (WMHT)'),  # VT, http://www.wmht.org/home/
-        ('wtjx.org', 'WTJX Channel 12 (WTJX)'),  # VI, http://www.wtjx.org/
-        ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # VA, http://www.blueridgepbs.org/
-        ('ideastations.org', 'WCVE PBS (WCVE)'),  # VA, http://ideastations.org/
-        ('whro.org', 'WHRO (WHRO)'),  # VA, http://whro.org
-        ('wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # VA, http://www.wvpt.net
-        ('easttennesseepbs.org', 'East Tennessee PBS (WSJK)'),  # VA, http://easttennesseepbs.org
-        ('ket.org', 'KET (WKLE)'),  # VA, http://www.ket.org/
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # VA, http://www.mpt.org
-        ('unctv.org', 'UNC-TV (WUNC)'),  # VA, http://www.unctv.org/
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # VA, http://www.weta.org
-        ('whut.org', 'Howard University Television (WHUT)'),  # VA, http://www.whut.org
-        ('kcts9.org', 'KCTS 9 (KCTS)'),  # WA, http://kcts9.org/
-        ('ksps.org', 'KSPS (KSPS)'),  # WA, http://www.ksps.org/home/
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # WA, http://idahoptv.org
-        ('opb.org', 'OPB (KOPB)'),  # WA, http://www.opb.org
-        ('kbtc.org', 'KBTC Public Television (KBTC)'),  # WA, http://kbtc.org
-        ('kwsu.org', 'KWSU/Channel 10 & KTNW/Channel 31 (KWSU)'),  # WA, http://www.kwsu.org
-        ('wvpublic.org', 'WVPB (WVPB)'),  # WV, http://wvpublic.org/
-        ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # WV, http://www.blueridgepbs.org/
-        ('ket.org', 'KET (WKLE)'),  # WV, http://www.ket.org/
-        ('mpt.org', 'mpt/Maryland Public Television (WMPB)'),  # WV, http://www.mpt.org
-        ('weta.org', 'WETA Television and Radio (WETA)'),  # WV, http://www.weta.org
-        ('WesternReservePublicMedia.org', 'Western Reserve PBS (WNEO)'),  # WV, http://www.WesternReservePublicMedia.org/
-        ('wosu.org', 'WOSU TV (WOSU)'),  # WV, http://wosu.org/
-        ('woub.org', 'WOUB/WOUC (WOUB)'),  # WV, http://woub.org/tv/index.php?section=5
-        ('wqed.org', 'WQED Multimedia (WQED)'),  # WV, http://www.wqed.org/
-        ('wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # WV, http://www.wvpt.net
-        ('whut.org', 'Howard University Television (WHUT)'),  # WV, http://www.whut.org
-        ('mptv.org', 'Milwaukee Public Television (WMVS)'),  # WI, http://www.mptv.org
-        ('wpt.org', 'WPT (WPNE)'),  # WI, http://www.wpt.org/
-        ('iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # WI, http://www.iptv.org/
-        ('lakelandptv.org', 'Lakeland Public Television (KAWE)'),  # WI, http://www.lakelandptv.org
-        ('wdse.org', 'WDSE - WRPT (WDSE)'),  # WI, http://www.wdse.org/
-        ('tpt.org', 'TPT (KTCA)'),  # WI, http://www.tpt.org
-        ('nmu.edu', 'WNMU-TV Public TV 13 (WNMU)'),  # WI, http://wnmutv.nmu.edu
-        ('wttw.com', 'WTTW11 (WTTW)'),  # WI, http://www.wttw.com/
-        ('ksmq.org', 'KSMQ (KSMQ)'),  # WI, http://www.ksmq.org/
-        ('wycc.org', 'WYCC PBS Chicago (WYCC)'),  # WI, http://www.wycc.org
-        ('wyomingpbs.org', 'Wyoming PBS (KCWC)'),  # WY, http://www.wyomingpbs.org
-        ('idahoptv.org', 'Idaho Public Television (KAID)'),  # WY, http://idahoptv.org
-        ('kued.org', 'KUED Channel 7 (KUED)'),  # WY, http://www.kued.org
-        ('montanapbs.org', 'MontanaPBS (KUSM)'),  # WY, http://montanapbs.org
-        ('netnebraska.org', 'NET1 (KUON)'),  # WY, http://netnebraska.org
-        ('rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # WY, http://www.rmpbs.org
-        ('sdpb.org', 'SDPB Television (KUSD)'),  # WY, http://www.sdpb.org
-        ('cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # WY, http://www.cpt12.org/
-        ('kbyutv.org', 'KBYU-TV (KBYU)'),  # WY, http://www.kbyutv.org/
+        ('video.pbs.org', 'PBS: Public Broadcasting Service'),  # http://www.pbs.org/
+        ('video.aptv.org', 'APT - Alabama Public Television (WBIQ)'),  # http://aptv.org/
+        ('video.gpb.org', 'GPB/Georgia Public Broadcasting (WGTV)'),  # http://www.gpb.org/
+        ('video.mpbonline.org', 'Mississippi Public Broadcasting (WMPN)'),  # http://www.mpbonline.org
+        ('video.wnpt.org', 'Nashville Public Television (WNPT)'),  # http://www.wnpt.org
+        ('video.wfsu.org', 'WFSU-TV (WFSU)'),  # http://wfsu.org/
+        ('video.wsre.org', 'WSRE (WSRE)'),  # http://www.wsre.org
+        ('video.wtcitv.org', 'WTCI (WTCI)'),  # http://www.wtcitv.org
+        ('video.pba.org', 'WPBA/Channel 30 (WPBA)'),  # http://pba.org/
+        ('video.alaskapublic.org', 'Alaska Public Media (KAKM)'),  # http://alaskapublic.org/kakm
+        # ('kuac.org', 'KUAC (KUAC)'),  # http://kuac.org/kuac-tv/
+        # ('ktoo.org', '360 North (KTOO)'),  # http://www.ktoo.org/
+        # ('azpm.org', 'KUAT 6 (KUAT)'),  # http://www.azpm.org/
+        ('video.azpbs.org', 'Arizona PBS (KAET)'),  # http://www.azpbs.org
+        ('portal.knme.org', 'KNME-TV/Channel 5 (KNME)'),  # http://www.newmexicopbs.org/
+        ('video.vegaspbs.org', 'Vegas PBS (KLVX)'),  # http://vegaspbs.org/
+        ('watch.aetn.org', 'AETN/ARKANSAS ETV NETWORK (KETS)'),  # http://www.aetn.org/
+        ('video.ket.org', 'KET (WKLE)'),  # http://www.ket.org/
+        ('video.wkno.org', 'WKNO/Channel 10 (WKNO)'),  # http://www.wkno.org/
+        ('video.lpb.org', 'LPB/LOUISIANA PUBLIC BROADCASTING (WLPB)'),  # http://www.lpb.org/
+        ('videos.oeta.tv', 'OETA (KETA)'),  # http://www.oeta.tv
+        ('video.optv.org', 'Ozarks Public Television (KOZK)'),  # http://www.optv.org/
+        ('watch.wsiu.org', 'WSIU Public Broadcasting (WSIU)'),  # http://www.wsiu.org/
+        ('video.keet.org', 'KEET TV (KEET)'),  # http://www.keet.org
+        ('pbs.kixe.org', 'KIXE/Channel 9 (KIXE)'),  # http://kixe.org/
+        ('video.kpbs.org', 'KPBS San Diego (KPBS)'),  # http://www.kpbs.org/
+        ('video.kqed.org', 'KQED (KQED)'),  # http://www.kqed.org
+        ('vids.kvie.org', 'KVIE Public Television (KVIE)'),  # http://www.kvie.org
+        ('video.pbssocal.org', 'PBS SoCal/KOCE (KOCE)'),  # http://www.pbssocal.org/
+        ('video.valleypbs.org', 'ValleyPBS (KVPT)'),  # http://www.valleypbs.org/
+        ('video.cptv.org', 'CONNECTICUT PUBLIC TELEVISION (WEDH)'),  # http://cptv.org
+        ('watch.knpb.org', 'KNPB Channel 5 (KNPB)'),  # http://www.knpb.org/
+        ('video.soptv.org', 'SOPTV (KSYS)'),  # http://www.soptv.org
+        # ('klcs.org', 'KLCS/Channel 58 (KLCS)'),  # http://www.klcs.org
+        # ('krcb.org', 'KRCB Television & Radio (KRCB)'),  # http://www.krcb.org
+        # ('kvcr.org', 'KVCR TV/DT/FM :: Vision for the Future (KVCR)'),  # http://kvcr.org
+        ('video.rmpbs.org', 'Rocky Mountain PBS (KRMA)'),  # http://www.rmpbs.org
+        ('video.kenw.org', 'KENW-TV3 (KENW)'),  # http://www.kenw.org
+        ('video.kued.org', 'KUED Channel 7 (KUED)'),  # http://www.kued.org
+        ('video.wyomingpbs.org', 'Wyoming PBS (KCWC)'),  # http://www.wyomingpbs.org
+        ('video.cpt12.org', 'Colorado Public Television / KBDI 12 (KBDI)'),  # http://www.cpt12.org/
+        ('video.kbyueleven.org', 'KBYU-TV (KBYU)'),  # http://www.kbyutv.org/
+        ('video.thirteen.org', 'Thirteen/WNET New York (WNET)'),  # http://www.thirteen.org
+        ('video.wgbh.org', 'WGBH/Channel 2 (WGBH)'),  # http://wgbh.org
+        ('video.wgby.org', 'WGBY (WGBY)'),  # http://www.wgby.org
+        ('watch.njtvonline.org', 'NJTV Public Media NJ (WNJT)'),  # http://www.njtvonline.org/
+        # ('ripbs.org', 'Rhode Island PBS (WSBE)'),  # http://www.ripbs.org/home/
+        ('watch.wliw.org', 'WLIW21 (WLIW)'),  # http://www.wliw.org/
+        ('video.mpt.tv', 'mpt/Maryland Public Television (WMPB)'),  # http://www.mpt.org
+        ('watch.weta.org', 'WETA Television and Radio (WETA)'),  # http://www.weta.org
+        ('video.whyy.org', 'WHYY (WHYY)'),  # http://www.whyy.org
+        ('video.wlvt.org', 'PBS 39 (WLVT)'),  # http://www.wlvt.org/
+        ('video.wvpt.net', 'WVPT - Your Source for PBS and More! (WVPT)'),  # http://www.wvpt.net
+        ('video.whut.org', 'Howard University Television (WHUT)'),  # http://www.whut.org
+        ('video.wedu.org', 'WEDU PBS (WEDU)'),  # http://www.wedu.org
+        ('video.wgcu.org', 'WGCU Public Media (WGCU)'),  # http://www.wgcu.org/
+        # ('wjct.org', 'WJCT Public Broadcasting (WJCT)'),  # http://www.wjct.org
+        ('video.wpbt2.org', 'WPBT2 (WPBT)'),  # http://www.wpbt2.org
+        ('video.wucftv.org', 'WUCF TV (WUCF)'),  # http://wucftv.org
+        ('video.wuft.org', 'WUFT/Channel 5 (WUFT)'),  # http://www.wuft.org
+        ('watch.wxel.org', 'WXEL/Channel 42 (WXEL)'),  # http://www.wxel.org/home/
+        ('video.wlrn.org', 'WLRN/Channel 17 (WLRN)'),  # http://www.wlrn.org/
+        ('video.wusf.usf.edu', 'WUSF Public Broadcasting (WUSF)'),  # http://wusf.org/
+        ('video.scetv.org', 'ETV (WRLK)'),  # http://www.scetv.org
+        ('video.unctv.org', 'UNC-TV (WUNC)'),  # http://www.unctv.org/
+        # ('pbsguam.org', 'PBS Guam (KGTF)'),  # http://www.pbsguam.org/
+        ('video.pbshawaii.org', 'PBS Hawaii - Oceanic Cable Channel 10 (KHET)'),  # http://www.pbshawaii.org/
+        ('video.idahoptv.org', 'Idaho Public Television (KAID)'),  # http://idahoptv.org
+        ('video.ksps.org', 'KSPS (KSPS)'),  # http://www.ksps.org/home/
+        ('watch.opb.org', 'OPB (KOPB)'),  # http://www.opb.org
+        ('watch.nwptv.org', 'KWSU/Channel 10 & KTNW/Channel 31 (KWSU)'),  # http://www.kwsu.org
+        ('video.will.illinois.edu', 'WILL-TV (WILL)'),  # http://will.illinois.edu/
+        ('video.networkknowledge.tv', 'Network Knowledge - WSEC/Springfield (WSEC)'),  # http://www.wsec.tv
+        ('video.wttw.com', 'WTTW11 (WTTW)'),  # http://www.wttw.com/
+        # ('wtvp.org', 'WTVP & WTVP.org, Public Media for Central Illinois (WTVP)'),  # http://www.wtvp.org/
+        ('video.iptv.org', 'Iowa Public Television/IPTV (KDIN)'),  # http://www.iptv.org/
+        ('video.ninenet.org', 'Nine Network (KETC)'),  # http://www.ninenet.org
+        ('video.wfwa.org', 'PBS39 Fort Wayne (WFWA)'),  # http://wfwa.org/
+        ('video.wfyi.org', 'WFYI Indianapolis (WFYI)'),  # http://www.wfyi.org
+        ('video.mptv.org', 'Milwaukee Public Television (WMVS)'),  # http://www.mptv.org
+        ('video.wnin.org', 'WNIN (WNIN)'),  # http://www.wnin.org/
+        ('video.wnit.org', 'WNIT Public Television (WNIT)'),  # http://www.wnit.org/
+        ('video.wpt.org', 'WPT (WPNE)'),  # http://www.wpt.org/
+        ('video.wvut.org', 'WVUT/Channel 22 (WVUT)'),  # http://wvut.org/
+        ('video.weiu.net', 'WEIU/Channel 51 (WEIU)'),  # http://www.weiu.net
+        ('video.wqpt.org', 'WQPT-TV (WQPT)'),  # http://www.wqpt.org
+        ('video.wycc.org', 'WYCC PBS Chicago (WYCC)'),  # http://www.wycc.org
+        # ('lakeshorepublicmedia.org', 'Lakeshore Public Television (WYIN)'),  # http://lakeshorepublicmedia.org/
+        ('video.wipb.org', 'WIPB-TV (WIPB)'),  # http://wipb.org
+        ('video.indianapublicmedia.org', 'WTIU (WTIU)'),  # http://indianapublicmedia.org/tv/
+        ('watch.cetconnect.org', 'CET  (WCET)'),  # http://www.cetconnect.org
+        ('video.thinktv.org', 'ThinkTVNetwork (WPTD)'),  # http://www.thinktv.org
+        ('video.wbgu.org', 'WBGU-TV (WBGU)'),  # http://wbgu.org
+        ('video.wgvu.org', 'WGVU TV (WGVU)'),  # http://www.wgvu.org/
+        ('video.netnebraska.org', 'NET1 (KUON)'),  # http://netnebraska.org
+        ('video.pioneer.org', 'Pioneer Public Television (KWCM)'),  # http://www.pioneer.org
+        ('watch.sdpb.org', 'SDPB Television (KUSD)'),  # http://www.sdpb.org
+        ('video.tpt.org', 'TPT (KTCA)'),  # http://www.tpt.org
+        ('watch.ksmq.org', 'KSMQ (KSMQ)'),  # http://www.ksmq.org/
+        ('watch.kpts.org', 'KPTS/Channel 8 (KPTS)'),  # http://www.kpts.org/
+        ('watch.ktwu.org', 'KTWU/Channel 11 (KTWU)'),  # http://ktwu.org
+        # ('shptv.org', 'Smoky Hills Public Television (KOOD)'),  # http://www.shptv.org
+        # ('kcpt.org', 'KCPT Kansas City Public Television (KCPT)'),  # http://kcpt.org/
+        # ('blueridgepbs.org', 'Blue Ridge PBS (WBRA)'),  # http://www.blueridgepbs.org/
+        ('watch.easttennesseepbs.org', 'East Tennessee PBS (WSJK)'),  # http://easttennesseepbs.org
+        ('video.wcte.tv', 'WCTE-TV (WCTE)'),  # http://www.wcte.org
+        ('video.wljt.org', 'WLJT, Channel 11 (WLJT)'),  # http://wljt.org/
+        ('video.wosu.org', 'WOSU TV (WOSU)'),  # http://wosu.org/
+        ('video.woub.org', 'WOUB/WOUC (WOUB)'),  # http://woub.org/tv/index.php?section=5
+        ('video.wvpublic.org', 'WVPB (WVPB)'),  # http://wvpublic.org/
+        ('video.wkyupbs.org', 'WKYU-PBS (WKYU)'),  # http://www.wkyupbs.org
+        # ('wyes.org', 'WYES-TV/New Orleans (WYES)'),  # http://www.wyes.org
+        ('video.kera.org', 'KERA 13 (KERA)'),  # http://www.kera.org/
+        ('video.mpbn.net', 'MPBN (WCBB)'),  # http://www.mpbn.net/
+        ('video.mountainlake.org', 'Mountain Lake PBS (WCFE)'),  # http://www.mountainlake.org/
+        ('video.nhptv.org', 'NHPTV (WENH)'),  # http://nhptv.org/
+        ('video.vpt.org', 'Vermont PBS (WETK)'),  # http://www.vpt.org
+        ('video.witf.org', 'witf (WITF)'),  # http://www.witf.org
+        ('watch.wqed.org', 'WQED Multimedia (WQED)'),  # http://www.wqed.org/
+        ('video.wmht.org', 'WMHT Educational Telecommunications (WMHT)'),  # http://www.wmht.org/home/
+        ('video.deltabroadcasting.org', 'Q-TV (WDCQ)'),  # http://www.deltabroadcasting.org
+        ('video.dptv.org', 'WTVS Detroit Public TV (WTVS)'),  # http://www.dptv.org/
+        ('video.wcmu.org', 'CMU Public Television (WCMU)'),  # http://www.wcmu.org
+        ('video.wkar.org', 'WKAR-TV (WKAR)'),  # http://wkar.org/
+        ('wnmuvideo.nmu.edu', 'WNMU-TV Public TV 13 (WNMU)'),  # http://wnmutv.nmu.edu
+        ('video.wdse.org', 'WDSE - WRPT (WDSE)'),  # http://www.wdse.org/
+        ('video.wgte.org', 'WGTE TV (WGTE)'),  # http://www.wgte.org
+        ('video.lptv.org', 'Lakeland Public Television (KAWE)'),  # http://www.lakelandptv.org
+        # ('prairiepublic.org', 'PRAIRIE PUBLIC (KFME)'),  # http://www.prairiepublic.org/
+        ('video.kmos.org', 'KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS)'),  # http://www.kmos.org/
+        ('watch.montanapbs.org', 'MontanaPBS (KUSM)'),  # http://montanapbs.org
+        ('video.krwg.org', 'KRWG/Channel 22 (KRWG)'),  # http://www.krwg.org
+        ('video.kacvtv.org', 'KACV (KACV)'),  # http://www.panhandlepbs.org/home/
+        ('video.kcostv.org', 'KCOS/Channel 13 (KCOS)'),  # www.kcostv.org
+        ('video.wcny.org', 'WCNY/Channel 24 (WCNY)'),  # http://www.wcny.org
+        ('video.wned.org', 'WNED (WNED)'),  # http://www.wned.org/
+        ('watch.wpbstv.org', 'WPBS (WPBS)'),  # http://www.wpbstv.org
+        ('video.wskg.org', 'WSKG Public TV (WSKG)'),  # http://wskg.org
+        ('video.wxxi.org', 'WXXI (WXXI)'),  # http://wxxi.org
+        ('video.wpsu.org', 'WPSU (WPSU)'),  # http://www.wpsu.org
+        # ('wqln.org', 'WQLN/Channel 54 (WQLN)'),  # http://www.wqln.org
+        ('on-demand.wvia.org', 'WVIA Public Media Studios (WVIA)'),  # http://www.wvia.org/
+        ('video.wtvi.org', 'WTVI (WTVI)'),  # http://www.wtvi.org/
+        # ('whro.org', 'WHRO (WHRO)'),  # http://whro.org
+        ('video.westernreservepublicmedia.org', 'Western Reserve PBS (WNEO)'),  # http://www.WesternReservePublicMedia.org/
+        ('video.ideastream.org', 'WVIZ/PBS ideastream (WVIZ)'),  # http://www.wviz.org/
+        ('video.kcts9.org', 'KCTS 9 (KCTS)'),  # http://kcts9.org/
+        ('video.basinpbs.org', 'Basin PBS (KPBT)'),  # http://www.basinpbs.org
+        ('video.houstonpbs.org', 'KUHT / Channel 8 (KUHT)'),  # http://www.houstonpublicmedia.org/
+        # ('tamu.edu', 'KAMU - TV (KAMU)'),  # http://KAMU.tamu.edu
+        # ('kedt.org', 'KEDT/Channel 16 (KEDT)'),  # http://www.kedt.org
+        ('video.klrn.org', 'KLRN (KLRN)'),  # http://www.klrn.org
+        ('video.klru.tv', 'KLRU (KLRU)'),  # http://www.klru.org
+        # ('kmbh.org', 'KMBH-TV (KMBH)'),  # http://www.kmbh.org
+        # ('knct.org', 'KNCT (KNCT)'),  # http://www.knct.org
+        # ('ktxt.org', 'KTTZ-TV (KTXT)'),  # http://www.ktxt.org
+        ('video.wtjx.org', 'WTJX Channel 12 (WTJX)'),  # http://www.wtjx.org/
+        ('video.ideastations.org', 'WCVE PBS (WCVE)'),  # http://ideastations.org/
+        ('video.kbtc.org', 'KBTC Public Television (KBTC)'),  # http://kbtc.org
     )
 
     IE_NAME = 'pbs'
@@ -491,13 +183,13 @@ class PBSIE(InfoExtractor):
     _VALID_URL = r'''(?x)https?://
         (?:
            # Direct video URL
-           (?:videos?|watch)\.(?:%s)/(?:viralplayer|video)/(?P<id>[0-9]+)/? |
+           (?:%s)/(?:viralplayer|video)/(?P<id>[0-9]+)/? |
            # Article with embedded player (or direct video)
            (?:www\.)?pbs\.org/(?:[^/]+/){2,5}(?P<presumptive_id>[^/]+?)(?:\.html)?/?(?:$|[?\#]) |
            # Player
            (?:video|player)\.pbs\.org/(?:widget/)?partnerplayer/(?P<player_id>[^/]+)/
         )
-    ''' % '|'.join(list(zip(*_STATIONS))[0])
+    ''' % '|'.join(re.escape(p) for p in list(zip(*_STATIONS))[0])
 
     _TESTS = [
         {

youtube_dl/extractor/safari.py

@@ -6,12 +6,12 @@ import re
 from .common import InfoExtractor
 from .brightcove import BrightcoveLegacyIE
 
-from ..compat import compat_urllib_parse
 from ..utils import (
     ExtractorError,
     sanitized_Request,
     smuggle_url,
     std_headers,
+    urlencode_postdata,
 )
 
 
@@ -57,7 +57,7 @@ class SafariBaseIE(InfoExtractor):
         }
 
         request = sanitized_Request(
-            self._LOGIN_URL, compat_urllib_parse.urlencode(login_form), headers=headers)
+            self._LOGIN_URL, urlencode_postdata(login_form), headers=headers)
         login_page = self._download_webpage(
             request, None, 'Logging in as %s' % username)
 

youtube_dl/extractor/vevo.py

@@ -3,7 +3,10 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_etree_fromstring
+from ..compat import (
+    compat_etree_fromstring,
+    compat_urlparse,
+)
 from ..utils import (
     ExtractorError,
     int_or_none,
@@ -67,6 +70,17 @@ class VevoIE(InfoExtractor):
         'params': {
             'skip_download': 'true',
         }
+    }, {
+        'note': 'No video_info',
+        'url': 'http://www.vevo.com/watch/k-camp-1/Till-I-Die/USUV71503000',
+        'md5': '8b83cc492d72fc9cf74a02acee7dc1b0',
+        'info_dict': {
+            'id': 'USUV71503000',
+            'ext': 'mp4',
+            'title': 'Till I Die - K Camp ft. T.I.',
+            'duration': 193,
+        },
+        'expected_warnings': ['Unable to download SMIL file'],
     }]
     _SMIL_BASE_URL = 'http://smil.lvl3.vevo.com/'
 
@@ -81,11 +95,17 @@ class VevoIE(InfoExtractor):
         if webpage is False:
             self._oauth_token = None
         else:
+            if 'THIS PAGE IS CURRENTLY UNAVAILABLE IN YOUR REGION' in webpage:
+                raise ExtractorError('%s said: This page is currently unavailable in your region.' % self.IE_NAME, expected=True)
+
             self._oauth_token = self._search_regex(
                 r'access_token":\s*"([^"]+)"',
                 webpage, 'access token', fatal=False)
 
     def _formats_from_json(self, video_info):
+        if not video_info:
+            return []
+
         last_version = {'version': -1}
         for version in video_info['videoVersions']:
             # These are the HTTP downloads, other types are for different manifests
@@ -110,9 +130,8 @@ class VevoIE(InfoExtractor):
             })
         return formats
 
-    def _formats_from_smil(self, smil_xml):
+    def _formats_from_smil(self, smil_doc):
         formats = []
-        smil_doc = compat_etree_fromstring(smil_xml.encode('utf-8'))
         els = smil_doc.findall('.//{http://www.w3.org/2001/SMIL20/Language}video')
         for el in els:
             src = el.attrib['src']
@@ -145,14 +164,14 @@ class VevoIE(InfoExtractor):
             })
         return formats
 
-    def _download_api_formats(self, video_id):
+    def _download_api_formats(self, video_id, video_url):
         if not self._oauth_token:
             self._downloader.report_warning(
                 'No oauth token available, skipping API HLS download')
             return []
 
-        api_url = 'https://apiv2.vevo.com/video/%s/streams/hls?token=%s' % (
-            video_id, self._oauth_token)
+        api_url = compat_urlparse.urljoin(video_url, '//apiv2.vevo.com/video/%s/streams/hls?token=%s' % (
+            video_id, self._oauth_token))
         api_data = self._download_json(
             api_url, video_id,
             note='Downloading HLS formats',
@@ -166,18 +185,26 @@ class VevoIE(InfoExtractor):
             preference=0)
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
+        video_id = self._match_id(url)
+
+        webpage = None
 
         json_url = 'http://videoplayer.vevo.com/VideoService/AuthenticateVideo?isrc=%s' % video_id
         response = self._download_json(json_url, video_id)
-        video_info = response['video']
+        video_info = response['video'] or {}
 
-        if not video_info:
+        if not video_info and response.get('statusCode') != 909:
             if 'statusMessage' in response:
                 raise ExtractorError('%s said: %s' % (self.IE_NAME, response['statusMessage']), expected=True)
             raise ExtractorError('Unable to extract videos')
 
+        if not video_info:
+            if url.startswith('vevo:'):
+                raise ExtractorError('Please specify full Vevo URL for downloading', expected=True)
+            webpage = self._download_webpage(url, video_id)
+
+        title = video_info.get('title') or self._og_search_title(webpage)
+
         formats = self._formats_from_json(video_info)
 
         is_explicit = video_info.get('isExplicit')
@@ -189,11 +216,11 @@ class VevoIE(InfoExtractor):
             age_limit = None
 
         # Download via HLS API
-        formats.extend(self._download_api_formats(video_id))
+        formats.extend(self._download_api_formats(video_id, url))
 
         # Download SMIL
         smil_blocks = sorted((
-            f for f in video_info['videoVersions']
+            f for f in video_info.get('videoVersions', [])
             if f['sourceType'] == 13),
             key=lambda f: f['version'])
         smil_url = '%s/Video/V2/VFILE/%s/%sr.smil' % (
@@ -205,23 +232,26 @@ class VevoIE(InfoExtractor):
             if smil_url_m is not None:
                 smil_url = smil_url_m
         if smil_url:
-            smil_xml = self._download_webpage(
-                smil_url, video_id, 'Downloading SMIL info', fatal=False)
-            if smil_xml:
-                formats.extend(self._formats_from_smil(smil_xml))
+            smil_doc = self._download_smil(smil_url, video_id, fatal=False)
+            if smil_doc:
+                formats.extend(self._formats_from_smil(smil_doc))
 
         self._sort_formats(formats)
-        timestamp_ms = int_or_none(self._search_regex(
+        timestamp = int_or_none(self._search_regex(
             r'/Date\((\d+)\)/',
-            video_info['launchDate'], 'launch date', fatal=False))
+            video_info['launchDate'], 'launch date', fatal=False),
+            scale=1000) if video_info else None
+
+        duration = video_info.get('duration') or int_or_none(
+            self._html_search_meta('video:duration', webpage))
 
         return {
             'id': video_id,
-            'title': video_info['title'],
+            'title': title,
             'formats': formats,
-            'thumbnail': video_info['imageUrl'],
-            'timestamp': timestamp_ms // 1000,
-            'uploader': video_info['mainArtists'][0]['artistName'],
-            'duration': video_info['duration'],
+            'thumbnail': video_info.get('imageUrl'),
+            'timestamp': timestamp,
+            'uploader': video_info['mainArtists'][0]['artistName'] if video_info else None,
+            'duration': duration,
             'age_limit': age_limit,
         }

youtube_dl/extractor/wdr.py

@@ -10,8 +10,8 @@ from ..compat import (
     compat_urlparse,
 )
 from ..utils import (
-    determine_ext,
     unified_strdate,
+    qualities,
 )
 
 
@@ -33,6 +33,7 @@ class WDRIE(InfoExtractor):
             'params': {
                 'skip_download': True,
             },
+            'skip': 'Page Not Found',
         },
         {
             'url': 'http://www1.wdr.de/themen/av/videomargaspiegelisttot101-videoplayer.html',
@@ -47,6 +48,7 @@ class WDRIE(InfoExtractor):
             'params': {
                 'skip_download': True,
             },
+            'skip': 'Page Not Found',
         },
         {
             'url': 'http://www1.wdr.de/themen/kultur/audioerlebtegeschichtenmargaspiegel100-audioplayer.html',
@@ -71,6 +73,7 @@ class WDRIE(InfoExtractor):
                 'upload_date': '20140717',
                 'is_live': False
             },
+            'skip': 'Page Not Found',
         },
         {
             'url': 'http://www1.wdr.de/mediathek/video/sendungen/quarks_und_co/filterseite-quarks-und-co100.html',
@@ -83,10 +86,10 @@ class WDRIE(InfoExtractor):
             'url': 'http://www1.wdr.de/mediathek/video/livestream/index.html',
             'info_dict': {
                 'id': 'mdb-103364',
-                'title': 're:^WDR Fernsehen [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
+                'title': 're:^WDR Fernsehen Live [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
                 'description': 'md5:ae2ff888510623bf8d4b115f95a9b7c9',
                 'ext': 'flv',
-                'upload_date': '20150212',
+                'upload_date': '20150101',
                 'is_live': True
             },
             'params': {
@@ -150,25 +153,52 @@ class WDRIE(InfoExtractor):
         if upload_date:
             upload_date = unified_strdate(upload_date)
 
+        formats = []
+        preference = qualities(['S', 'M', 'L', 'XL'])
+
         if video_url.endswith('.f4m'):
-            video_url += '?hdcore=3.2.0&plugin=aasp-3.2.0.77.18'
-            ext = 'flv'
+            f4m_formats = self._extract_f4m_formats(video_url + '?hdcore=3.2.0&plugin=aasp-3.2.0.77.18', page_id, f4m_id='hds', fatal=False)
+            if f4m_formats:
+                formats.extend(f4m_formats)
         elif video_url.endswith('.smil'):
-            fmt = self._extract_smil_formats(video_url, page_id)[0]
-            video_url = fmt['url']
-            sep = '&' if '?' in video_url else '?'
-            video_url += sep
-            video_url += 'hdcore=3.3.0&plugin=aasp-3.3.0.99.43'
-            ext = fmt['ext']
+            smil_formats = self._extract_smil_formats(video_url, page_id, False, {
+                'hdcore': '3.3.0',
+                'plugin': 'aasp-3.3.0.99.43',
+            })
+            if smil_formats:
+                formats.extend(smil_formats)
         else:
-            ext = determine_ext(video_url)
+            formats.append({
+                'url': video_url,
+                'http_headers': {
+                    'User-Agent': 'mobile',
+                },
+            })
+
+        m3u8_url = self._search_regex(r'rel="adaptiv"[^>]+href="([^"]+)"', webpage, 'm3u8 url', default=None)
+        if m3u8_url:
+            m3u8_formats = self._extract_m3u8_formats(m3u8_url, page_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False)
+            if m3u8_formats:
+                formats.extend(m3u8_formats)
+
+        direct_urls = re.findall(r'rel="web(S|M|L|XL)"[^>]+href="([^"]+)"', webpage)
+        if direct_urls:
+            for quality, video_url in direct_urls:
+                formats.append({
+                    'url': video_url,
+                    'preference': preference(quality),
+                    'http_headers': {
+                        'User-Agent': 'mobile',
+                    },
+                })
+
+        self._sort_formats(formats)
 
         description = self._html_search_meta('Description', webpage, 'description')
 
         return {
             'id': page_id,
-            'url': video_url,
-            'ext': ext,
+            'formats': formats,
             'title': title,
             'description': description,
             'thumbnail': thumbnail,

youtube_dl/extractor/youku.py

@@ -25,8 +25,8 @@ class YoukuIE(InfoExtractor):
     '''
 
     _TESTS = [{
+        # MD5 is unstable
         'url': 'http://v.youku.com/v_show/id_XMTc1ODE5Njcy.html',
-        'md5': '5f3af4192eabacc4501508d54a8cabd7',
         'info_dict': {
             'id': 'XMTc1ODE5Njcy_part1',
             'title': '★Smile﹗♡ Git Fresh -Booty Music舞蹈.',
@@ -42,6 +42,7 @@ class YoukuIE(InfoExtractor):
             'title': '武媚娘传奇 85',
         },
         'playlist_count': 11,
+        'skip': 'Available in China only',
     }, {
         'url': 'http://v.youku.com/v_show/id_XMTI1OTczNDM5Mg==.html',
         'info_dict': {
@@ -49,7 +50,6 @@ class YoukuIE(InfoExtractor):
             'title': '花千骨 04',
         },
         'playlist_count': 13,
-        'skip': 'Available in China only',
     }, {
         'url': 'http://v.youku.com/v_show/id_XNjA1NzA2Njgw.html',
         'note': 'Video protected with password',
@@ -63,7 +63,7 @@ class YoukuIE(InfoExtractor):
         },
     }]
 
-    def construct_video_urls(self, data1, data2):
+    def construct_video_urls(self, data):
         # get sid, token
         def yk_t(s1, s2):
             ls = list(range(256))
@@ -81,34 +81,24 @@ class YoukuIE(InfoExtractor):
             return bytes(s)
 
         sid, token = yk_t(
-            b'becaf9be', base64.b64decode(data2['ep'].encode('ascii'))
+            b'becaf9be', base64.b64decode(data['security']['encrypt_string'].encode('ascii'))
         ).decode('ascii').split('_')
 
         # get oip
-        oip = data2['ip']
-
-        # get fileid
-        string_ls = list(
-            'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ/\:._-1234567890')
-        shuffled_string_ls = []
-        seed = data1['seed']
-        N = len(string_ls)
-        for ii in range(N):
-            seed = (seed * 0xd3 + 0x754f) % 0x10000
-            idx = seed * len(string_ls) // 0x10000
-            shuffled_string_ls.append(string_ls[idx])
-            del string_ls[idx]
+        oip = data['security']['ip']
 
         fileid_dict = {}
-        for format in data1['streamtypes']:
-            streamfileid = [
-                int(i) for i in data1['streamfileids'][format].strip('*').split('*')]
-            fileid = ''.join(
-                [shuffled_string_ls[i] for i in streamfileid])
-            fileid_dict[format] = fileid[:8] + '%s' + fileid[10:]
+        for stream in data['stream']:
+            format = stream.get('stream_type')
+            fileid = stream['stream_fileid']
+            fileid_dict[format] = fileid
 
         def get_fileid(format, n):
-            fileid = fileid_dict[format] % hex(int(n))[2:].upper().zfill(2)
+            number = hex(int(str(n), 10))[2:].upper()
+            if len(number) == 1:
+                number = '0' + number
+            streamfileids = fileid_dict[format]
+            fileid = streamfileids[0:8] + number + streamfileids[10:]
             return fileid
 
         # get ep
@@ -123,15 +113,15 @@ class YoukuIE(InfoExtractor):
 
         # generate video_urls
         video_urls_dict = {}
-        for format in data1['streamtypes']:
+        for stream in data['stream']:
+            format = stream.get('stream_type')
             video_urls = []
-            for dt in data1['segs'][format]:
-                n = str(int(dt['no']))
+            for dt in stream['segs']:
+                n = str(stream['segs'].index(dt))
                 param = {
-                    'K': dt['k'],
+                    'K': dt['key'],
                     'hd': self.get_hd(format),
                     'myp': 0,
-                    'ts': dt['seconds'],
                     'ypp': 0,
                     'ctype': 12,
                     'ev': 1,
@@ -142,7 +132,7 @@ class YoukuIE(InfoExtractor):
                 video_url = \
                     'http://k.youku.com/player/getFlvPath/' + \
                     'sid/' + sid + \
-                    '_' + str(int(n) + 1).zfill(2) + \
+                    '_00' + \
                     '/st/' + self.parse_ext_l(format) + \
                     '/fileid/' + get_fileid(format, n) + '?' + \
                     compat_urllib_parse.urlencode(param)
@@ -153,23 +143,31 @@ class YoukuIE(InfoExtractor):
 
     def get_hd(self, fm):
         hd_id_dict = {
+            '3gp': '0',
+            '3gphd': '1',
             'flv': '0',
+            'flvhd': '0',
             'mp4': '1',
+            'mp4hd': '1',
+            'mp4hd2': '1',
+            'mp4hd3': '1',
             'hd2': '2',
             'hd3': '3',
-            '3gp': '0',
-            '3gphd': '1'
         }
         return hd_id_dict[fm]
 
     def parse_ext_l(self, fm):
         ext_dict = {
+            '3gp': 'flv',
+            '3gphd': 'mp4',
             'flv': 'flv',
+            'flvhd': 'flv',
             'mp4': 'mp4',
+            'mp4hd': 'mp4',
+            'mp4hd2': 'flv',
+            'mp4hd3': 'flv',
             'hd2': 'flv',
             'hd3': 'flv',
-            '3gp': 'flv',
-            '3gphd': 'mp4'
         }
         return ext_dict[fm]
 
@@ -178,9 +176,13 @@ class YoukuIE(InfoExtractor):
             '3gp': 'h6',
             '3gphd': 'h5',
             'flv': 'h4',
+            'flvhd': 'h4',
             'mp4': 'h3',
+            'mp4hd': 'h3',
+            'mp4hd2': 'h4',
+            'mp4hd3': 'h4',
             'hd2': 'h2',
-            'hd3': 'h1'
+            'hd3': 'h1',
         }
         return _dict[fm]
 
@@ -188,45 +190,46 @@ class YoukuIE(InfoExtractor):
         video_id = self._match_id(url)
 
         def retrieve_data(req_url, note):
-            req = sanitized_Request(req_url)
+            headers = {
+                'Referer': req_url,
+            }
+            self._set_cookie('youku.com', 'xreferrer', 'http://www.youku.com')
+            req = sanitized_Request(req_url, headers=headers)
 
             cn_verification_proxy = self._downloader.params.get('cn_verification_proxy')
             if cn_verification_proxy:
                 req.add_header('Ytdl-request-proxy', cn_verification_proxy)
 
             raw_data = self._download_json(req, video_id, note=note)
-            return raw_data['data'][0]
+
+            return raw_data['data']
 
         video_password = self._downloader.params.get('videopassword', None)
 
         # request basic data
-        basic_data_url = 'http://v.youku.com/player/getPlayList/VideoIDS/%s' % video_id
+        basic_data_url = "http://play.youku.com/play/get.json?vid=%s&ct=12" % video_id
         if video_password:
-            basic_data_url += '?password=%s' % video_password
+            basic_data_url += '&pwd=%s' % video_password
 
-        data1 = retrieve_data(
-            basic_data_url,
-            'Downloading JSON metadata 1')
-        data2 = retrieve_data(
-            'http://v.youku.com/player/getPlayList/VideoIDS/%s/Pf/4/ctype/12/ev/1' % video_id,
-            'Downloading JSON metadata 2')
+        data = retrieve_data(basic_data_url, 'Downloading JSON metadata')
 
-        error_code = data1.get('error_code')
-        if error_code:
-            error = data1.get('error')
-            if error is not None and '因版权原因无法观看此视频' in error:
+        error = data.get('error')
+        if error:
+            error_note = error.get('note')
+            if error_note is not None and '因版权原因无法观看此视频' in error_note:
                 raise ExtractorError(
                     'Youku said: Sorry, this video is available in China only', expected=True)
             else:
-                msg = 'Youku server reported error %i' % error_code
+                msg = 'Youku server reported error %i' % error.get('code')
                 if error is not None:
-                    msg += ': ' + error
+                    msg += ': ' + error_note
                 raise ExtractorError(msg)
 
-        title = data1['title']
+        # get video title
+        title = data['video']['title']
 
         # generate video_urls_dict
-        video_urls_dict = self.construct_video_urls(data1, data2)
+        video_urls_dict = self.construct_video_urls(data)
 
         # construct info
         entries = [{
@@ -235,10 +238,11 @@ class YoukuIE(InfoExtractor):
             'formats': [],
             # some formats are not available for all parts, we have to detect
             # which one has all
-        } for i in range(max(len(v) for v in data1['segs'].values()))]
-        for fm in data1['streamtypes']:
+        } for i in range(max(len(v.get('segs')) for v in data['stream']))]
+        for stream in data['stream']:
+            fm = stream.get('stream_type')
             video_urls = video_urls_dict[fm]
-            for video_url, seg, entry in zip(video_urls, data1['segs'][fm], entries):
+            for video_url, seg, entry in zip(video_urls, stream['segs'], entries):
                 entry['formats'].append({
                     'url': video_url,
                     'format_id': self.get_format_name(fm),

youtube_dl/postprocessor/ffmpeg.py

@@ -52,7 +52,7 @@ class FFmpegPostProcessor(PostProcessor):
 
     def _determine_executables(self):
         programs = ['avprobe', 'avconv', 'ffmpeg', 'ffprobe']
-        prefer_ffmpeg = self._downloader.params.get('prefer_ffmpeg', False)
+        prefer_ffmpeg = False
 
         self.basename = None
         self.probe_basename = None
@@ -60,6 +60,7 @@ class FFmpegPostProcessor(PostProcessor):
         self._paths = None
         self._versions = None
         if self._downloader:
+            prefer_ffmpeg = self._downloader.params.get('prefer_ffmpeg', False)
             location = self._downloader.params.get('ffmpeg_location')
             if location is not None:
                 if not os.path.exists(location):

youtube_dl/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2015.12.09'
+__version__ = '2015.12.13'