Compare commits

..

1614 Commits

Author SHA1 Message Date
3a4cca687f release 2015.02.02.3 2015-02-02 22:56:15 +01:00
7d3d06a16c [vevo] Restore SMIL support (#3656) 2015-02-02 22:48:12 +01:00
c21b1fbeeb release 2015.02.02.2 2015-02-02 21:58:58 +01:00
f920ce295e [ntvru] Remove unused import 2015-02-02 21:58:17 +01:00
7a7bd19c45 [n-tv.de] Use native m3u8 as best format 2015-02-02 21:57:48 +01:00
8f4b58d70e [ntvde] Add new extractor (Fixes #4850) 2015-02-02 21:48:54 +01:00
3fd45e03bf [ntvru] Rename from NTV to clarify the difference between n-tv.de and ntv.ru 2015-02-02 20:43:02 +01:00
869b4aeff4 release 2015.02.02.1 2015-02-02 20:35:04 +01:00
cc9ca3ba6e [downloader/external] Simplify source_address
'' might actually be passed in, so check for None.
2015-02-02 20:33:25 +01:00
ea71034bd3 Merge remote-tracking branch 'origin/master'
Conflicts:
	youtube_dl/downloader/external.py
2015-02-02 20:32:07 +01:00
9fffd0469f [options] Mark --fixup as non-experimental and correct its help 2015-02-02 20:28:18 +01:00
ae7773942e [downloader/external] Simplify 2015-02-02 21:51:38 +06:00
469a64cebf [downloader/external] Simplify 2015-02-02 21:40:52 +06:00
aae3fdcfae Merge pull request #4845 from vijayanandnandam/master
Passing source address option to external downloaders
2015-02-02 21:38:22 +06:00
6a66904f8e passing source address option to external downloaders 2015-02-02 20:51:40 +05:30
78271e3319 [drtv] Extract material id (Closes #4814) 2015-02-02 21:11:25 +06:00
92bf0bcdf8 [historicfilms] Add extractor (Closes #4825) 2015-02-02 20:52:37 +06:00
1283204917 [http] PEP8 (#4831) 2015-02-02 12:05:39 +01:00
6789defea9 Merge pull request #4831 from light94/master
Handling Connection Reset by Peer Error
2015-02-02 12:03:28 +01:00
acf2a6e97b release 2015.02.02 2015-02-02 01:49:40 +01:00
8cfb6efe6f [jsinterp] Correct div command 2015-02-02 01:49:32 +01:00
04edb9caf5 Merge pull request #4838 from raunaqrox/patch-1
supported sites link was not opening from README
2015-02-01 23:32:44 +01:00
044131ba21 supported sites was not opening
required a .md at the end.
2015-02-02 03:49:28 +05:30
0a7055c90d release 2015.02.01 2015-02-01 22:42:19 +01:00
9e3f19919a [jsinterp] Beef up and add tests
In preparation for #4822, extend jsinterp by a lot. (We may even have to/want to write a proper interpreter with actual parsing)
2015-02-01 22:38:29 +01:00
4a3da4ebdb [hls] Simplify 2015-02-01 23:54:38 +06:00
027008b14e [hls] Fix encode issues on python2 @ Windows 2015-02-01 23:49:23 +06:00
c6df692466 [rutv] Extract all m3u8 formats 2015-02-01 23:48:23 +06:00
acf757f42e [youtube] Complete test definition 2015-02-01 15:33:32 +01:00
dd8982f19c [wdr] Complete test definition 2015-02-01 15:27:16 +01:00
654bd52f58 [teletask] Fix test definition 2015-02-01 15:25:33 +01:00
a9551e9020 [soundcloud:set] Fix test definition 2015-02-01 15:24:38 +01:00
4e980275b5 [test_download] Remove references to "file" in test definitions 2015-02-01 15:21:18 +01:00
c172440ac5 [ringtv] Modernize test definition 2015-02-01 15:21:02 +01:00
e332772531 [servingsys] Modernize 2015-02-01 15:18:52 +01:00
437cac8cc1 [sina] Modernize and simplify 2015-02-01 15:16:35 +01:00
9f281cacd2 [keezmovies] Fix extraction and modernize test 2015-02-01 15:13:44 +01:00
748a0fab8a Remove unused imports 2015-02-01 15:08:50 +01:00
c1f06d6307 [macgamestore] Modernize 2015-02-01 15:08:33 +01:00
c4e817ce4a [france2.fr:generation-quoi] Modernize 2015-02-01 15:06:55 +01:00
9a3e5e6955 [kankan] Modernize 2015-02-01 15:03:55 +01:00
228d30ed06 [la7] Modernize 2015-02-01 15:03:03 +01:00
057c0609fc [toutv] Modernize test definition 2015-02-01 15:01:33 +01:00
17d2712d9c [teamcoco] Modernize and fix extraction 2015-02-01 15:00:54 +01:00
fc09240e24 [vimeo] Modernize test definition 2015-02-01 12:12:27 +01:00
146303136f [nerdcubed] Modernize test definition 2015-02-01 12:11:20 +01:00
96aded8d3d [rottentomatoes] Modernize test definition 2015-02-01 12:11:14 +01:00
2886be15aa [defense] Modernize 2015-02-01 12:10:15 +01:00
ca0f500ecf [mtv] Modernize and clean up test 2015-02-01 12:08:21 +01:00
29aef5a33c [ard] Remove deleted video test case 2015-02-01 12:00:47 +01:00
9158b2b301 [mpora] Modernize 2015-02-01 11:58:37 +01:00
0196149c5b [compat] Correct compat_basestring definition 2015-02-01 11:37:00 +01:00
8f9312c387 Appease pyflakes8-3 2015-02-01 11:30:56 +01:00
439b9a9e9b Merge branch 'kinetoskombi-globo-fix' 2015-02-01 04:36:57 +06:00
8c72beb25e [globo] Properly extract m3u8 formats (#4346 #4832) 2015-02-01 04:36:24 +06:00
1ee94db2d0 [globo] Fix error on some globo videos 2015-01-31 20:07:43 -02:00
e77d2975af Handling Connection Reset by Peer Error 2015-02-01 00:10:58 +05:30
e41b1f7385 Fix flake8 errors 2015-01-31 10:51:39 +01:00
cd596028d6 [rtve] Recognize mobile urls (fixes #4823) 2015-01-30 23:46:55 +01:00
cc57bd33a8 [rtve] Fix extraction
Skip live stream test, we can't use the f4m manifest yet
2015-01-30 23:46:55 +01:00
6d593c3276 [YoutubeDL] Fix video+audio format_id (Closes #4824) 2015-01-31 03:50:11 +06:00
91755ee384 [comedycentral:shows] Generate better IDs and add a test for that 2015-01-30 19:43:46 +01:00
0692ef86ef [bbccouk] Improve _VALID_URL 2015-01-30 23:47:09 +06:00
439d9be27d [mixcloud] Remove unused import 2015-01-30 23:21:58 +06:00
b80505a409 [mixcloud] Fix extraction (Closes #4784) 2015-01-30 23:21:44 +06:00
e4c17d7274 [nhl:news] Add extractor (Closes #4805) 2015-01-30 23:12:27 +06:00
2c58674e0e [nhl] Improve _VALID_URL (#4805) 2015-01-30 22:46:53 +06:00
ef1269fb07 [drtv] Improve _VALID_URL (#4814) 2015-01-30 22:42:11 +06:00
e525d9a3df [mtv] Extract subtitles (Closes #4811) 2015-01-30 21:57:59 +06:00
20b4492c71 [spike] Improve _VALID_URL 2015-01-30 21:54:48 +06:00
dee3f73787 [spike] Modernize 2015-01-30 21:54:14 +06:00
d543bdc351 [downloader/f4m] Clarify that we should eventually just implement the DRM scheme (#3000) 2015-01-30 16:06:55 +01:00
c7ff0c6422 Merge remote-tracking branch 'rzhxeo/f4m-drm' 2015-01-30 16:00:47 +01:00
01c46659c4 [washingtonpost] Catch more UUIDs 2015-01-30 15:53:58 +01:00
b04b885271 [extractor/common] Document all protocol values 2015-01-30 15:53:16 +01:00
dc35bfd2d5 [test/helper] Clarify which keys have to be added 2015-01-30 15:52:56 +01:00
70fca8d694 [youtube] Remove unused import 2015-01-30 10:59:19 +01:00
a52c633536 [cinchcast] Wrap overly long lines (#4820) 2015-01-30 10:59:07 +01:00
7b6c60393e Merge remote-tracking branch 'codesparkle/master' 2015-01-30 10:56:53 +01:00
83e7a314b4 dedup AUTHORS 2015-01-30 10:48:39 +01:00
749f2ca044 Smotri info extractor: removed unreachable code and updated old md5 for test video 2015-01-30 20:35:20 +11:00
5468ff4d91 Remove duplicate dictionary keys 2015-01-30 20:11:51 +11:00
1d2daaea63 Simplify list creation 2015-01-30 20:10:12 +11:00
52585fd6dc The opening curly brace { is a regex reserved [control character](http://stackoverflow.com/a/400316/1106367), so it needs to be escaped. 2015-01-30 18:41:40 +11:00
c03844a4ec release 2015.01.30.2 2015-01-30 04:44:00 +01:00
6449cd807e [youtube] Fall back to embed webpage when content is blocked on main (Fixes #4717) 2015-01-30 04:43:50 +01:00
e2a08185c6 [README] Add an FAQ for YouTube IDs starting with a dash (Closes #4800) 2015-01-30 04:17:44 +01:00
5d6677ca28 release 2015.01.30.1 2015-01-30 03:50:40 +01:00
5a8a29cfea release 2015.01.30 2015-01-30 03:46:57 +01:00
c1708b89c0 [youtube:truncated_url] Add hl= forms (#4799) 2015-01-30 03:45:29 +01:00
83fddfd493 [dctp] PEP8 2015-01-30 03:45:06 +01:00
1798791df1 Credit @bastik for dctp.tv (#4796) 2015-01-30 03:16:36 +01:00
6ebb0dca9f [dctp] Improve output (#4796) 2015-01-30 03:15:34 +01:00
cf8d6ec865 Merge remote-tracking branch 'bastik/dev-dctp' 2015-01-30 03:12:53 +01:00
f452f72c6b Credit @yan12125 for ctsnews and nextmedia 2015-01-30 03:12:12 +01:00
3198291f26 [ctsnews] Remove unused import 2015-01-30 03:09:51 +01:00
02c1d5e285 Merge remote-tracking branch 'yan12125/IE_NextMedia' 2015-01-30 03:09:19 +01:00
ec4161a57d [ctsnews] Remove news count check (#4802)
That's the number of articles (or at least some constant value); view count is the number of views of the video!
Also add a better description so that we can distinguish the two downloads.
2015-01-30 03:08:50 +01:00
03d8d4df38 [test_http] PEP8 2015-01-30 03:06:40 +01:00
03d2d6d51b Merge remote-tracking branch 'yan12125/IE_CtsNews' 2015-01-30 03:03:19 +01:00
83fda3c000 Add a test for --no-check-certificate 2015-01-30 02:57:37 +01:00
4fe8495a23 [viddler] PEP8 2015-01-30 02:13:37 +01:00
a16f6643f0 Merge pull request #4815 from jaimeMF/https_handler
[utils] YoutubeDLHTTPSHandler.https_open: pass all required arguments to to do_open
2015-01-30 01:57:46 +01:00
adc0ae3ceb [__init__] Provide a better error messages if URLs are missing (Closes #4813) 2015-01-30 01:56:15 +01:00
7bb3ceb4c7 [dctp] prefix real_time parameter with rtmp_ 2015-01-30 00:47:43 +01:00
75a4fc5b72 [dctp] fix test 2015-01-30 00:35:53 +01:00
87673cd438 [dctp] follow id conventions 2015-01-29 23:34:56 +01:00
f345fe9db7 [dctp] fix python 2.6 compatibility 2015-01-29 23:32:23 +01:00
e683a48d0e [ctsnews] Detect youtube embedde videos 2015-01-30 05:50:47 +08:00
a7a14d9586 [YoutubeDL] set the 'thumbnails' field if the info_dict has the 'thumbnails' field
Since the '--write-thumbnail' uses the 'thumbnails' field and we didn't updated the info_dict, it wouldn't detect the thumbnail. (fixes #4812)
2015-01-29 20:15:38 +01:00
219337990b [xuite] Fix _VALID_URL 2015-01-29 23:11:13 +06:00
376a770cc4 Merge pull request #4809 from bastik/fix-sr
fix srmediathek description
2015-01-29 22:14:15 +06:00
7e500dbd93 Merge branch 'yan12125-IE_Xuite' 2015-01-29 22:10:17 +06:00
affd04a45d [xuite] Simplify and improve 2015-01-29 22:09:59 +06:00
c84130e865 Merge branch 'IE_Xuite' of https://github.com/yan12125/youtube-dl into yan12125-IE_Xuite 2015-01-29 20:21:34 +06:00
4f264c02c7 [utils] YoutubeDLHTTPSHandler.https_open: pass all required arguments to do_open
With this change the '--no-check-certificate' works again (#4807).
2015-01-29 15:01:27 +01:00
d205476103 fix srmediathek description 2015-01-29 01:36:15 +01:00
367cc95aa7 [CtsNews] Add new extractor 2015-01-29 03:49:56 +08:00
206dba27a4 [NextMedia] Add new extractor 2015-01-29 03:18:53 +08:00
dcf53d4408 [YoutubeDL] Set format_id for video+audio (Closes #3634) 2015-01-29 00:44:40 +06:00
63be3b8989 [ivi] Modernize 2015-01-28 23:58:14 +06:00
18b4e9e79d [viddler] Extract comment count 2015-01-28 23:21:17 +06:00
cb454b333d [generic] Improve some regexes 2015-01-28 23:07:37 +06:00
e0d9f85aee Credit @HyShai for viddler fixes (#4794) 2015-01-28 22:56:06 +06:00
b04fbd789c [viddler] Modernize 2015-01-28 22:49:42 +06:00
aad9556414 Merge pull request #4794 from HyShai/viddler
[viddler] fixed viddler support; also added a viddler generic extractor
2015-01-28 22:34:46 +06:00
48a1e5141a added test for dctp 2015-01-28 08:59:58 +01:00
0865f397ae added extractor for dctp.tv 2015-01-28 08:21:04 +01:00
796df3c631 fixed viddler support - needed a Referer header; also added a viddler
generic extractor
2015-01-28 00:08:19 -05:00
a28383834b [xuite] Update tests 2015-01-28 01:30:14 +08:00
3a0d2f520a [YoutubeDL] Temporary fix for subprocess encoding issues on python2 @ Windows (Closes #4787)
For now filenames will be encoded with preferrefencoding before written to disk
2015-01-27 22:38:28 +06:00
6348ad12a0 [xuite] Add height information for the two formats 2015-01-28 00:13:40 +08:00
fe7710cbcc [xuite] Add new extractor 2015-01-27 23:55:22 +08:00
2103d038b3 [lnkgo] Adapt to website changes 2015-01-27 15:38:54 +02:00
6ca85be6f8 Filter DRM protected media in f4m downloader 2015-01-26 20:44:48 +01:00
9f0df77ab1 [YoutubeDL] Allow format filtering by fps 2015-01-26 14:36:22 +01:00
e72c7e4123 [YoutubeDL] Always set the '_filename' field in the info_dict (reported in #4053)
It's also useful when you use the '--write-info-json' option.
2015-01-26 12:01:43 +01:00
2b1bd292ae release 2015.01.25 2015-01-25 21:40:43 +01:00
71e7da6533 Merge branch 'master' of github.com:rg3/youtube-dl 2015-01-25 21:39:50 +01:00
80a49d3d7b Credit @David-Development for rtl2 (#4780) 2015-01-26 02:08:29 +06:00
d862a4f94f [spiegel] Use generalized formats pre-testing 2015-01-26 00:34:31 +06:00
a57e8ce658 [lynda] Pre-test video URLs for HTTP errors (Closes #2185, closes #4782) 2015-01-26 00:33:42 +06:00
96a53167fa [common] Generalize URLs' HTTP errors pre-testing 2015-01-26 00:32:31 +06:00
6d2749aac4 [drtv] Prefer the version without spoken subtitles (fixes #4779)
For example for http://www.dr.dk/tv/se/moderne-klassikere/moderne-klassikere-one-republic-apologize#!/, there's a version where everytime someone speaks in English a computer voice translates it.
2015-01-25 18:56:04 +01:00
b1b0b1ca30 [generic] Improve description testcase in rss test 2015-01-25 18:14:59 +01:00
3dee7826e7 [rtl2] PEP8, simplify, make rtmp tests run (#470) 2015-01-25 18:09:48 +01:00
c9326b38b8 flake8: Ignore .git 2015-01-25 18:09:09 +01:00
d4f64cabf4 Merge remote-tracking branch 'David-Development/rtl2.py' 2015-01-25 17:55:31 +01:00
fe41ddbb28 refactoring - bug fixes 2015-01-25 11:53:53 +01:00
ee69b99af6 [YoutubeDL] clarify hook documentation 2015-01-25 06:15:54 +01:00
767ff0a2d1 Merge branch 'travis-rtmp' 2015-01-25 05:30:47 +01:00
8604e882a8 [ubu] Fix test and modernize 2015-01-25 05:23:21 +01:00
cc1237f484 [__init__] Work around flake8 false positive 2015-01-25 05:17:43 +01:00
37f4ce538a [smotri] Fix test case 2015-01-25 05:17:15 +01:00
7d346331b5 [audiomack:album] Update testcase 2015-01-25 05:15:47 +01:00
e1ccc04e9f Test rtmpdump on travis (Fixes #1601) 2015-01-25 04:56:32 +01:00
881e6a1f5c Add --xattr-set-filesize option (Fixes #1348) 2015-01-25 04:49:44 +01:00
baeaeffce5 [options] Add support for infinite retries (Fixes #507) 2015-01-25 04:34:38 +01:00
c14e88f0f5 [YoutubeDL] Add --playlist-items option (Fixes #2662) 2015-01-25 04:24:55 +01:00
8940b8608e Merge remote-tracking branch 'h-collector/master'
Conflicts:
	youtube_dl/extractor/fc2.py
2015-01-25 03:48:26 +01:00
ec82d85acd [YoutubeDL] Implement --write-all-thumbnails (Closes #2269) 2015-01-25 03:11:12 +01:00
cfb56d1af3 Add --list-thumbnails 2015-01-25 02:43:19 +01:00
1e10802990 [krasview] Fix extraction 2015-01-25 05:21:39 +06:00
6695916045 Merge branch 'rtl2.py' of https://github.com/David-Development/youtube-dl into rtl2.py
Conflicts:
	youtube_dl/extractor/rtl2.py
2015-01-25 00:09:21 +01:00
7906d199a1 [rtl2] Add new extractor 2015-01-25 00:07:15 +01:00
1070711d60 [YoutubeDL._calc_cookies] Restore the 'is_unverifiable'
I should have check everything was copied before commiting 4b405cfc6e.
2015-01-24 20:12:47 +01:00
4b405cfc6e [YoutubeDL._calc_cookies] Restore the 'has_header' method
I didn't copied it from downloader/external
2015-01-24 20:08:24 +01:00
e5660ee6ae [YoutubeDL] Fill the info dict 'http_headers' field with all the headers available
Useful for external tools using the json output.

The methods '_calc_headers' and '_calc_cookies' have been copied from the downloader/external, now they just use "info_dict['http_headers']".
2015-01-24 18:56:04 +01:00
8011fba3ae [rtl2] Add new extractor 2015-01-24 18:28:16 +01:00
587a9c2749 [downloader/external] Use the 'http_headers' field 2015-01-24 18:25:09 +01:00
e1554a407d [extractors] Use http_headers for setting the User-Agent and the Referer 2015-01-24 18:23:53 +01:00
3fcfb8e9fa [utils] YoutubeDLHandler: don't use 'Youtubedl-user-agent' for overriding the default user agent
Setting the 'User-Agent' header is enough
2015-01-24 18:07:21 +01:00
384b62028a [downloader/external] Add curl and aria2c (Closes #182) 2015-01-24 13:33:45 +01:00
b95aab8482 [youtube:truncated_url] Add x-yt-cl URLs (#4773) 2015-01-24 11:42:39 +01:00
fc2d6abfe7 [videott] Improve _VALID_URL and add test 2015-01-24 16:11:40 +06:00
27de5625d4 Merge pull request #4771 from irfancharania/videott
[videott] improve extraction
2015-01-24 16:07:42 +06:00
6aa4f54d66 [videott] improve extraction 2015-01-23 17:41:07 -08:00
222516d97d [downloader] Lay groundwork for external downloaders.
This comes with a very simply implementation for wget; the real work is in setting up the infrastructure.
2015-01-24 01:38:48 +01:00
a055469faf [downloader] Improve downloader selection 2015-01-23 23:50:31 +01:00
fdaaaaa878 README: Recommend using flake8 instead of pyflake and pep8 separately 2015-01-23 21:10:10 +01:00
12d1fb5aa9 [twitch] PEP8 2015-01-23 21:05:07 +01:00
48f00d15b1 [auengine] Remove extractor
The test is probably infringing copyright and nobody has provided a new test (see #4643).
2015-01-23 21:03:00 +01:00
3e055aa5c3 [cliphunter] Fix extraction and update test (Fixes #4362) 2015-01-23 21:23:40 +02:00
6896a52721 release 2015.01.23.4 2015-01-23 18:58:32 +01:00
5779b3e1fe Merge remote-tracking branch 'origin/master' 2015-01-23 18:58:28 +01:00
62cd676c74 [youtube] Fixup DASH m4a headers
This fixes #2288, #2506, #2607, #3681, #4741, #4767.
2015-01-23 18:39:12 +01:00
0c17278843 [atresplayer] Extract subtitles 2015-01-23 22:54:29 +06:00
d229ee70da Merge remote-tracking branch 'origin/master' 2015-01-23 17:22:45 +01:00
26e274666d [liveleak] Add original videos (Fixes #4768) 2015-01-23 17:22:14 +01:00
ebd46aed51 [atresplayer] Filter URLs and clarify android format ids 2015-01-23 22:21:55 +06:00
e793f7671c [liveleak] Modernize 2015-01-23 17:09:26 +01:00
c2e64f71d0 [twitch] Add support for bookmarks 2015-01-23 21:58:40 +06:00
0920e5830f [atresplayer] Don't include f4m formats if they are protected by DRM (fixes #4705) 2015-01-23 16:39:23 +01:00
bf7fa94ec7 [downloader/f4m] build_fragments_list: Support videos with more than 1 segment 2015-01-23 16:31:52 +01:00
6f58db8982 release 2015.01.23.3 2015-01-23 12:17:19 +01:00
aa42e87340 [utils] Catch strange Windows errors (Closes #4733) 2015-01-23 12:17:12 +01:00
649f7966f7 Fix --sleep-interval (#3426) 2015-01-23 12:07:13 +01:00
5f0d813d93 Merge remote-tracking branch 'rupertbaxter2/master'
Conflicts:
	youtube_dl/__init__.py
	youtube_dl/downloader/common.py
2015-01-23 12:05:01 +01:00
501f13fbf3 [generic] Add support for Cinerama player (Fixes #4752) 2015-01-23 12:00:25 +01:00
ba55168157 release 2015.01.23.2 2015-01-23 11:20:24 +01:00
d79323136f [utils] Simplify HTTPS socket creation
We were duplicating (bad) code and doing crazy things with SSL.
Just use TLSv1 across the board, and do with one implementation of HTTPSConnection.connect.
Fixes #4696.
2015-01-23 11:15:18 +01:00
08ff6ab07e [gamestar] Modernize slightly 2015-01-23 01:34:24 +01:00
ba655a0e4c Merge remote-tracking branch 'derrotebaron/master' 2015-01-23 01:32:52 +01:00
b59c17e543 Merge pull request #4745 from BitLooter/master
Embed description and URL as MP4 tags
2015-01-23 01:22:19 +01:00
61ca9a80b3 [generic] Add support for BOMs (Fixes #4753) 2015-01-23 01:21:30 +01:00
317239b097 release 2015.01.23.1 2015-01-23 00:33:14 +01:00
c2a30b250c [testtube] Add new extractor (Fixes #4759) 2015-01-23 00:32:35 +01:00
c994e6bd63 release 2015.01.23 2015-01-23 00:06:11 +01:00
3ee2aa7a16 Merge remote-tracking branch 'origin/master' 2015-01-23 00:06:02 +01:00
083c9df93b [YoutubeDL] Allow filtering by properties (Fixes #4584) 2015-01-23 00:04:05 +01:00
50789175ed [pornhub] Detect private videos and emit an error message (Closes #4764) 2015-01-22 23:48:58 +01:00
dc1b027cd4 [twitch] PEP8 2015-01-22 23:06:03 +01:00
f353cbdb2f [twitch:stream] Randomize query 2015-01-22 23:34:40 +06:00
73e449b226 Merge branch 'master' of github.com:rg3/youtube-dl 2015-01-22 18:21:27 +01:00
b4a64c592b [README] Add an FAQ entry about destination folder 2015-01-22 18:21:17 +01:00
78111136db [twitch] Move URL matching tests into extractor 2015-01-22 18:18:21 +01:00
650ab5beeb [comedycentral:shows] Remove references to colbert report 2015-01-22 18:15:58 +01:00
7932de6352 [hearthisat] Correct error message 2015-01-22 18:15:04 +01:00
240b9b7a5c [twitch] Add support for streams (Closes #893, closes #3693, closes #1884) 2015-01-22 23:11:22 +06:00
bb6e38787d [videomega] Fix extraction (Closes #4763) 2015-01-22 18:36:49 +02:00
898c23c03f release 2015.01.22 2015-01-22 12:04:26 +01:00
b55ee18ff3 [hearthisat] Add support for more high-quality download links 2015-01-22 12:04:13 +01:00
e5763a7a7e [hearthisat] Add new extractor (Closes #4743) 2015-01-21 21:47:55 +02:00
8bb1bdfae9 [twitch:past_broadcasts] Fix IE_NAME 2015-01-21 23:06:16 +06:00
c62b449765 Credit @yan12125 for streetvoice (#4758) 2015-01-21 22:56:28 +06:00
bb0aa4cb3c [streetvoice] Improve 2015-01-21 22:53:51 +06:00
d63528c8c7 Merge pull request #4758 from yan12125/IE_streetvoice
[StreetVoice] Add new extractor
2015-01-21 22:36:50 +06:00
c5db6bb32b [twitch] Refactor and add support for past broadcasts 2015-01-21 22:27:21 +06:00
c8dc41a6e7 [StreetVoice] Add new extractor 2015-01-21 23:05:47 +08:00
47e0e1e0e2 [nbc] Fix pep8 issue 2015-01-21 10:36:15 +01:00
efcddaebe9 [cnn] Use edition.cnn.com for getting the information (fixes #4757)
Some videos (like http://edition.cnn.com/videos/us/2015/01/20/orig-yellowstone-oil-spill.cnn) will fail if we use cnn.com.
2015-01-21 10:31:57 +01:00
5fe5112589 [CNNArticle] Update test 2015-01-21 10:27:18 +01:00
564bb5e964 [tinypic] Tweak VALID_URL regex (Closes #4754) 2015-01-21 02:15:28 +06:00
2df54b4ba8 [nbcnews] Ignore HTTP errors while coping with playlists (Closes #4749) 2015-01-20 21:23:51 +06:00
030aa5d9e7 [tvp] Fix extraction 2015-01-19 23:00:22 +06:00
c511f13f22 [ndtv] Modernize 2015-01-19 10:10:05 +01:00
bd3cbe0716 Embed description and URL as MP4 tags 2015-01-18 15:01:05 -08:00
fdb2ed7455 [abc7news] Add extractor (Closes #4734) 2015-01-18 08:09:18 +06:00
ba319696a9 [options] Clarify that --password can be left out (#4723) 2015-01-17 23:56:34 +01:00
910c552052 release 2015.01.16 2015-01-16 14:20:38 +01:00
cce81f192c [bandcamp:album] Fix title extraction (Fixes #4721) 2015-01-16 14:20:25 +01:00
9d22a7dfb0 [fourtube] Fix extraction 2015-01-16 13:44:44 +01:00
4f4f642822 [npo] Remove unused import 2015-01-16 13:44:36 +01:00
2875cf01bb FFmpegEmbedSubtitlePP: simplify command 2015-01-16 13:37:37 +01:00
e205db3bcd FFmpegEmbedSubtitlePP: don't fail if the video doesn't have an audio stream (fixes #4718)
Instead of specifying which streams ffmpeg must copy, we tell it to copy all.
2015-01-16 13:29:01 +01:00
31d4a6e212 release 2015.01.15.1 2015-01-15 22:38:11 +01:00
aaeb86f682 [youtube] Add test for #4706 2015-01-16 01:25:03 +06:00
9fa6ea2680 Merge pull request #4706 from pkulak/master
Fix Youtube encrypted sigs.
2015-01-16 01:12:50 +06:00
a9b6b5cd15 Looks like Google switched to a new JS compiler that includes dollar signs in function names. 2015-01-15 10:23:05 -08:00
a45c0a5d67 [videomega] Fix extraction (Closes #4703) 2015-01-15 19:57:36 +02:00
c8dfe360eb [atresplayer] Add authentication support (Closes #4700) 2015-01-15 21:43:35 +06:00
4cfaf85c65 release 2015.01.15 2015-01-15 12:42:11 +01:00
be5f2c192c [ssl] Correct connect creation
We want to authenticate the server, see https://docs.python.org/dev/library/ssl.html#ssl.Purpose.SERVER_AUTH .
2015-01-15 02:06:50 +01:00
c9ef44ce29 [smotri] Improve extraction (Closes #4698) 2015-01-14 21:50:36 +06:00
e92d4a11f5 [spiegel] Test format video URLs for 404 (Closes #4579) 2015-01-14 20:27:14 +06:00
f2cbc96c3e [lnkgo] Make more robust 2015-01-14 00:51:48 +02:00
a69801e2c6 [utils] Add additional format to unified_strdate 2015-01-14 00:16:34 +02:00
034206cec1 [lnkgo] Add new extractor 2015-01-14 00:14:59 +02:00
04e0bac233 [npo:live] Add extractor (Closes #4691) 2015-01-13 20:54:03 +06:00
fbef83f399 [README] Add FAQ for playing from another machine (Fixes #4693) 2015-01-13 08:10:17 +01:00
3d5f7a3947 [utils] Prevent override of custom headers.
The dict of headers of request objects in urllib has its keys always
capitalized.

This causes the lookup to fail and overwrite the header. If for example
a Extractor tries to add a "User-Agent" header the internal
representation in the request object is "User-agent". The header is
therefore clobbered by the "User-Agent" in std_headers, because the
strings are not equal.
2015-01-12 22:38:51 +01:00
a5fb718c50 [test_utils] Add more tests for parse_duration 2015-01-12 21:39:58 +06:00
227d4822ff [utils] Disallow non string types in parse_duration (Closes #4679) 2015-01-12 21:06:26 +06:00
5c4a81d934 [npo] Remove unused import 2015-01-11 23:43:09 +01:00
263255eb8d Credit @Josso for drbonanza (#4581) 2015-01-11 23:42:24 +01:00
8e2ec95575 [drbonanza] Simplify and fix duration (#4687) 2015-01-11 23:41:55 +01:00
8e7a9016d5 [DRBonanza] Add new extractor (fixing #4581) 2015-01-11 23:23:10 +01:00
c85f368370 [npo] Make extension check less strict and add test (#4680) 2015-01-11 23:45:52 +06:00
a0977064ce [npo] Fix non asf streams (Closes #4680) 2015-01-11 23:18:45 +06:00
15aecd8711 release 2015.01.11 2015-01-11 17:47:04 +01:00
20dd0b2d20 Merge branch 'master' of github.com:rg3/youtube-dl 2015-01-11 17:46:22 +01:00
f934860a07 Merge pull request #4684 from Josso/patch-1
[drtv] Updated with support for https
2015-01-11 22:41:15 +06:00
2aeb06d6dc [utils] Improve colon handling (Fixes #4683) 2015-01-11 17:40:45 +01:00
6ccbb335d2 [drtv] Updated with support for https 2015-01-11 17:39:16 +01:00
4340decad2 check for overwriting files in the downloader (fixes #3916, closes #3829) 2015-01-11 12:02:27 +01:00
f3ff1a3696 YoutubeDL: Make the decision about removing the original file after each postprocessor is run (fixes #2261)
If one of the processors said the file should be kept, it wouldn't pay
attention to the response from the following processors. This was wrong if the
'keep_video' option was False, if the first extractor modifies the original file
and then we extract its audio we don't want to keep the original video file.
2015-01-11 11:35:18 +01:00
aa24de39aa [veehd] Update test 2015-01-11 16:20:39 +06:00
a798e64c15 [veehd] Improve extraction 2015-01-11 16:20:16 +06:00
6a5fa75490 [karaoketv] Remove unused import 2015-01-11 10:48:20 +01:00
8ad6b5ed9f [compat] Correct socket error class reference 2015-01-11 10:47:39 +01:00
d5bb814d34 [veehd] Capture removed video message 2015-01-11 15:42:53 +06:00
d156a1d981 [xboxclips] Fix extraction 2015-01-11 15:25:29 +06:00
987493aef3 [test_compat] Fix alphabetic order to make test_all_present pass 2015-01-11 15:13:03 +06:00
8bfa75451b [options] Add --no-call-home
While we're at it, also drop "experimental" moniker for --call-home - should work fine.
2015-01-10 21:09:18 +01:00
c071733fd4 [README] Highlight that bug reports should include the -v output 2015-01-10 21:07:44 +01:00
cd3063f3fa release 2015.01.10.2 2015-01-10 21:03:00 +01:00
58b1f00d19 [YoutubeDL] Add new --call-home option for debugging 2015-01-10 21:02:27 +01:00
149f05c7b6 release 2015.01.10.1 2015-01-10 20:06:13 +01:00
8a1b9b068e Merge remote-tracking branch 'origin/master' 2015-01-10 20:06:01 +01:00
c5a59d9391 [utils] Fix call to _create_http_connection
Avoid confusion over args/kwargs.
2015-01-10 20:05:30 +01:00
500b8b41c1 [options] Add -4 and -6 options
Fixes #520, fixes #3626.
2015-01-10 20:02:02 +01:00
be4a824d74 Add new option --source-address
Closes #3618, fixes #721, fixes #2481, fixes #4551, closes #1020.
2015-01-10 19:56:51 +01:00
ed3958d714 [collegerama] Add extractor (#4540) 2015-01-11 00:40:46 +06:00
6ce08764a1 Credit @dinesh for rte.ie (#4015) 2015-01-10 18:58:03 +01:00
c80ede5f13 [karaoketv] Simplify (#3853) 2015-01-10 18:03:36 +01:00
bc694039e4 Merge remote-tracking branch 'lenaten/karaoketv' 2015-01-10 17:59:35 +01:00
3462af03e6 [rte] PEP8 2015-01-10 17:59:07 +01:00
ea1d5bdcdd [rte] Make more robust and add a new testcase (#4015) 2015-01-10 17:57:21 +01:00
121c09c7be Merge remote-tracking branch 'Dineshs91/f4m-2.0' 2015-01-10 17:51:52 +01:00
76bfaf6daf [nrk] Improve subtitle support (#3092) 2015-01-10 17:46:01 +01:00
d89c6e336a [atttechchannel] Add extractor (Closes #3938) 2015-01-10 19:44:29 +06:00
776dc3992a [utils] Clarify more day-month-first ambiguous formats 2015-01-10 19:43:52 +06:00
27ca82ebc6 [orf:oe1] Add konsole URL schema (Fixes #4675) 2015-01-10 14:27:27 +01:00
385f8ae468 [eighttracks] PEP8 2015-01-10 14:25:11 +01:00
b9f030cc26 [orf] Fix typo 2015-01-10 14:23:54 +01:00
52afb2ac1b [ffmpeg] Call encodeFilename on filenames 2015-01-10 06:13:18 +01:00
43bc88903d Merge remote-tracking branch 'ivan/muxed-mtime' 2015-01-10 06:10:18 +01:00
6ef9f88299 release 2015.01.10 2015-01-10 05:51:22 +01:00
f71fdb0acc [eighttracks] Improve waiting (#3954) 2015-01-10 05:51:07 +01:00
c24dfef63c Merge remote-tracking branch 'lenaten/8tracks' 2015-01-10 05:47:05 +01:00
6271f1cad9 [youtube|ffmpeg] Automatically correct video with non-square pixels (Fixes #4674) 2015-01-10 05:45:51 +01:00
fb4b030aaf [tvp] Update tests and improve output 2015-01-10 02:38:35 +01:00
ff21a8e0ee Merge remote-tracking branch 'Tithen-Firion/master' 2015-01-10 02:26:21 +01:00
904fffffeb [audiomack] Better titles, simplify code 2015-01-10 02:24:46 +01:00
51897bb77c Merge remote-tracking branch 'xavierbeynon/master' 2015-01-10 02:03:46 +01:00
bd1a281ede [options] PEP8 and simpler --merge-output-format handling (#4673) 2015-01-10 02:03:00 +01:00
45598f1578 Merge remote-tracking branch 'aft90/merge-output-format'
Conflicts:
	youtube_dl/YoutubeDL.py
2015-01-10 01:59:14 +01:00
d02115f837 Use the option in preparing the merge output filename 2015-01-10 00:29:06 +00:00
34c781a24d Passing the option into the main program's arguments 2015-01-10 00:03:11 +00:00
1302394603 release 2015.01.09.2 2015-01-09 23:59:29 +01:00
dd622d7c4e [netzkino] Add new extractor (Fixes #4669) 2015-01-09 23:59:18 +01:00
d120e9013f Added an option to specify an output format for merges when downloading separate video & audio 2015-01-09 22:03:56 +00:00
b8da6b9fc6 [elpais] Modernize 2015-01-09 22:43:49 +01:00
4baea47c42 release 2015.01.09.1 2015-01-09 21:33:16 +01:00
176cf9e0c3 [wdr] Support overviews (Fixes #4651) 2015-01-09 21:33:07 +01:00
7b6faddfc8 [wdr] Modernize 2015-01-09 20:52:49 +01:00
f90ad27375 [YoutubeDL] Copy over format metadata when merging (Fixes #4671) 2015-01-09 20:50:23 +01:00
230b2287dd [youtube] Add acodec information
The codec seems to be consistently aac, so state that in our metadata.
2015-01-09 20:44:21 +01:00
754c838903 release 2015.01.09 2015-01-09 20:20:55 +01:00
aa2fd59857 [update] Use utils HTTPS handler (Fixes #4666)
On FreeBSD, the default HTTPS handler is missing certificates, so use our own.
2015-01-09 20:20:48 +01:00
9932a65370 [vk] Remove debug assert statement (fixes #4672, fixes #4514) 2015-01-09 20:13:53 +01:00
5e4166478d [README] Add an FAQ entry for how to install on Windows 2015-01-09 19:17:15 +01:00
b0e87c3110 [ffmpeg] Correctly encode paths on Windows
On Python 2.x on Windows, if there are any unicode arguments in the command argument list, the whole list is converted to unicode internally.
Therefore, we need to call encodeArgument on every argument.

Fixes #4337 and #4668.
2015-01-09 19:02:07 +01:00
ff0813313a Minor style changes 2015-01-08 18:35:33 -06:00
c0bdf32a3c Add --print-json (Closes #2845) 2015-01-08 18:03:29 +01:00
92b065dc53 [tudou] Fix extraction 2015-01-08 18:03:29 +01:00
9298d4e3df [discovery] Fix extractor 2015-01-08 18:03:29 +01:00
740a7fcbc8 [gdcvault] Skip test that is now restricted 2015-01-08 18:03:29 +01:00
5fbf25a681 [test_age_restriction] remove misbehaving test
We now test for the age_limit being set right in test_download, so we don't need more than two tests for the actual age limit handling.
2015-01-08 18:03:29 +01:00
db6e625005 [buzzfeed] Fix test 2015-01-08 18:03:29 +01:00
811cacdc2c [bet] Correct test IDs 2015-01-08 18:03:29 +01:00
ce08a86462 Merge pull request #4647 from aajanki/hds_metadata
[downloader/f4m] Improved metadata handling
2015-01-08 16:37:49 +01:00
11497d5bba release 2015.01.08 2015-01-08 16:15:08 +01:00
0217c78377 [YoutubeDL] Allow selection by more extensions 2015-01-08 16:14:50 +01:00
bd6b25ce0e [fktv] Fix download URL 2015-01-08 16:14:50 +01:00
d51a853d5c [zdf] Fix test case 2015-01-08 16:14:50 +01:00
9ed99402f5 [youtube] Fix test case 2015-01-08 16:14:50 +01:00
ec3a6a3137 [tunein] Ignore reliability if it's >90% (#4097) 2015-01-08 16:14:50 +01:00
796858a53f [sexykarma] Add age_limit designation 2015-01-08 16:14:50 +01:00
5b78caca94 [mit] Amend test definitions 2015-01-08 16:14:50 +01:00
bec2248141 [InfoExtractor/common] Correct and test meta tag matching 2015-01-08 16:14:50 +01:00
211503c39f [teachertube] Modernize 2015-01-08 16:14:50 +01:00
adb1307b9a [imdb] Remove test md5
They seem to reencode quite frequently, so simply remove the md5 sum.
2015-01-08 16:14:50 +01:00
99673f04bc [washingtonpost] Modernize and correct test case 2015-01-08 16:14:49 +01:00
e9a537774d Merge pull request #4665 from Li4ick/patch-1
Change path name to MSDN standard.
2015-01-08 14:53:20 +01:00
367f539769 Merge pull request #4664 from kieranoreilly/patch-1
Spelling
2015-01-08 19:09:46 +06:00
398133cf55 [huffpost] Make extraction more robust (Closes #4663) 2015-01-08 19:07:28 +06:00
52fc3ba405 Change path name to MSDN standard.
<Yourname> changed to <user name>, which is more general.
2015-01-08 14:17:26 +02:00
fdd6e18b75 Spelling
Corrected the spelling of incapacitated
2015-01-07 20:23:36 -08:00
58a84b8cb6 [bilibili] Fix extraction (Closes #4660) 2015-01-08 01:33:22 +06:00
c5d666d374 Fix build with python 2.6
* Packages cannot be executed
* '.format' needs the index of the argument

(Reported in https://github.com/Homebrew/homebrew/issues/35616)
2015-01-07 16:09:43 +01:00
5d8993b06a [extractor/__init__] Remove unused import 2015-01-07 11:59:15 +01:00
c758bf9fd7 [nrktv] Remove 'proxy' parameter from tests 2015-01-07 11:56:22 +01:00
900813a328 release 2015.01.07.2 2015-01-07 07:41:48 +01:00
2bad0e5d20 [/__init__] Define public API 2015-01-07 07:41:05 +01:00
ccc5842bc9 [gameone] Modernize 2015-01-07 07:37:21 +01:00
fd86c2026d release 2015.01.07.1 2015-01-07 07:31:38 +01:00
e4a8eae701 Merge commit '8ee3415' 2015-01-07 07:30:57 +01:00
75e51819d0 release 2015.01.07 2015-01-07 07:22:28 +01:00
8ee341500d [viki] Modernize 2015-01-07 07:21:24 +01:00
0590062925 Respect age_limit when listing extractors (Fixes #4653) 2015-01-07 07:20:20 +01:00
799d88d3d8 [nrktv] Add support for playlists (Closes #4656) 2015-01-07 06:46:56 +06:00
760aea9a96 Merge branch 'oskar456-ceskatelevizesrt' 2015-01-07 05:05:30 +06:00
d6a31b1766 Credit @oskar456 for ceskatelevize subtitles support (#4622) 2015-01-07 05:05:18 +06:00
0b54a5b10a [ceskatelevize] Add subtitles tests 2015-01-07 05:04:15 +06:00
6309cb9b41 [ceskatelevize] Fix python 2.6 format issue 2015-01-07 05:03:34 +06:00
27a82a1b93 [ceskatelevize] Simplify 2015-01-07 05:03:14 +06:00
ecd1936695 Merge branch 'ceskatelevizesrt' of https://github.com/oskar456/youtube-dl into oskar456-ceskatelevizesrt 2015-01-07 05:02:27 +06:00
76b3c61012 [youtube] Add formats 308 and 315 (closes #4650) 2015-01-06 11:59:41 +01:00
0df2dea73b [giga] Add extractor (Closes #4090) 2015-01-06 06:54:31 +06:00
f8bb576c4f release 2015.01.05.1 2015-01-05 22:42:38 +01:00
ee61f6f3e2 [youtube] Handle cases where format comes without a preference (Fixes #4648) 2015-01-05 22:42:17 +01:00
f14f2a6d79 [downloader/f4m] Minor cleanup 2015-01-05 21:12:33 +02:00
2c322cc5d6 [downloader/f4m] The last value in a tag is the tag length 2015-01-05 21:07:15 +02:00
3b8f3a1504 [downloader/f4m] <metadata> is optional according to the F4M specs 2015-01-05 21:07:13 +02:00
8f9529cd05 [motorsport] Fix extraction and make trailing '/' optional
They directly embed a youtube video now.
2015-01-05 19:19:01 +01:00
f4bca0b348 release 2015.01.05 2015-01-05 18:44:29 +01:00
6291438073 [auengine] Simplify (#4643) 2015-01-05 18:21:32 +01:00
18c3c15391 Merge remote-tracking branch 'Oteng/master' 2015-01-05 18:18:15 +01:00
dda620e88c [radiobremen] Make code more readable and more resilient to failures 2015-01-05 18:17:03 +01:00
d7cc31b63e [generic] PEP8 2015-01-05 18:16:47 +01:00
5e3e1c82d8 Credit @ckrooss for radiobremen (#4632) 2015-01-05 18:14:39 +01:00
aa80652f47 [radiobremen] Add test for thumbnail 2015-01-05 18:14:09 +01:00
9d247bbd2d [radiobremen] Fix under Python 2.6 and fix duration 2015-01-05 18:13:19 +01:00
93e40a7b2f Merge remote-tracking branch 'ckrooss/master' 2015-01-05 18:07:16 +01:00
03ff2cc1c4 [Auengine] corrected extractions logic
The way the video download url was been extracted was
not working well so i change it for it to extract the
correct url
2015-01-05 16:28:24 +00:00
a285b6377b [normalboots] Skip download in test, it uses rtmp 2015-01-05 13:59:49 +01:00
cd791a5ea0 [ted] Add support for embed-ssl.ted.com embedded videos 2015-01-05 13:11:13 +01:00
87830900a9 [generic] Update some tests 2015-01-05 13:07:24 +01:00
dfc9d9f50a Merge pull request #4639 from bartkappenburg/patch-1
Update rtlnl.py
2015-01-05 12:31:07 +01:00
75311a7e16 .travis.yml: Remove my email from the list 2015-01-05 12:29:32 +01:00
628bc4d1e7 [khanacademy] Update test 2015-01-05 12:28:35 +01:00
a4c3f48639 [vimple] Replace tests
The first one seems to be no longer available and the second was an episode from a tv show.
2015-01-05 11:54:14 +01:00
bdf80aa542 Update rtlnl.py
Added support for the non-www version of rtlxl.nl by making "www." optional.
2015-01-05 11:51:24 +01:00
adf3c58ad3 [lrt] Fix missing provider key
Also, modernize a bit.
2015-01-05 02:55:12 +02:00
caf90bfaa5 [webofstories] Add new extractor (Closes #4585) 2015-01-05 02:22:01 +02:00
2f985f4bb4 [youtube:toplist] Remove extractor
They use now normal playlists (their id is PL*).
2015-01-05 00:18:43 +01:00
67c2bcdf4c Remove extractors which infringe copyright (#4554) 2015-01-04 19:19:18 +01:00
1d2d0e3ff2 utils: Remove blank line at the end of file 2015-01-04 14:07:06 +01:00
9fda6ee39f [tf1] Remove unused import 2015-01-04 14:06:23 +01:00
bc3e582fe4 Don't use '-shortest' option for merging formats (closes #4220, closes #4580)
With avconv and older versions of ffmpeg the video is partially copied.
The duration difference between the audio and the video seem to be really small, so it's probably not noticeable.
2015-01-04 14:02:17 +01:00
bc1fc5ddbc Don't check for height as it's not provided 2015-01-04 14:02:07 +01:00
63948fc62c [downloader/hls] Respect the 'prefer_ffmpeg' option 2015-01-04 13:41:49 +01:00
f4858a7103 Add support for Radio Bremen 2015-01-04 13:33:26 +01:00
26886e6140 release 2015.01.04 2015-01-04 03:15:48 +01:00
7a1818c99b [vk] Add support for rutube embeds (Fixes #4514) 2015-01-04 03:15:27 +01:00
2ccd1b10e5 [soulanime] Fix under Python 3 2015-01-04 02:20:45 +01:00
788fa208c8 Merge branch 'master' of github.com:rg3/youtube-dl 2015-01-04 02:08:38 +01:00
8848314c08 [Makefile] Make offline tests actually work offline 2015-01-04 02:08:18 +01:00
c11125f9ed [tests] Remove format 138 from tests (#4559) 2015-01-04 02:06:53 +01:00
95ceeec722 Remove unused import 2015-01-04 02:05:35 +01:00
b68ff25917 Add various anime sites (Closes #4554) 2015-01-04 02:05:26 +01:00
3e3327ea17 Merge pull request #4629 from t0mm0/tf1-tfou
[tf1] add support for TFOU
2015-01-04 06:51:28 +06:00
b158bb8693 [tf1] simplify regex 2015-01-04 00:45:23 +00:00
2bf098eda4 [tf1] fix test 2015-01-04 00:43:55 +00:00
382e05fa56 [tf1] add support for TFOU 2015-01-04 00:05:31 +00:00
19b05d886e release 2015.01.03 2015-01-03 18:35:30 +01:00
e65566a9cc [youtube] Correct handling when DASH manifest is not necessary to find all formats 2015-01-03 18:33:38 +01:00
baa3c3f0f6 [ellentv] Improve extraction 2015-01-03 21:54:18 +06:00
f4f339529c [ellentv] Clean up and simplify 2015-01-03 21:44:47 +06:00
7d02fae85b Merge pull request #4626 from gauravb7090/ellentube
Added support for EllenTube along with EllenTV
2015-01-03 21:40:39 +06:00
6e46c3f1fd Added support for EllenTube along with EllenTV 2015-01-03 20:30:28 +05:30
c7e675940c [bbccouk] Add support for music clips (Closes #4143) 2015-01-03 20:43:40 +06:00
d26b1317ed [downloader/mplayer] Use check_executable 2015-01-03 00:33:36 +01:00
a221f22969 [crunchyroll] Fix format extraction
Reported in https://github.com/rg3/youtube-dl/issues/2782#issuecomment-68556780
2015-01-02 21:17:10 +01:00
817f786fbb [canalplus] Raise an error if the video is georestricted (closes #4472) 2015-01-02 21:02:34 +01:00
62420c73cb [played] Skip test 2015-01-02 22:31:55 +06:00
2522a0b7da [kontrtube] Extract display_id
Trailing slash in URL is mandatory now
2015-01-02 22:28:48 +06:00
46d32a12c9 [bet] Update test 2015-01-02 22:23:00 +06:00
c491418526 [bbccouk] Update test 2015-01-02 22:13:26 +06:00
c067545c17 ceskatelevize: Closed captions support 2015-01-02 17:12:20 +01:00
823a155293 [vier:videos] Tune _VALID_URL not to match single videos 2015-01-02 22:09:00 +06:00
324b2c78fa [xtube] Fix uploader regex 2015-01-02 21:46:57 +06:00
d34f98289b [xhamster] Remove identical tests 2015-01-02 21:12:25 +06:00
644096b15c Merge pull request #4615 from dwemthy/https_xhamster
[xhamster] Add HTTPS support
2015-01-02 21:09:28 +06:00
15cebcc363 Merge branch 'master' of github.com:rg3/youtube-dl 2015-01-02 20:57:12 +06:00
faa4ea68c0 [generic] Add BBC iPlayer playlist test 2015-01-02 20:56:42 +06:00
29a9385ff0 release 2015.01.02 2015-01-02 15:56:26 +01:00
476eae0c2a [generic] Generalize BBC iPlayer playlist extraction 2015-01-02 20:55:09 +06:00
8399267671 [generic] Make getter None by default 2015-01-02 20:54:30 +06:00
db546cf87f [generic] Add support for BBC iPlayer embeds (Closes #4619) 2015-01-02 20:46:17 +06:00
317639758a [bbccouk] Improve _VALID_URL 2015-01-02 20:37:54 +06:00
fdbabca85f [vier:videos] Tune _VALID_URL 2015-01-02 20:21:41 +06:00
6f790e5821 Credit @lovebug356 for vier (#4617) 2015-01-02 20:16:43 +06:00
6f5cdeb611 Merge branch 'lovebug356-vier' 2015-01-02 20:15:59 +06:00
9eb4f404cb [vier] Simplify, add support for more URL formats, extract all playlist pages when page is not specified 2015-01-02 20:15:40 +06:00
f58487b392 [vier] Add new extractor 2015-01-02 13:35:47 +01:00
5b9aefef77 [xhamster] Add HTTPS support 2015-01-02 11:54:38 +00:00
defaf19f5d Push api updates to simplify audiomack, add support for albums 2015-01-02 02:20:04 -06:00
772fd5cc44 [youtube] Add a pseudo-extractor for truncated YouTube video IDs (#4610) 2015-01-01 23:44:39 +01:00
50a0f6df7e [/__init__] Add another cute search example 2015-01-01 22:47:21 +01:00
9f435c5f1c Add an extractor for common mistakes (#4610) 2015-01-01 22:34:58 +01:00
931e2d1d26 [bbccouk] PEP8 2015-01-01 22:15:46 +01:00
a42419da42 [options] Upper-case options and URL in --help output
Hopefully, this reduces confusion as in #4610.
2015-01-01 22:01:47 +01:00
9a237b776c release 2015.01.01 2015-01-01 21:41:42 +01:00
02ec32a1ef [ceskatelevize] Adapt to new API (Closes #4531) 2015-01-01 20:01:55 +06:00
a1e9e6440f [moevideo] Skip removed video test 2015-01-01 00:46:03 +06:00
5878e6398c [nrktv] Update tests' checksums 2015-01-01 00:37:57 +06:00
6c6f1408f2 [extractor/common] Allow multiline content tags 2015-01-01 00:37:14 +06:00
b7a7319c38 [slideshare] Fix extraction 2015-01-01 00:26:19 +06:00
68f705cac5 [tnaflix] Make sure config URL has correct scheme 2015-01-01 00:12:41 +06:00
079d1dcd80 [tnaflix] Fix title extraction 2015-01-01 00:11:56 +06:00
7b24bbdf49 [xboxclips] Fix extraction 2014-12-31 23:59:16 +06:00
f86d543ebb [pbs] Catch geoblocking errors (closes #4516) 2014-12-31 17:43:49 +01:00
60e47a2699 [youtube] Use '_download_xml' for getting the available subtitles 2014-12-31 15:44:15 +01:00
b8bc7a696b [openfilm] Add extractor (Closes #4538) 2014-12-31 19:40:35 +06:00
7d900ef1bf [youtube] Add support for automatically translated subtitles (fixes #4555)
They have a manually uploaded subtitles track and YouTube can transtale it.
2014-12-31 14:15:16 +01:00
1931a73f39 [echomsk] Add extractor (Closes #4600) 2014-12-31 18:03:51 +06:00
966ea3aebd [README] Typo / clarify FAQ 2014-12-30 23:41:29 +01:00
b3013681ff Merge remote-tracking branch 'origin/master' 2014-12-30 19:41:04 +01:00
416c7fcbce Add documentation about supported sites (Fixes #4503) 2014-12-30 19:35:35 +01:00
e83eebb12f [atresplayer] Fix python3 bug 2014-12-30 22:46:23 +06:00
a349873226 [atresplayer] Add extractor (Closes #2341) 2014-12-30 22:28:07 +06:00
fccae2b911 [youtube] Add test for age-gate video with encrypted signature 2014-12-30 17:26:21 +06:00
3ee08848db Credit @0xced for #4598 2014-12-30 17:12:12 +06:00
0129b4dd45 Merge pull request #4598 from 0xced/encrypted-age-gate
[youtube] Fix videos with age gate and encrypted signatures
2014-12-30 17:09:02 +06:00
1c57e7f1f4 [daum] Improve full_id regex 2014-12-30 16:55:53 +06:00
d0caf3a11e Merge pull request #4599 from t0mm0/daum_fix
[daum] update 'full id' regex
2014-12-30 16:52:02 +06:00
a87bb090d9 [daum] update 'full id' regex
fixes #4566
2014-12-29 23:06:56 +00:00
beb95e7781 [youtube] Fix videos with age gate and encrypted signatures
The `sts` value is available on the embed webpage, get it from there.

Fixes #4108.
2014-12-29 22:58:14 +01:00
5435d7af91 Merge branch 't0mm0-hitbox' 2014-12-30 03:22:25 +06:00
0c0a70f4c6 [hitbox] Minor changes 2014-12-30 03:22:07 +06:00
e3947e2b7f [hitbox] add support for live streams 2014-12-29 20:12:23 +00:00
da3f7fb7f8 [hitbox] add extractor for hitbox vods 2014-12-29 20:12:23 +00:00
429ddfd38d [cnn] Add support for hln URL format (Closes #4595) 2014-12-30 01:50:28 +06:00
479514d015 Merge branch 'peugeot-hellporno' 2014-12-29 21:33:57 +06:00
355e41466d [hellporno] Extract all formats and improve 2014-12-29 21:33:41 +06:00
03d9aad87c Merge branch 'hellporno' of https://github.com/peugeot/youtube-dl into peugeot-hellporno 2014-12-29 21:13:09 +06:00
3e2bcf530b Merge branch 'peugeot-xxxymovies' 2014-12-29 21:05:41 +06:00
6343a5f68e [xxxymovies] Improve 2014-12-29 21:05:21 +06:00
00de9a9828 Merge branch 'xxxymovies' of https://github.com/peugeot/youtube-dl into peugeot-xxxymovies 2014-12-29 20:38:28 +06:00
7fc2cd819e [cnn] Improve regexes and fix test 2014-12-29 20:27:09 +06:00
974739aab5 Merge pull request #4543 from akretz/cnn_fix
[cnn] Add support for articles with videos (fixes #4541)
2014-12-29 20:21:39 +06:00
0cc4f8e385 [xxxymovies] new ectractor 2014-12-29 11:31:22 +01:00
513fd2a872 [hellporno] simplify 2014-12-29 10:38:07 +01:00
ae6986fb89 [bbccouk] Switch to new JSON playlist format (Closes #4588) 2014-12-29 03:00:24 +06:00
e8e28989eb [archiveorg] Add test, simplify and modernize 2014-12-29 02:08:46 +06:00
0fa629d05b Merge pull request #4590 from derrotebaron/master
[archiveorg] most metadata fields are optional
2014-12-29 01:53:59 +06:00
ff7a07d5c4 [archiveorg] most metadata fields are optional
Example: https://archive.org/details/Cops1922
2014-12-28 20:31:25 +01:00
5a18403057 [arte.tv] Fix typo 2014-12-28 15:42:29 +06:00
1b7b1d6eac [arte.tv:+7] Make quality optional (Closes #4583) 2014-12-28 15:41:52 +06:00
23cfa4ae45 Merge branch 'peugeot-alphaporno' 2014-12-27 00:08:25 +06:00
e82def52a9 [alphaporno] Improve 2014-12-27 00:08:04 +06:00
bcfe9db299 Merge branch 'alphaporno' of https://github.com/peugeot/youtube-dl into peugeot-alphaporno 2014-12-26 23:34:12 +06:00
cf00ae7640 Merge branch 'peugeot-eroprofile' 2014-12-26 23:33:01 +06:00
f9b9e88646 [eroprofile] Simplify 2014-12-26 23:32:41 +06:00
c2500434c3 Merge branch 'eroprofile' of https://github.com/peugeot/youtube-dl into peugeot-eroprofile 2014-12-26 23:16:25 +06:00
f74b341dde expect_info_dict actual-expected argument consistency 2014-12-26 23:07:24 +06:00
461b00f34a [eroprofile] new extractor 2014-12-26 17:15:34 +01:00
4cda41ac7b [alphaporno] new extractor 2014-12-26 16:17:35 +01:00
6a1c4fbfcb [hellporno] new extractor 2014-12-26 15:49:12 +01:00
31424c126f [sunporno] Modernize 2014-12-26 19:28:51 +06:00
53096539dc Merge pull request #4568 from peugeot/sunporno
[sunporno] fix duration
2014-12-26 19:25:05 +06:00
2c0b475235 [sunporno] fix duration 2014-12-26 12:49:13 +01:00
a542405200 Credit @MaxReimann for teletask (#4533) 2014-12-25 23:29:10 +06:00
3e2b085ef9 Merge branch 'MaxReimann-teletask' 2014-12-25 23:27:23 +06:00
885e4384a1 [teletask] Simplify 2014-12-25 23:26:57 +06:00
2b8f151094 Merge branch 'teletask' of https://github.com/MaxReimann/youtube-dl into MaxReimann-teletask 2014-12-25 23:06:26 +06:00
5ac71f0b27 [sohu] Modernize and extract all formats and more metadata (Closes #4409, closes #2056, closes #3009) 2014-12-25 22:25:05 +06:00
39ac7c9435 [gameone] Extract duration as float 2014-12-24 19:18:59 +06:00
ed7bdc8a90 Merge pull request #4553 from tobidope/gameone
[gameone] This fix resolves issue #4552
2014-12-24 19:05:05 +06:00
55f0cab3a3 [gameone] This fix resolves issue #4552
The duration metadata for certain episodes contained floating point
numbers instead of integers. Now only the integer part will be
interpreted. Also added a test for this
2014-12-23 22:09:21 +01:00
544dec6298 [smotri] Skip broken tests 2014-12-23 20:33:56 +06:00
e0ae1814b1 [sportdeutschland] Fix extraction (fixes #4544) 2014-12-22 22:24:19 +01:00
9532d72371 [cnn] Add support for articles with videos (fixes #4541) 2014-12-22 18:40:36 +01:00
1362bbbb4b [adobetv] Add extractor (Closes #4536) 2014-12-22 22:05:47 +06:00
f00fd51dae Don't write the description file if info_dict['description'] is None (#3166) 2014-12-21 20:49:14 +01:00
a8896c5ac2 [crunchyroll] Add .fr domain (#4537) 2014-12-22 00:58:15 +06:00
5d3808524d [extractor/common] Update docstring: replace FileDownloader with YoutubeDL 2014-12-21 16:58:29 +01:00
c8f167823f [dbtv] Make sure the 'id' field is a string 2014-12-21 16:57:07 +01:00
70f6796e7d [telecinco] Rename 'episode' group to 'id' in the _VALID_URL regex
MiTeleIE now uses '_match_id'
2014-12-21 16:54:53 +01:00
85d253af6b [internetvideoarchive] Update test's duration field 2014-12-21 15:37:21 +01:00
a86cbf5876 [rtp] Fix test's id field 2014-12-21 15:28:40 +01:00
3f1399de8a [tmz] Fix test's thumbnail field 2014-12-21 15:26:00 +01:00
1f809a8560 [nerdcubed] Style fixes 2014-12-21 15:22:30 +01:00
653d14e2f9 [yahoo] Update extraction process
Their webpage uses now https://video.media.yql.yahoo.com/v1/video/sapi/streams/ for getting the video info.
2014-12-21 14:47:44 +01:00
85fab7e47b [yahoo] Replace two tests
The first one returned an internal server error.
The other doesn't seem to contain a video anymore.
2014-12-21 14:47:12 +01:00
3aa9176f08 [yahoo] Improve video id detection (fixes #4521) 2014-12-21 14:09:00 +01:00
33b53b6021 [teletask] Add new extractor 2014-12-21 12:26:47 +01:00
3f7421b71b fix test and remove lengthy description 2014-12-21 11:13:59 +01:00
ee45625290 Add extractor for teletask 2014-12-21 11:01:28 +01:00
2c2a42587b [dvtv] Fix thumbnail scheme 2014-12-21 07:38:55 +06:00
e2f65efcf9 Merge branch 'petrkutalek-dvtv' 2014-12-21 07:34:27 +06:00
081d6e4784 [dvtv] Simplify 2014-12-21 07:33:58 +06:00
1d4247f64e [dvtv] Add support for playlists 2014-12-21 01:24:05 +01:00
1ff30d7b79 [npo] Add support for streams (Closes #4276) 2014-12-20 18:30:56 +06:00
16ea817968 [xtube] Fix and modernize (Closes #4489) 2014-12-19 21:56:44 +06:00
a2a4bae929 Credit @willglynn for nerdcubed (#4515) 2014-12-19 10:32:20 +01:00
c58843b3a1 [nerdcubed] Add new extractor
nerdcubed.co.uk describes videos in a single a feed.json file, providing
references to and metadata on >1300 YouTube videos spread across 3 main
channels as well as guest appareances on other channels via a single HTTP
request.

NerdCubedFeedIE transforms this feed into a youtube-dl playlist, preserving
information present in the upstream JSON (allowing zero-cost title/date
matches) and ultimately referencing the embedded YouTube videos.
2014-12-18 22:32:24 -06:00
a22524b004 [twitch] Add support for vods (Closes #4512) 2014-12-18 21:25:42 +06:00
87c4c21e75 Credit @petrkutalek for dvtv (#4502) 2014-12-17 23:38:11 +01:00
b9465395cb [dvtv] PEP8 and correct format sorting (#4502) 2014-12-17 23:18:06 +01:00
edf41477f0 Merge remote-tracking branch 'petrkutalek/dvtv' 2014-12-17 23:12:38 +01:00
5f627b4448 [dvtv] Add new extractor 2014-12-17 15:52:54 +01:00
60e5428925 [flake8] Ignore build/ directory
That directory is temporarily generated when building for PyPi and may be present if something goes wrong with the upload.
2014-12-17 15:36:18 +01:00
748ec66725 [theplatform] Extract captions (Closes #4495) 2014-12-17 20:20:40 +06:00
e54a3a2f01 [screencastomatic] Remove unused variable 2014-12-17 14:56:30 +01:00
0e4cb4f406 YoutubeDL: style fix 2014-12-17 14:55:27 +01:00
f7ffe72ac7 Merge pull request #4501 from AndroKev/master
only add video-id to archive, when successful
2014-12-17 13:31:33 +01:00
cd58dc3e56 Update YoutubeDL.py 2014-12-17 13:21:22 +01:00
c33bcf2051 only add video-id to archive, when successful
Example:
no space left--> youtube-dl adds the id to archive, but the video isn't fully downloaded
2014-12-17 13:05:19 +01:00
7642c08763 release 2014.12.17.2 2014-12-17 11:39:25 +01:00
fdc8000810 [downloader] Handle a file ./- (Fixes #4498) 2014-12-17 11:39:06 +01:00
a91c9b15e3 release 2014.12.17.1 2014-12-17 11:29:52 +01:00
27d67ea2ba [comedycentral] Match URLs with a second ID (fixes #4499) 2014-12-17 11:29:35 +01:00
d6a8160902 release 2014.12.17 2014-12-17 10:53:17 +01:00
6e1b9395c6 [screencastomatic] Add new extractor (Fixes #4497) 2014-12-17 10:53:12 +01:00
b1ccbed3d4 [nhl] Allow upper-case video IDs (Fixes #4494) 2014-12-17 00:26:04 +01:00
37381350f8 [aljazeera] Add unicode_literals marker 2014-12-17 00:08:04 +01:00
7af808a5ef Improve code style 2014-12-17 00:06:41 +01:00
876bef5937 [mit] Modernize 2014-12-17 00:04:24 +01:00
a16af51873 flake8: Add more ignored files
* setup.py: the '__version__' variable is not defined in the script, it is loadded from youtube_dl/version.py
* devscripts/buildserver.py: Produces a lot of messages
2014-12-16 20:38:59 +01:00
dc9a441bfa Move flake8 configuration to setup.cfg
It will be used when calling flake8 from any directory in the project
2014-12-16 20:34:07 +01:00
ee6dfe8308 Use flake8 instead of pyflakes and pep8
It combines both tools
2014-12-16 20:34:07 +01:00
2cb5b03e53 [test/test_unicode_literals] Remove duplicated imports 2014-12-16 20:33:23 +01:00
964b190350 release 2014.12.16.2 2014-12-16 16:45:35 +01:00
13d27a42cc [orf:tvthek] Add support for topic URLs (Fixes #4474) 2014-12-16 16:45:28 +01:00
ec05fee43a [brightcove] Add shorter URL scheme for other extractors 2014-12-16 16:38:26 +01:00
b50e3bc67f [README] Add table of contents (Closes #4458) 2014-12-16 16:33:23 +01:00
ac78b5e97b release 2014.12.16.1 2014-12-16 16:03:57 +01:00
17e0d63957 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-16 16:03:46 +01:00
9209fe3878 [allocine] Add test for new URL format 2014-12-16 21:03:10 +06:00
84d84211ac [youtube:feeds] (Fixes #4486) 2014-12-16 15:59:31 +01:00
b4116dcdd5 Merge pull request #4490 from Tailszefox/master
[Allocine] Support for more URLs
2014-12-16 20:59:07 +06:00
bb18d787b5 [aljazeera] Add extractor (closes #4487) 2014-12-16 15:48:01 +01:00
0647084f39 [Allocine] Support for more URLs 2014-12-16 15:46:04 +01:00
734ea11e3c Drop hash character in downloader output (#4484) 2014-12-16 00:37:42 +01:00
3940450878 release 2014.12.16 2014-12-16 00:24:30 +01:00
ccbfaa83b0 [devscripts/make_contributing] Switch to optparse (Fixes #4483) 2014-12-16 00:24:11 +01:00
d86007873e [YoutubeDL] Document where details for format can be found 2014-12-16 00:22:23 +01:00
4b7df0d30c [youtube:playlist] Work around buggy playlists (fixes #4449)
They show a "Load more" button, but they don't have more videos.
The continuation url in the json file was a link to itself, so we ended up in an infinite loop.
2014-12-15 19:19:15 +01:00
caff59499c [README] Fix code rendering 2014-12-15 11:14:06 +01:00
99a0f9824a [README] Highlight code examples 2014-12-15 11:11:52 +01:00
3013bbb27d Remove unused imports 2014-12-15 08:24:50 +01:00
6f9b54933f [streamcloud] Modernize 2014-12-15 03:32:17 +02:00
1bbe317508 [mooshare] Modernize 2014-12-15 03:31:54 +02:00
e97a534f13 release 2014.12.15 2014-12-15 01:36:46 +01:00
8acb83d993 [README] Make example audio sound not that horrible ;) 2014-12-15 01:34:39 +01:00
71b640cc5b [YoutubeDL] Add declarative version of progress hooks 2014-12-15 01:26:20 +01:00
4f026fafbc [YoutubeDL] Make postprocessors declarative
Instead of having to configure PPs in code, this allows us and embedding programs not to worry about imports or finer details, similarly to how we handle IEs.
2014-12-15 01:06:25 +01:00
39f594d660 [Makefile] Ensure that offline test really is offline 2014-12-15 00:59:23 +01:00
cae97f6521 Improve and test ffmpeg version detection 2014-12-14 21:59:59 +01:00
6cbf345f28 Remove test/write_info_json
This is now covered by every single test_download testcase anyways :)
2014-12-14 21:56:12 +01:00
a0ab29f8a1 Add offlinetest make target 2014-12-14 21:55:57 +01:00
4a4fbfc967 [yesjapan] Look for datetime inside submit_info
Oops..
2014-12-14 18:03:05 +02:00
408b5839b1 [yesjapan] Add new extractor (Closes #4466) 2014-12-14 17:59:25 +02:00
60620368d7 [youtube] Fix player ID detection 2014-12-14 00:43:34 +01:00
4927de4f86 release 2014.12.14 2014-12-14 00:13:17 +01:00
bad5c1a303 [rtp] Also match e-id-less URLs (#4382) 2014-12-14 00:13:07 +01:00
6f18cc9abc release 2014.12.13.1 2014-12-13 23:51:57 +01:00
4d144be8b0 [bandcamp:album] Do not match plain Bandcamp URLs (#4461)
The _VALID_URL 1fa174692a is to broad, since it matches everything beginning with bandcamp.com.
2014-12-13 23:50:06 +01:00
2128b696b8 [utils] Do not make an exception for SSLv3
SSLv3 is terminally vulnerable to POODLE; web browsers are currently deprecating/removing it.
Closes #4459, fixes #4294
2014-12-13 23:45:34 +01:00
a23669220a [utils] Make ssl work on Python 2.7.8 2014-12-13 23:27:21 +01:00
051c46256b release 2014.12.13 2014-12-13 23:13:48 +01:00
d5524947b5 Merge remote-tracking branch 'fstirlitz/master' 2014-12-13 23:05:41 +01:00
74f91c4af7 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-13 23:05:28 +01:00
da4d4191a9 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-13 23:05:22 +01:00
2564300e55 Credit @Mortal for restudy (#4463) 2014-12-14 03:42:42 +06:00
cb0713d2c9 Merge branch 'Mortal-restudy' 2014-12-14 03:41:17 +06:00
ac265bef1e [restudy] Simplify and extract all formats 2014-12-14 03:41:00 +06:00
4a0132c570 [Restudy] Add new extractor for restudy.dk 2014-12-13 22:25:32 +01:00
1fa174692a [bandcamp:album] Make path optional (Closes #4461) 2014-12-14 02:00:54 +06:00
04c9544187 [bbccouk] Fix vpid warning 2014-12-13 18:47:34 +06:00
8085fc15cc [adultswim] Improve segment duration extraction 2014-12-13 18:42:29 +06:00
2f15832f56 Merge pull request #3927 from qrtt1/master
apply ratelimit to f4m
2014-12-13 12:59:12 +01:00
1557ed153c [test_unicode_literals] Import from test.helper 2014-12-13 12:45:09 +01:00
a6620ac28d [orf] Modernize 2014-12-13 12:41:38 +01:00
89e36657cc [keek] remove unused import 2014-12-13 12:36:46 +01:00
7129bed51b [keek] Modernize and extract uploader 2014-12-13 12:35:45 +01:00
1cc79574fc Fix imports and general cleanup
· Import from compat what comes from compat. Yes, some names are available in utils too, but that's an implementation detail.
· Use _match_id consistently whenever possible
· Fix some outdated tests
· Use consistent valid URL (always match the whole protocol, no ^ at start required)
· Use modern test definitions
2014-12-13 12:35:45 +01:00
20e35880bf [streamcz] Update extractor 2014-12-13 12:35:45 +01:00
5e1912cfc1 [5min] Remove helper method and modernize
Previously, other extractor would go call a private(!) helper method. Instead, just hardcode the 5min:video_id format - it's not if that would ever change.
2014-12-13 12:35:45 +01:00
293f0f39ce [utils] make_HTTPS_handler: Remove try/except block that would always raise an exception
This code is only run for Python < 3.4, where context.load_default_certs doesn't exist
2014-12-12 23:43:25 +01:00
0db261ba56 [utils] make_HTTPS_handler: Use ssl.create_default_context in Python 2.7.9
The new features in the ssl module have been backported from 3.4, see https://docs.python.org/dev/whatsnew/2.7.html#pep-466-network-security-enhancements-for-python-2-7
2014-12-12 23:35:17 +01:00
7668a2c5cb [comcarcoff] add webpage_url datum 2014-12-12 23:20:34 +01:00
26c06f0c51 [youtube:playlist] Remove unused property 2014-12-12 22:26:50 +01:00
23d3608c6b [youtube:channel] Fix extraction (fixes #4435)
It uses now the same pagination system as playlists
2014-12-12 22:23:54 +01:00
baa7081d68 [urort] Update to new multi-format protocol 2014-12-12 20:55:18 +01:00
19bf2b4e88 [comcarcoff] Add unicode_literals declaration 2014-12-12 20:37:58 +01:00
6a1b20de2a [urort] Modernize 2014-12-12 20:37:28 +01:00
3c864e930d [comcarcoff] Adapt c62159ea91a04ef82560472b254aef1cc9f70a11 2014-12-12 20:35:17 +01:00
dc5596ff54 [comcarcoff] (#4454) 2014-12-12 20:32:02 +01:00
46d9760f5e Merge remote-tracking branch 'fstirlitz/master' 2014-12-12 20:17:26 +01:00
90d71d3f08 [ooyala] Remove test md5sums 2014-12-12 20:12:51 +01:00
e9404524cc [ninegag] Test for additional properties 2014-12-12 20:10:15 +01:00
dc65a213fd comediansincarsgettingcoffee.com support 2014-12-12 19:58:44 +01:00
4237ba10dc [pornotube] Adapt to new interface 2014-12-12 19:44:25 +01:00
c3f3b29b92 [rtp] Add new extractor (Closes #4382) 2014-12-12 20:22:24 +02:00
1c985da0ca release 2014.12.12.7 2014-12-12 18:25:58 +01:00
7a60322abf release 2014.12.12.6 2014-12-12 17:52:50 +01:00
07bc9a3530 [nowvideo] Add .li domain (Closes #4453) 2014-12-12 22:44:16 +06:00
a099965bad release 2014.12.12.5 2014-12-12 17:40:27 +01:00
146323a7f8 [groupon] Add extractor (Fixes #4386) 2014-12-12 17:39:33 +01:00
57e086dcea [ebaumsworld] Modernize 2014-12-12 17:24:05 +01:00
2101f5d4cc release 2014.12.12.4 2014-12-12 17:18:22 +01:00
cc8c9281e6 [downloader/common] Do not use classic int division 2014-12-12 17:17:09 +01:00
cf372f0778 Merge remote-tracking branch 'SyxbEaEQ2/rate-limit' 2014-12-12 17:16:13 +01:00
34bc0ae667 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-12 17:12:25 +01:00
2865cf0419 Deprecate --auto-number (Closes #2704) 2014-12-12 17:11:53 +01:00
58c1f6f0a7 [nbc] Fix extraction (Closes #4441) 2014-12-12 22:10:32 +06:00
7c7a0d395c Remove unused imports 2014-12-12 17:07:39 +01:00
8bdcb436f9 [test_unicode_literals] Fix test 2014-12-12 17:06:52 +01:00
ff815fe65a Download playlist items in reverse order
Series of videos are typically uploaded to YouTube playlists in
chronological order.  By default, these videos are downloaded
latest-to-earliest; this is great for seeing the latest videos in a
series, but prevents streaming video in the order that the videos were
produced.  Add an option to download videos in reverse order,
earliest-to-latest.

Conflicts:
	youtube_dl/YoutubeDL.py
	youtube_dl/__init__.py
2014-12-12 16:56:29 +01:00
da3a2d8137 release 2014.12.12.3 2014-12-12 16:47:38 +01:00
13dcfd41bd [CONTRIBUTING.md] Remove the section about embedding; that is not applicable for youtube-dl contributors 2014-12-12 16:47:22 +01:00
e56190b378 [Makefile] Add CONTRIBUTING.md (Fixes #2984) 2014-12-12 16:42:40 +01:00
a79553f39f Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-12 16:41:12 +01:00
b3efb3ebae [README] More concise and nicer bug reporting instructions 2014-12-12 16:40:37 +01:00
68d301ffd4 [giantbomb] Add extractor (Closes #4432) 2014-12-12 21:23:42 +06:00
3b0bec8d11 release 2014.12.12.2 2014-12-12 15:56:45 +01:00
412c617d0f [cnet] Update to new theplatform infrastructure (Fixes #2736) 2014-12-12 15:55:55 +01:00
751536f5c8 [goldenmoustache] Remove view count
view count is not present anymore, so we can't extract it.
2014-12-12 13:09:55 +01:00
025f30ba38 [channel9] Do not return compat_list results anymore 2014-12-12 13:07:43 +01:00
0d2fb1d193 [helsinki] Fix extraction 2014-12-12 13:03:16 +01:00
82b34105d3 [goshgay] Fix extraction 2014-12-12 12:55:13 +01:00
73aeb2dc56 [goshgay] Modernize 2014-12-12 12:44:50 +01:00
c6973bd412 [compat] Simplify kwarg detection code
This enables nuitka to compile youtube-dl.
2014-12-12 12:42:35 +01:00
f8780e6d11 Merge remote-tracking branch 'grompe/patch-1' 2014-12-12 11:35:04 +01:00
e2f89ec7aa Revert "[utils] Work around PyPy stupidity with Windows DLLs (Fixes #4392)"
This reverts commit 16040f46d6.
2014-12-12 11:33:55 +01:00
62651c556a [howstuffworks] Parse only once, but right (#4383) 2014-12-12 04:23:34 +01:00
bf94e38d3d Merge remote-tracking branch 'Tithen-Firion/hsw-update' 2014-12-12 04:10:55 +01:00
4f97852316 Remove unused imports 2014-12-12 04:09:32 +01:00
16040f46d6 [utils] Work around PyPy stupidity with Windows DLLs (Fixes #4392) 2014-12-12 04:01:08 +01:00
d068ba24f3 release 2014.12.12.1 2014-12-12 03:34:33 +01:00
f5e43bc695 [vine] Provide alt_title (Fixes #4448) 2014-12-12 03:34:28 +01:00
6a5308ab49 release 2014.12.12 2014-12-12 03:02:56 +01:00
63e0f29564 [vine] Modernize 2014-12-12 02:59:52 +01:00
42bdd9d051 [cinchcast] Add new extractor (Fixes #4428) 2014-12-12 02:57:36 +01:00
4e40de6e2a Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-12 02:14:31 +01:00
0fa2b899d1 [Makefile] remove *.info.json in clean target 2014-12-12 02:14:04 +01:00
f17e4c9c28 [screenwavemedia] Simplify (#3766) 2014-12-12 02:11:58 +01:00
807962f4a1 [pornhd] Adapt to new sources scheme (Fixes #4446) 2014-12-11 23:50:25 +01:00
9c1aa1d668 [mixcloud] Fix metadata extraction (fixes #4443) 2014-12-11 23:16:40 +01:00
69f491f14e Merge remote-tracking branch 'fstirlitz/master' 2014-12-11 17:11:25 +01:00
cb007f47c1 release 2014.12.11 2014-12-11 17:08:31 +01:00
9abd500a74 [zdf:channel] Simplify (#4427) 2014-12-11 17:07:59 +01:00
cf68bcaeff Merge remote-tracking branch 'akretz/master' 2014-12-11 16:35:45 +01:00
cbe2bd914d [youtube] Amend test 2014-12-11 16:34:37 +01:00
75111274ed [youtube] Do not warn if DASH manifest is missing (#4442) 2014-12-11 16:33:28 +01:00
624dcebff6 [youtube] Make category optional (#4442) 2014-12-11 16:32:48 +01:00
9684f17cde Merge remote-tracking branch 'akretz/youtube_fix' 2014-12-11 16:28:10 +01:00
e52a40abf7 [youtube] Add test case for #4431 2014-12-11 16:28:07 +01:00
0daa05961b Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-11 16:23:01 +01:00
158731f83e [tvplay] Don't raise an exception if is_geo_blocked is True
Videos which return `is_geo_blocked' to be True can actually be downloaded from
the country to which the video is restricted
2014-12-11 17:07:50 +02:00
24270b0301 [youtube] The case that 'url_encoded_fmt_stream_map' or 'adaptive_fmts' is the empty string is handled accordingly (fixes #4431) 2014-12-11 16:00:46 +01:00
3c1b81b957 [ntv] Rename flash_ver to flash_version in the format dict
RTMP downloader uses `flash_version`
2014-12-11 16:58:45 +02:00
45c24df512 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-11 15:27:54 +01:00
bf671b605e [behindkink] Remove superfluous whitespace 2014-12-11 20:09:52 +06:00
09c82fbc9a [behindkink] Simplify 2014-12-11 20:06:19 +06:00
3bca0409fe Merge pull request #4440 from 5moufl/behindkink-fix
[BehindKink] update
2014-12-11 19:58:31 +06:00
d6f78a354d [BehindKink] Replace test
Old one is not accessible anymore
2014-12-11 14:26:59 +01:00
e0b9d47387 [BehindKink] Update URL extraction 2014-12-11 14:25:26 +01:00
f8795e102b [utils] Add "yesterday" as a date keyword 2014-12-11 10:29:30 +01:00
4bb4a18876 [youtube] Fix imports 2014-12-11 10:08:17 +01:00
8560c61842 [zdf] Add support for channels 2014-12-10 17:29:03 +01:00
a81bbebf44 [smotri:broadcast] Fix extraction 2014-12-10 20:22:49 +06:00
72e3ffeb74 release 2014.12.10.3 2014-12-10 15:19:08 +01:00
2fc9f2b41d [facebook] Make thumbnail and duration optional
Fixes #4425.
Looks like both properties aren't given to us anymore. For now, just fall back to not returning them.
2014-12-10 15:18:36 +01:00
5f3544baa3 release 2014.12.10.2 2014-12-10 14:39:06 +01:00
da27660014 [youtube] Pass in all variables to DASH manifest (Fixes #4424) 2014-12-10 14:39:00 +01:00
b8a6114309 release 2014.12.10.1 2014-12-10 13:21:49 +01:00
774e208f94 [youtube] Handle missing DASH manifest (Fixes #4421, fixes #4420) 2014-12-10 13:21:24 +01:00
f20b52778b release 2014.12.10 2014-12-10 12:21:40 +01:00
83e865a370 Fix PEP8 issue E713 2014-12-09 23:11:26 +01:00
b89a938687 [bet] Add extractor (Closes #4416) 2014-12-09 22:29:01 +06:00
e89a2aabed [extractor/common] Add generic SMIL formats extraction routine 2014-12-09 22:28:28 +06:00
f58766ce5c [extractor/common] Document ie_key in url results 2014-12-09 10:58:06 +01:00
15644a40df Merge pull request #4395 from cryptonaut/issue2883
Handle --get-url with merged formats (fixes #2883)
2014-12-08 17:21:56 +01:00
d4800f3c3f Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-08 17:17:31 +01:00
09a5dd2d3b [bliptv] Add support for audio-only files (Fixes #4404) 2014-12-08 17:17:22 +01:00
819039ee63 [tvigle] Update test and modernize 2014-12-08 22:03:02 +06:00
ce36339575 add teamfourstar.com support 2014-12-08 17:01:22 +01:00
684712076f add direct screenwavemedia.com URL support 2014-12-08 17:01:22 +01:00
603c92080f [nhl] Make sure we add '_sd' before the extension (fixes #4397)
'.replace' would find the first dot in the path.
2014-12-07 11:26:07 +01:00
16ae61f655 Handle --get-url with merged formats (fixes #2883)
Outputs one URL per line
2014-12-06 12:55:07 -08:00
0ef4d4ab7e Credit @akretz for prosiebensat1 playlist support (#4394) 2014-12-07 01:48:45 +06:00
4542535f94 Merge branch 'akretz-master' 2014-12-07 01:47:09 +06:00
6a52eed80e [prosiebensat1] Improve and simplify 2014-12-07 01:46:44 +06:00
acf5cbfe93 [extractor/common] Add description to playlist_result 2014-12-07 01:46:30 +06:00
8d1c8cae9c [prosiebensat1] Fix broken tests 2014-12-06 19:21:05 +01:00
c84890f708 [prosiebensat1] Add support for playlists (fixes #4357) 2014-12-06 19:05:22 +01:00
6d0886204a [radio.de] Add support for radio.de websites (Closes #4393) 2014-12-06 23:01:52 +06:00
04d02a9d57 [twitch] Add login support (#3986) 2014-12-06 21:24:20 +06:00
6ac4e8065a Fix utils.py for PyPy on Windows
The line
```python
from __future__ import unicode_literals
```
introduced in commit [ecc0c5ee01](ecc0c5ee01) broke youtube-dl for PyPy on Windows, making it unable to locate WinAPI functions.
Error: "TypeError: function name must be a string or integer"

Adding "b" prefix to strings with WinAPI function names fixes it.
2014-12-06 20:15:41 +07:00
b82f815f37 Allow iterators for playlist result entries 2014-12-06 14:02:19 +01:00
158f8cadc0 [adultswim] PEP8 2014-12-06 14:01:59 +01:00
7d70cf4157 [nba] Remove unused import 2014-12-06 13:59:37 +01:00
6591fdf51f [tagesschau] Look at the right place for download links 2014-12-06 13:59:10 +01:00
47d7c64274 [test_utils] Make test more realistically (#4377) 2014-12-06 12:36:23 +01:00
db175341c7 Credit @cryptonaut for adultswim (#4388) 2014-12-06 12:32:01 +01:00
9ff6772790 [youtube] Modernize 2014-12-06 12:20:54 +01:00
5f9b83944d [ffmpeg] Improve version check and call it from hls (Fixes #4377) 2014-12-06 12:14:26 +01:00
f6735be4da Merge remote-tracking branch 'cryptonaut/adultswim' 2014-12-06 11:55:24 +01:00
6a3e0103bb [nba] Add test for #4387 2014-12-06 11:26:17 +01:00
0b5cc1983e [nba] Modernize 2014-12-06 11:15:25 +01:00
1a9f8b1ad4 [nba] Improve _VALID_URL regex (fixes #4387)
Allows for optional trailing / or /index.html
2014-12-06 01:49:22 -08:00
754f0008ec fix increment operator 2014-12-06 09:20:35 +02:00
7115599121 [adultswim] Updated to work with new site format (fixes #4317) 2014-12-05 21:55:47 -08:00
0df23ba9f9 release 2014.12.06.1 2014-12-06 00:48:34 +01:00
58daf5ebed [youporn] Fix JSON parameter regexp (Fixes #4384) 2014-12-06 00:48:29 +01:00
1a7c6c69d3 release 2014.12.06 2014-12-06 00:43:04 +01:00
045c48847a [tagesschau] Add suppot for sendung (Fixes #4378) 2014-12-06 00:42:43 +01:00
e638e83662 [howstuffworks] Update extractor 2014-12-05 19:46:49 +01:00
90644a6843 [azubu] Add extractor (Closes #4379) 2014-12-05 22:08:30 +06:00
d958fa9ff9 [howstuffworks] Rewrite extractor 2014-12-05 12:21:21 +01:00
ebb6419960 [common] Split _download_json
Add ability for extractor to use _parse_json
2014-12-05 12:21:21 +01:00
122c2f87c1 [tagesschau] Modernize 2014-12-05 10:59:55 +01:00
a154eb3d15 release 2014.12.04.2 2014-12-04 17:43:39 +01:00
81028ff9eb [xminus] Capture description (#4300) 2014-12-04 17:43:34 +01:00
e8df5cee12 [minhateca] Fix duration parsing 2014-12-04 17:35:40 +01:00
ab07963b5c release 2014.12.04.1 2014-12-04 17:02:23 +01:00
7e26084d09 Merge branch 'master' of github.com:rg3/youtube-dl 2014-12-04 17:02:14 +01:00
4349c07dd7 [minhateca] Add extractor (Fixes #4094) 2014-12-04 17:02:05 +01:00
1139a54d9b [foxnews] Add extractor (Closes #4352) 2014-12-04 21:19:08 +06:00
b128c9ed68 [vine:user] Add support for another URL format (Closes #4365) 2014-12-04 20:12:06 +06:00
2415951ead [tvp] Modernize 2014-12-04 14:16:09 +01:00
995ad69c54 [common] Add new parameters for _download_webpage 2014-12-04 14:16:09 +01:00
225e4b9633 [tvp] Remove unnecessary code 2014-12-04 14:16:09 +01:00
9776bc7f57 release 2014.12.04 2014-12-04 08:34:12 +01:00
e703fc66c2 Merge remote-tracking branch 'origin/master'
Conflicts:
	youtube_dl/extractor/audiomack.py
2014-12-04 08:33:37 +01:00
39c52bbd32 [myvidster] Enforce age limit in test 2014-12-04 08:31:55 +01:00
6219802165 Merge remote-tracking branch 'zackfern/myvidster' 2014-12-04 08:30:22 +01:00
8b97115358 Credit @zackfern for foxgay (#4371) 2014-12-04 08:28:41 +01:00
810fb84d5e pep8 and minor beautification all around 2014-12-04 08:27:40 +01:00
5f5e993dc6 [bbccouk] Remove unused import 2014-12-04 08:22:53 +01:00
191cc41ba4 [foxgay] Add thumbnail to test definition 2014-12-04 08:22:20 +01:00
abe70fa044 [audiomack] Modernize test definition 2014-12-04 08:21:29 +01:00
7f142293df Merge remote-tracking branch 'zackfern/foxgay' 2014-12-04 08:20:01 +01:00
d4e06d4a83 [options] Standardize mentoined configuration file location (Fixes #4367) 2014-12-04 07:57:18 +01:00
ecd7ea1e6b [myvidster] Added support for Myvidster 2014-12-03 22:22:36 -08:00
b92c548693 [foxgay] Initial support 2014-12-03 20:22:48 -08:00
6ce2c6783b [tvp] Add extractor 2014-12-04 05:14:09 +01:00
29f400b97d [tvp] Update extractor 2014-12-04 02:54:25 +01:00
eecd6a467d [vgtv] Update tests 2014-12-04 01:34:24 +01:00
dce2a3cf9e [break] Remove md5sum from test 2014-12-04 01:33:30 +01:00
9095aa38ac [audiomack] Update test 2014-12-04 00:42:01 +01:00
0403b06985 [soundcloud] Improve_VALID_URL
Add support for links from Audiomack
2014-12-04 00:42:01 +01:00
de9bd74bc2 [ted] Fix type_watch links extraction 2014-12-03 21:17:11 +06:00
233d37fb6b [brightcove] Make sure that the 'ext' variable is set (fixes #4360) 2014-12-03 13:25:49 +01:00
c627f7d48c release 2014.12.03 2014-12-03 12:15:34 +01:00
163c8babaa [nhl] Simplify 2014-12-03 00:08:26 +01:00
6708542099 Merge branch 'master' of https://github.com/akretz/youtube-dl 2014-12-03 00:00:05 +01:00
ea2ee40357 [nhl.com:videocenter] Don't match url with 'id=*' before 'catid' in the query
Since the order extractors are added is not defined, it would match instead of NHLIE.
2014-12-02 23:56:30 +01:00
62d8b56655 [nhl] Support videos which don't have mp4-extension (fixes #4348) 2014-12-02 23:26:37 +01:00
c492970b4b [rts] Improve _VALID_URL 2014-12-02 22:24:47 +06:00
ac5633592a [24video] Add extractor (Closes #4350) 2014-12-02 22:23:23 +06:00
706d7d4ee7 [YoutubeDL] Avoid negative timestamps on Windows 2014-12-02 21:18:07 +06:00
752c8c9b76 [rts] Improve _VALID_URL 2014-12-02 20:53:19 +06:00
b1399a144d [rts] Add support for the new URL format and extract display id (Closes #4349) 2014-12-02 20:45:43 +06:00
05177b34a6 [rutube] Extract m3u8 formats (fixes #3984) 2014-12-01 18:20:36 +01:00
c41a9650c3 [youtube] Extract framerate from the dash manifest
Not all videos have 60 fps, for example they can have 48 fps.
2014-12-01 17:36:12 +01:00
df015c69ea release 2014.12.01 2014-12-01 17:28:34 +01:00
1434bffa1f [tunein] Use station API 2014-12-01 18:10:15 +02:00
94aa25b995 Credit @Tithen-Firion for the myspace changes (#4341) 2014-12-01 16:15:09 +01:00
d128cfe393 [slideshare] Fix description extraction 2014-12-01 20:18:42 +06:00
954f36f890 [myspace] Cleanup 2014-12-01 00:10:12 +01:00
19e92770c9 [myspace] Replace removed test video and fix the others 2014-12-01 00:10:12 +01:00
95c673a148 [myspace] Add extractor for albums 2014-12-01 00:10:12 +01:00
a196a53265 [myspace] Update tests 2014-12-01 00:10:12 +01:00
3266f0c68e [myspace] Redirect to other extractors
There are many songs just linked from Vevo/YouTube to MySpace.
Vevo example: https://myspace.com/threedaysgrace/music/song/animal-i-have-become-28400208-28218041
YouTube example: https://myspace.com/starset2/music/song/first-light-95799905-106964426
2014-12-01 00:10:12 +01:00
1940fadd53 [myspace] Handle non-playable songs
I'm adding this because sometimes there is a song page, but you cannot play it.
Example: https://myspace.com/starset2/music/song/let-it-die-maniac-agenda-remix-bonus-track-95799916-106964439
It will be useful for downloading whole album with songs like this.
2014-12-01 00:10:11 +01:00
03fd72d996 [myspace] Add more data to info dict
`uploader` is an artist
`playlist` is an album
2014-12-01 00:10:11 +01:00
f2b44a2513 [myspace] Use player_url for faster download
It keeps reconnecting without it. Download time decreased from 7+ minutes to 25 seconds for me.
2014-12-01 00:10:11 +01:00
c522adb1f0 [youtube] Add a normal age-gate test video 2014-11-30 21:45:49 +01:00
7160532d41 [youtube] Simplify code for getting the dash manifest url
video_info contains now the 'ytplayer.config.args' dictionary
2014-11-30 21:07:50 +01:00
4e62ebe250 [youtube] Try to extract the video_info from the webpage before requesting the 'get_video_info' pages
The YouTube player doesn't seem to use them except for embedded videos, so we can skip a network request.
But they still provide better error mesagges (for removed videos for example).
2014-11-30 20:56:32 +01:00
4472f84f0c [test/test_subtitles] Update checksum for vimeo subtitle file 2014-11-30 19:42:54 +01:00
b766eb2707 [youtube] Update test 2014-11-30 19:18:39 +01:00
10a404c335 [youtube] Add format 313 (fixes #4339) 2014-11-30 18:56:14 +01:00
c056efa2e3 [bbccouk] Fix extraction (#4104, #4214) 2014-11-30 22:37:56 +06:00
283ac8d592 Merge pull request #4338 from t0mm0/x-minus-fix
[xminus] update tkn extraction regex
2014-11-30 17:11:05 +01:00
313d4572ce [xminus] update tkn extraction regex 2014-11-30 16:04:04 +00:00
42939b6129 [youtube] Use a cookie for seeting the language
This way, we don't have to do an aditional request
2014-11-30 00:03:59 +01:00
37ea8164d3 [youtube] Don't confirm age when initializing
It seems that all the videos with age restriction use now the age gate method, which doesn't require any confirmation.
2014-11-29 23:46:39 +01:00
8c810a7db3 Merge pull request #4333 from ymln/bliptv-fixes
[bliptv] Fix some videos not downloading
2014-11-29 20:20:45 +01:00
248a0b890f [bliptv] Fix \n\n at the end of real_url
See https://github.com/rg3/youtube-dl/issues/3544#issuecomment-53166516
2014-11-29 19:17:56 +02:00
96b7c7fe3f [bliptv] Fix resolution of lookup id in some videos
In some videos (for example, http://blip.tv/play/gbk766dkj4Yn) resolving
lookup id would fail, because page at
http://blip.tv/play/gbk766dkj4Yn.x?p=1 would have no "config.id" in
it. Fixed by requesting different URL and inspecting the URL which the
client is redirected to.
2014-11-29 19:17:56 +02:00
e987e91fcc [playvid] Capture and output error message 2014-11-29 22:16:35 +06:00
cb6444e197 [noco] Add support for multi language videos (Closes #4326) 2014-11-28 20:38:47 +06:00
93b8a10e3b release 2014.11.27 2014-11-27 15:44:49 +01:00
4207558e8b [buzzfeed] Add support for more video types (#4259) 2014-11-27 15:44:35 +01:00
ad0d800fc3 release 2014.11.26.4 2014-11-26 22:53:02 +01:00
e232f787f6 [buzzfeed] Add new extractor (Fixes #4259) 2014-11-26 22:52:52 +01:00
155f9550c0 [test/helper] Fix newlines in output of missing test fields 2014-11-26 22:52:28 +01:00
72476fcc42 release 2014.11.26.3 2014-11-26 22:08:30 +01:00
29e950f7c8 release 2014.11.26.2 2014-11-26 22:06:27 +01:00
7c8ea53b96 release 2014.11.26.1 2014-11-26 22:01:06 +01:00
dcddc10a50 [test_unicode_literals] Arm unicode_literals check
From now on, the line

from __future__ import unicode_literals

should be contained in every single Python file lest we run into any more 2.x/3.x issues.
Going forward, we're likely to develop on 3.x only and would likely miss subtle bugs otherwise.
2014-11-26 20:01:22 +01:00
a1008af412 [gorillavid] Update IE_DESC 2014-11-27 00:24:19 +06:00
61c0663c1e [udemy] Generalize download json and fix login 2014-11-26 21:25:43 +06:00
81a7a521c5 [gorillavid] Remove unused import 2014-11-26 21:02:46 +06:00
e293711802 [udemy] Set session cookies to API requests (Closes #4124, closes #4219, closes #4308) 2014-11-26 21:00:18 +06:00
ceb3367320 [gorillavid] Generalize extraction with countdown timeout and support faststream.in (Closes #4297) 2014-11-26 20:02:40 +06:00
a03aaaed2e Declare Python 3.2 compatibility 2014-11-26 13:08:42 +01:00
e075a44afb [tests] Remove useless u prefixes 2014-11-26 13:07:32 +01:00
8865bdeb37 Remove useless u prefixes 2014-11-26 13:06:02 +01:00
3aa578cad2 [ffmpeg] Modernize 2014-11-26 13:05:49 +01:00
d3b5101a91 [videopremium] Modernize 2014-11-26 13:03:22 +01:00
5c32110114 [videofyme] Modernize 2014-11-26 13:01:39 +01:00
24144e3b8d [tvp] Modernize 2014-11-26 12:58:53 +01:00
b3034f9df7 [trilulilu] Modernize 2014-11-26 12:56:43 +01:00
4c6d2ff8dc [sohu] Modernize 2014-11-26 12:53:55 +01:00
faf3494894 [redtube] Modernize 2014-11-26 12:52:45 +01:00
535a66ef66 [muzu] Modernize 2014-11-26 12:50:37 +01:00
5c40bba82f [hotnewhiphop] Modernize 2014-11-26 12:45:40 +01:00
855dc479c2 [subtitles] Modernize 2014-11-26 12:43:06 +01:00
0792d5634e [youtube] Remove useless u prefixes 2014-11-26 12:41:53 +01:00
e91cdcae1a [appletrailers] Modernize 2014-11-26 12:41:24 +01:00
27e1400f55 [aparat] Modernize 2014-11-26 12:40:51 +01:00
e0938e7731 [addanime] Modernize 2014-11-26 12:40:05 +01:00
b72823a0a4 [francetv] PEP8 2014-11-26 12:38:20 +01:00
673cf0e773 [update] Remove useless import 2014-11-26 12:37:45 +01:00
f8aace93cd [academicearth] Modernize 2014-11-26 12:35:57 +01:00
80310134e0 [mplayer] Modernize 2014-11-26 12:34:52 +01:00
4d2d638df4 [http] Modernize 2014-11-26 12:27:36 +01:00
0e44f90e18 [hls] Remove useless u porefixes 2014-11-26 12:26:21 +01:00
15938ab67a [update] Modernize 2014-11-26 12:24:57 +01:00
ab4ee31eb1 [utils] remove useless u prefix 2014-11-26 11:50:22 +01:00
b061ea6e9f [compat] Beautify assertion 2014-11-26 11:48:09 +01:00
4aae94f9d0 [YoutubeDL] Remove incorrect documentation 2014-11-26 11:25:43 +01:00
acda92f6bc Clarify --no-playlist documentation (Closes #4309) 2014-11-26 10:51:03 +01:00
ddfd0f2727 release 2014.11.26 2014-11-26 10:46:12 +01:00
d0720e7118 Merge branch 'master' of github.com:rg3/youtube-dl 2014-11-26 10:45:57 +01:00
4e262a8838 [generic] Detect direct video links (Fixes #4149, #4313) 2014-11-26 10:44:39 +01:00
b9ed3af343 [tass] Add extractor (Closes #4296) 2014-11-25 22:24:33 +06:00
63c9b2c1d9 release 2014.11.25.1 2014-11-25 14:34:29 +01:00
65f3a228b1 [generic] Add support for LazyYT embeds (Fixes #4306) 2014-11-25 14:34:19 +01:00
3004ae2c3a Credit @t0mm0 for xminus (#4302) 2014-11-25 12:16:48 +01:00
d9836a5917 release 2014.11.25 2014-11-25 09:56:52 +01:00
be64b5b098 [xminus] Simplify and extend (#4302) 2014-11-25 09:54:54 +01:00
c3e74731c2 [README] Mention _og_search_description (#4304)
Lots of sites do have this meta tag, so just add it to the example.
2014-11-25 09:36:27 +01:00
c920d7f00d [README] Adapt code to new style
Next to every IE will download the webpage first anyways.
2014-11-25 09:23:46 +01:00
0bbf12239c Merge remote-tracking branch 't0mm0/x-minus' 2014-11-25 09:22:33 +01:00
70d68eb46f Credit @MatthewRayfield for tmz (#4304) 2014-11-25 09:17:59 +01:00
c553fe5d29 [tmz] Simplify (#4304) 2014-11-25 09:16:40 +01:00
f0c3d729d7 [tmz] Add new extractor 2014-11-25 02:54:13 -05:00
1cdedfee10 [XMinus] Added new extractor. 2014-11-25 03:25:28 +00:00
93129d9442 release 2014.11.24 2014-11-24 22:56:43 +01:00
e8c8653e9d Merge remote-tracking branch 'origin/master' 2014-11-24 22:52:04 +01:00
fab89c67c5 Credit @ossi96 for bpb (#4298) 2014-11-24 22:47:49 +01:00
3d960a22fa [bpb] Simplify (#4298) 2014-11-24 22:47:23 +01:00
51bbb084d3 Merge remote-tracking branch 'ossi96/bpb' 2014-11-24 22:42:56 +01:00
2c25a2bd29 [tunein] Add new extractor (Closes #4097) 2014-11-24 23:15:33 +02:00
355682be01 bpb Add new extractor 2014-11-24 20:02:00 +01:00
00e9d396ab [francetv] Use the m3u8 manifest for georestricted videos (closes #3963)
Generating the correct urls for the f4m segments seems to require a lot of work.
Also raise an error if the video is not available from your location.
2014-11-24 19:49:43 +01:00
14d4e90eb1 [downloader/__init__] Define proper __all__ 2014-11-23 22:25:12 +01:00
b74e86f48a Fix all PEP8 issues except E501 2014-11-23 22:21:46 +01:00
3d36cea4ac [vk] PEP8 2014-11-23 22:14:27 +01:00
380b822003 Remove outdated transition helper scripts 2014-11-23 22:13:03 +01:00
b66e699877 [myspace] pep8 and modernization 2014-11-23 22:12:18 +01:00
27f8b0994e Merge remote-tracking branch 'jtwaleson/master' 2014-11-23 22:10:26 +01:00
e311b6389a Credit @daohoangson for zingmp3 (#4288) 2014-11-23 22:01:15 +01:00
fab6d4c048 remove useless line, the result is never used 2014-11-23 22:00:35 +01:00
4ffc31033e [zingmp3] Simplify and PEP8 (#4288) 2014-11-23 22:00:25 +01:00
c1777d5cb3 Merge remote-tracking branch 'daohoangson/zing-mp3' 2014-11-23 21:55:51 +01:00
9e1a5b8455 PEP8: applied even more rules 2014-11-23 21:39:15 +01:00
784b6d3a9b Merge remote-tracking branch 'jtwaleson/master' 2014-11-23 21:33:31 +01:00
c66bdc4869 [zingmp3] Added support for songs and albums 2014-11-24 03:25:47 +07:00
2514d2635e PEP8: E225,E227 2014-11-23 21:23:05 +01:00
8bcc875676 PEP8: more applied 2014-11-23 21:20:46 +01:00
5f6a1245ff PEP8 applied 2014-11-23 20:41:03 +01:00
f3a3407226 [youtube] Clarify keywords 2014-11-23 20:09:10 +01:00
598c218f7b [smotri] Adapt to new API and modernize 2014-11-23 23:53:41 +06:00
4698b14b76 [rtlxl] Strip additional dot from video URL (#4115) 2014-11-23 13:28:09 +02:00
835a22ef3f release 2014.11.23.1 2014-11-23 10:51:16 +01:00
7d4111ed14 Provide guidance when called with a YouTube ID starting with a dash.
Reported at https://news.ycombinator.com/item?id=8648121
2014-11-23 10:51:09 +01:00
d37cab2a9d Credit @WillSewell for vk:user (#4233) 2014-11-23 10:12:35 +01:00
d16abf434a [vk] Some PEP8 love 2014-11-23 10:11:52 +01:00
a8363f3ab7 [vk] Clarify test 2014-11-23 10:11:04 +01:00
010cd3a3ee Merge remote-tracking branch 'WillSewell/vk-playlists' 2014-11-23 10:09:45 +01:00
b9042def9d release 2014.11.23 2014-11-23 09:59:42 +01:00
aa79ac0c82 [youtube] Support controversy videos (Fixes #4275) 2014-11-23 09:59:02 +01:00
88125905cf Credit @nulloz for telebruxelles (#4270) 2014-11-23 09:49:23 +01:00
dd60be2bf9 [telebruxelles] Simplify (#4270) 2014-11-23 09:44:42 +01:00
119b3caa46 Merge remote-tracking branch 'nulloz/telebruxelles' 2014-11-23 09:38:18 +01:00
49f0da7ae1 [rtlxl] Use unencrypted m3u8 streams (#4115) 2014-11-22 21:06:45 +02:00
2cead7e7bc telebruxelles Add new extractor 2014-11-22 13:34:29 +01:00
9262867e86 [vk.com] Added newline at the end of the file. 2014-11-21 23:25:05 +00:00
b9272e8f8f [vk.com] Removed redundant log message -- this information is already being logged. 2014-11-21 23:22:52 +00:00
021a0db8f7 [vk.com] Simplified the page_id acquisition by using the id matched in the URL earlier on. 2014-11-21 23:22:44 +00:00
e1e8b6897b [vk.com] Updated the extract_videos_from_page function with a much simpler 1-liner. 2014-11-21 23:16:12 +00:00
53d1cd1f77 [vk.com] Updated the _VALID_URL regex for the playlist IE. Removed optional m, and named the id group. 2014-11-21 23:03:31 +00:00
cad985ab4d [vk.com] Updated the description to include vk.com. 2014-11-21 23:00:43 +00:00
c52331f30c [vk.com] Updated a test video that has been removed, and added a comment for others to update two other test videos that are also now removed. 2014-11-21 23:00:33 +00:00
42e1ff8665 [vk.com] Added upload_date variable to the test cases that still work. 2014-11-21 23:00:17 +00:00
2c64b8ba63 release 2014.11.21.1 2014-11-21 22:47:23 +01:00
42e12102a9 [YoutubeDL] Fix multi_video check 2014-11-21 22:39:57 +01:00
6127693ed9 [folketinget] Add extractor (Fixes #4262) 2014-11-21 22:36:24 +01:00
71069d2157 [sztv] Remove useless determine_ext call 2014-11-21 22:03:29 +01:00
f3391db889 [sztvhu] Modernize 2014-11-21 22:02:16 +01:00
9b32eca3ce [generic] Add support for single quotes in HTML5 videos (Fixes #4265) 2014-11-21 22:01:25 +01:00
ec06f0f610 release 2014.11.21 2014-11-21 10:41:18 +01:00
e6c9c8f6ee Merge pull request #4261 from tinybug/patch-4
Update jsinterp.py
2014-11-21 10:41:02 +01:00
85b9275517 Update jsinterp.py
http://s.ytimg.com/yts/jsbin/html5player-zh_HK-vfl1NK6PR/html5player.js 
fix raise ExtractorError
2014-11-21 17:09:22 +08:00
dfd5313afd [YoutubeDL] Support new _type multi_video 2014-11-21 00:25:46 +01:00
be53e2a737 [blip.tv:user] Modernize and add a test 2014-11-21 00:25:13 +01:00
a1c68b9ef2 Merge remote-tracking branch 'origin/master' 2014-11-21 00:17:58 +01:00
4d46c1c68c [brightcove] Improve error message detection (#4256) 2014-11-20 18:44:54 +01:00
d6f714f321 [brightcove] Remove the namespace from the BrightcoveExperience html object 2014-11-20 18:37:08 +01:00
8569f3d629 [vh1] Modernize 2014-11-20 16:51:33 +01:00
fed5d03260 [extractor/common] Document _type values (Motivated by #4254) 2014-11-20 16:47:59 +01:00
6adeffa7c6 [comedycentral] Modernize 2014-11-20 16:36:53 +01:00
b244b5c3f9 remove unused imports 2014-11-20 16:36:13 +01:00
f42c190769 [stanfordoc] Modernize 2014-11-20 16:34:54 +01:00
c9bf41145f [YoutubeDL] Warn if an extractor returns compat_list 2014-11-20 16:29:31 +01:00
5239075bb6 [mtv] Return a proper playlist result (#4254) 2014-11-20 16:25:19 +01:00
84437adfa3 release 2014.11.20.1 2014-11-20 12:20:57 +01:00
732ea2f09b [utils] Improve update on error message somewhat
We still may want to implement a bulletproof check for the current version, and a better place to add this message so that it works for all kind of other errors too.
2014-11-20 12:14:30 +01:00
aff2f4f4f5 [arte] Clean up format sorting mess
We now use our standard sorting facilities. As a side effect, it's finally possible to download German videos from French URLs and vice versa.
2014-11-20 12:06:35 +01:00
3b9f631c41 release 2014.11.20 2014-11-20 08:55:56 +01:00
3ba098a6a5 Merge pull request #4247 from ivan/info-json
Fix #4246 and #4244 .info.json bugs
2014-11-20 08:16:42 +01:00
0cd64bd077 Copy the mtime from the oldest source file to the file created by ffmpeg
Fixes #4245
2014-11-20 06:39:07 +00:00
1394646a0a Fix "ERROR: Cannot write metadata to JSON file" on Windows
Fixes #4246
2014-11-20 06:26:34 +00:00
61ee5aeb73 Fix UnicodeEncodeError with --write-info-json on Python 2.7 + Windows
Fixes #4244
2014-11-20 06:26:34 +00:00
07e378fa18 [compat] correct OptionGroup invocation for Python 3 (fixes #4243) 2014-11-20 07:21:12 +01:00
e07e931375 Work around 2.7.0 deficencies (Fixes #4223) 2014-11-19 18:21:58 +01:00
480b7c32a9 [rtlxl] Fix format order 2014-11-19 01:21:02 +02:00
f56875f271 [test/test_compat] Restore the old value of the HOME environment variable
If the test was run before the YoutubeIE tests (for example by running
"nosetests -v test/test_compat.py test/test_download.py -m 'Youtube_1|compat_expand'"),
it wrote the signatures cache to the 'C:\Documents and Settings\тест\Application Data' folder.
It failed due to a problem in the cache code and the write_json_file function (fixed in f03e33b89a622af13fa5275c46b63aaa4814c499)
2014-11-19 00:02:24 +01:00
92120217eb [cache] Fix writing to paths with unicode characters
* Use "compat_getenv"
* "write_json_file" now expects the filename to be a string
2014-11-19 00:02:24 +01:00
37eddd3143 [rtlxl] Use m3u8 streams instead of f4m (#4115, #4118) 2014-11-19 00:26:44 +02:00
02a12f9fe6 [vk] date_added is now extracted from the video page. 2014-11-18 20:19:56 +00:00
6fcd6e0e21 [vk] Updated the regex for matching user video pages. It now matches optional URL parameters too. 2014-11-18 19:34:12 +00:00
0857baade3 [youtube] Add webm audio formats (Fixes #4229) 2014-11-18 11:08:37 +01:00
469d4c8968 [vk] Added a new information extractor for pages that are a list of a user\'s videos on vk.com. It works in a same way to playlist style pages for the YT information extractors. 2014-11-17 17:53:34 -05:00
23ad44b57b [youtube] Better error message for DASH manifest 2014-11-17 20:12:52 +01:00
f48d3e9bbc [swfinterp] Improve undefined representation 2014-11-17 08:02:48 +01:00
fbf94a7815 [swfinterp] Add more builtins and improve static variables 2014-11-17 07:54:06 +01:00
1921b24551 [swfinterp] Add support for more complicated constants 2014-11-17 07:31:22 +01:00
28e614de5c [utils] Remove stray u' 2014-11-17 07:16:12 +01:00
cd9ad1d7e8 [swfinterp] Basic support for constants (only ints for now) 2014-11-17 07:14:02 +01:00
162f54eca6 [swfinterp] Implement bitand and pushshort operators 2014-11-17 05:08:39 +01:00
33a266f4ba [swfinterp] Implement charCodeAt 2014-11-17 05:03:46 +01:00
6b592d93a2 [swfinterp] Formalize built-in classes 2014-11-17 04:54:54 +01:00
4686ae4b64 [swfinterp] Implement various opcodes 2014-11-17 04:45:12 +01:00
8d05f2c16a [swfinterp] Add support for void methods 2014-11-17 04:36:23 +01:00
a4bb83956c [swfinterp] Implement pushtrue and pushfalse opcodes 2014-11-17 04:29:34 +01:00
eb5376044c [swfinterp] Implement equals opcode 2014-11-17 04:27:51 +01:00
3cbcff8a2d [swfinterp] Implement String basics 2014-11-17 04:25:10 +01:00
e983cf5277 [swfinterp] Interpret yet more opcodes 2014-11-17 04:00:41 +01:00
0ab1ca5501 [swfinterp] Better error message 2014-11-17 03:53:32 +01:00
4baafa229d [swfinterp] Intepret more multinames 2014-11-17 03:46:23 +01:00
7f3e33a147 [swfinterp] Implement member assignment 2014-11-17 01:27:34 +01:00
b7558d9881 [swfinterp] Allow function patching 2014-11-17 01:27:15 +01:00
a0f59cdcb4 [tests] Modernize 2014-11-16 15:17:48 +01:00
a4bc433619 [__init__] Modernize 2014-11-16 15:08:34 +01:00
b6b70730bf [downloader/common] Modernize 2014-11-16 15:06:59 +01:00
6a68bb574a [eporner] Fix duration (Closes #4188) 2014-11-16 14:55:22 +01:00
0cf166ad4f release 2014.11.16 2014-11-16 00:51:46 +01:00
2707b50ffe [spiegel] Correct handling of redirects to spiegel.tv (Closes #4211) 2014-11-16 00:51:31 +01:00
939fe70de0 [spiegeltv] Match hash-style URLs (Closes #4210) 2014-11-16 00:40:09 +01:00
89c15fe0b3 [spiegeltv] Modernize 2014-11-16 00:33:51 +01:00
ec5f601670 [utils] Fix "write_json_file" for unicode names in python 2.x (fixes #4125) 2014-11-15 22:00:32 +01:00
8caa0c9779 [bliptv] Fix the resolve of lookup ID (Closes #4197) 2014-11-15 16:56:04 +02:00
e2548b5b25 release 2014.11.15.1 2014-11-15 15:21:50 +01:00
bbefcf04bf [goldenmoustache] Fix title (Closes #4203) 2014-11-15 15:21:34 +01:00
c7b0add86f [compat] Work around kwargs bugs in old 2.6 Python releases (Fixes #3813) 2014-11-15 15:17:19 +01:00
a0155d93d9 release 2014.11.15 2014-11-15 11:01:54 +01:00
00d9ef0b70 [mailru] Adapt to new data format (Fixes #4201) 2014-11-15 11:01:17 +01:00
0cc8888038 [crunchyroll] Remove NOP code (#2782) 2014-11-15 00:34:03 +01:00
c735450e07 release 2014.11.14 2014-11-14 22:27:56 +01:00
71f8c7ce7a [mtvservices:embedded] Improve config url (fixes #4092) 2014-11-14 19:02:18 +01:00
5fee0eeac0 [ComedyCentralShows] Use the rtmp urls transform function from the MTV IE (fixes #3364)
It produces the right mp4 urls, so we stop prefering the rtmp urls.
2014-11-14 18:36:04 +01:00
eb4157fd17 [utils] Fix struct.pack call on very old Python versions (#4181) 2014-11-14 00:39:32 +01:00
69ede8ef81 release 2014.11.13.3 2014-11-13 16:28:24 +01:00
609a61e3e6 [npo] Improve npo.nl (Fixes #4173) 2014-11-13 16:28:05 +01:00
bf951c5e29 release 2014.11.13.2 2014-11-13 16:12:54 +01:00
af63fed7d8 [generic] Add support for livestream embeds (Fixes #4185) 2014-11-13 16:12:51 +01:00
68d1d41c03 Credit @yaccz for freevideo (#4131) 2014-11-13 15:59:48 +01:00
3deed1e91a [freevideo] Simplify and raise error for foreigners (Fixes #4131) 2014-11-13 15:59:22 +01:00
11b28e93d3 Merge remote-tracking branch 'yaccz/add-extractor/freevideo' 2014-11-13 15:53:16 +01:00
c3d582985f release 2014.11.13.1 2014-11-13 15:42:48 +01:00
4c0924bb24 [utils] Fix intlist_to_bytes in Python 2 (#4181) 2014-11-13 15:28:42 +01:00
3fa5bb3802 [sexu] Modernize (#4171) 2014-11-13 15:20:49 +01:00
c47ec62b83 Merge remote-tracking branch 'peugeot/sexu' 2014-11-13 15:18:38 +01:00
e4bdb37ec6 [spiegel] Add support for embeds 2014-11-13 15:02:31 +01:00
3e6e4999ca [test/helper] Improve output 2014-11-13 14:55:45 +01:00
0e15e725a0 [spiegel] Modernize 2014-11-13 14:45:17 +01:00
437f68d868 Update sexu.py 2014-11-13 14:02:53 +01:00
d91d124081 fix python 2 test 2014-11-13 13:57:10 +01:00
2d42905b68 release 2014.11.13 2014-11-13 09:57:58 +01:00
cbe71cb41d Merge pull request #4178 from awojnowski/master
Fix YouTube Signature Extraction
2014-11-13 08:24:29 +01:00
894dd8682e Fix YouTube signature extraction. 2014-11-13 00:33:27 -06:00
9e05d039e0 [dailymotion] Fix extraction of vevo videos (fixes #4168) 2014-11-12 23:32:27 +01:00
bbd5f2de5e [sexu] initial support 2014-11-12 20:41:13 +01:00
73689dafbf [tvplay] Fix f4m URL extraction (Closes #4119)
Add query parameters which are needed by AkamaiHD F4M player.
Also, modernize a bit.
2014-11-12 19:26:00 +02:00
4b50ba0989 Credit @xantares for goldenmoustache (#4128) 2014-11-12 15:53:00 +01:00
5ccaddf5b1 [goldenmoustache] Simplify (#4128) 2014-11-12 15:36:59 +01:00
0b201a3134 Merge remote-tracking branch 'xantares/goldenmoustache' 2014-11-12 15:34:31 +01:00
ffe38646ca [funnyordie] Remove test md5sum (Fixes #4113) 2014-11-12 15:33:15 +01:00
b703ab4d7f Merge remote-tracking branch 'michael-k/links' 2014-11-12 15:31:54 +01:00
c6afed48ff [YoutubeDL] guard against strange sys.stdouts 2014-11-12 15:30:26 +01:00
732c848c14 [abc] Update test case
Old video has expired.
2014-11-12 15:26:29 +01:00
9d2a4dae90 [allocine] Update test 2014-11-12 15:26:09 +01:00
7009a9047a [byutv] Update test 2014-11-12 15:24:37 +01:00
498942f187 [test_youtube_signature] Fix import
Broken in commit 8c25f81bee
2014-11-12 15:23:55 +01:00
28465df1ff [youjizz] Modernize (#4131) 2014-11-12 15:19:23 +01:00
ef89dba58f [myspass] Modernize test case 2014-11-12 15:01:52 +01:00
13ba3a6461 [bandcamp:album] Fix test case 2014-11-12 15:00:54 +01:00
8f6ec4bbe6 release 2014.11.12.1 2014-11-12 11:44:26 +01:00
c295490830 [YoutubeDL] Fix bug in the detection of formats that don't contain video (fixes #4150)
If the format requested was not available, we called the method '.get' in None.
2014-11-12 09:42:35 +01:00
eb4cb42a02 [ted] Extract duration (closes #4155) 2014-11-12 09:30:57 +01:00
7a8cbc72b2 release 2014.11.12 2014-11-12 08:46:34 +01:00
2774852c2f Fix MTV/GameTrailers "Bad Request" error
Bugfix for bug #4123 & #4153
2014-11-12 01:10:08 +01:00
bbcc21efd1 [wrzuta] Fallback to mp3 on unknown media type (#4156) 2014-11-11 16:47:54 +02:00
60526d6bcb [wrzuta] Fix audio extension lookup (Closes #4156)
Also, replace deleted test case
2014-11-11 16:23:06 +02:00
1d4df56d09 release 2014.11.09 2014-11-09 22:32:41 +01:00
a1cf99d03a [YoutubeDL] Add playlist_id and playlist_title fields (Fixes #4139) 2014-11-09 22:32:35 +01:00
3c6af203cc [streamcloud] Match URLs without fname (Closes #4144)
Also, modernize a bit.
2014-11-09 22:00:51 +02:00
1a92e086a7 [tapely] Add Referer header (Closes #4138) 2014-11-09 15:01:12 +02:00
519c73f267 Merge pull request #4136 from andikmu/master
fix swrmediathek for new formats.
2014-11-09 12:17:18 +01:00
a6dae6c09c [ndr] Improve video url regex (fixes #4140) 2014-11-09 11:15:50 +01:00
f866e474f3 [YoutubeDL] Don't dowload formats for merging if the first doesn't contain the video (#4132) 2014-11-09 10:59:56 +01:00
8bb9b97c97 Merge remote-tracking branch 'origin/master' 2014-11-09 08:30:12 +01:00
d6fdc38682 fix swrmediathek for new formats. 2014-11-08 15:56:35 +01:00
c2b61af548 [options] Document the syntax for merging formats (closes #3940, closes #4132) 2014-11-08 15:09:04 +01:00
2fdbf27ad8 [niconico:playlist] Use the same video url the webpage uses (closes #4133) 2014-11-08 14:53:23 +01:00
yac
3898c8a7b2 [FreeVideo] Add new extractor 2014-11-08 00:13:28 +01:00
29ed169cd6 [wrzuta] Add mp3 as a possible format (Closes #4126) 2014-11-07 22:53:54 +02:00
b868c972d1 Add support for goldenmoustache.com 2014-11-07 17:44:06 +00:00
9908e03528 Merge pull request #4076 from ghedo/direct_type
[generic] indicate when a direct video has been extracted
2014-11-06 22:23:14 +01:00
1fe8fb8c20 [vice] Re-add extractor (fixes #4120)
The generic extraction no longer works.
2014-11-06 21:44:07 +01:00
5d63b0aa93 [goshgay] Fix title extraction and modernize
Also remove width and height as they are not of the actual video.
2014-11-06 01:19:20 +02:00
4164f0117e [utils] Remove unused import 2014-11-05 23:56:54 +01:00
37aab27808 [brightcove] Extract m3u8 formats (#3541) 2014-11-06 00:14:33 +02:00
6110bbbfdd [niconico] Catch deleted videos (closes #4064) 2014-11-05 19:52:34 +01:00
cde9b380e6 Merge pull request #4110 from nemunaire/channel9-fix
[channel9] Fix extraction
2014-11-05 19:03:24 +01:00
dab647a7b6 [cinemassacre] Keep both extraction approaches and make more robust (Closes #4109) 2014-11-05 21:32:46 +07:00
a316a83d2b [channel9] Fix extraction 2014-11-05 11:23:11 +01:00
81b22aee8b [izlesene] Update test cases and modernize
The timestamp fluctuates with DST.
2014-11-05 01:00:33 +02:00
a80c96eab0 release 2014.11.04 2014-11-04 23:42:09 +01:00
20436c30c9 [youtube] Clarify output 2014-11-04 23:35:34 +01:00
3828505646 [utils] Use a regexp instead of HTMLParser for get_element_by_attribute 2014-11-04 23:33:43 +01:00
11fba1751d [imdb] Simplify 2014-11-04 23:26:23 +01:00
12ea2f30cf [utils] Remove unused get_meta_content function 2014-11-04 23:20:39 +01:00
9c3e870393 [gamespot] Remove unused import 2014-11-04 23:17:43 +01:00
44789f2457 [ustream] Use modern helper function instead of old HTML parser 2014-11-04 23:15:16 +01:00
711ede6e1b [heise] Fix description, thumbnail and format ID 2014-11-04 23:14:16 +01:00
a32f253112 [gamespot] Modernize 2014-11-04 23:04:12 +01:00
94bd361318 [youtube] Skip sts if missing (Fixes #4095, fixes #4103) 2014-11-04 22:45:43 +01:00
acd40f64ed [cnn] Modernize test definitions 2014-11-04 22:25:15 +01:00
766306450d [played] Capture and output error message 2014-11-04 17:34:53 +07:00
e7642ab572 [wimp] Fix video URL regex 2014-11-04 17:13:17 +07:00
bdf9701729 [generic/brightcove] Add a new test case for kijk.nl (#3541) 2014-11-03 23:13:46 +02:00
b5af6fcdad [brightcove] Make _VALID_URL less greedy and check for empty URLs (#3541) 2014-11-03 23:12:24 +02:00
278143df5b [test_compat] Ignore unicode_literals 2014-11-03 19:12:06 +01:00
fdca55fe34 [trutube] Strip title 2014-11-03 20:14:18 +07:00
4f195f55f0 Do not override stdlib html parser 'locatestarttagend' regex (fixes #4081)
'<a href="foo" ><img src="bar" / ></a>' wouldn't be parsed right (the problem is '/ >', '/>' worked fine).
We need to change it in python 2.6 (for example the description of youtube videos wouldn't be extracted).
2014-11-02 19:31:06 +01:00
ac35c26686 [tests] Don't auto init YoutubeDL
It would print the debug headers for each test.
And nose uses a StringIO object for stdout, which in python 2.x doesn't have the 'encoding' attribute.
2014-11-02 17:53:12 +01:00
982a58d049 [README] Replace links to kernel.org with links to git-scm.com
Unlike kernel.org, the documentation at git-scm.com is up to date and
the rest of the git documentation is easily accessible to any git
newby.
2014-11-02 16:07:40 +01:00
42f7d2f588 [test_download] Fix import 2014-11-02 11:46:12 +01:00
39f0a2a6b7 [test_swfinterp] Correct compilation on modern mxmlc versions 2014-11-02 11:41:33 +01:00
ecc0c5ee01 [utils] Modernize 2014-11-02 11:37:49 +01:00
451948b28c [compat] Modernize 2014-11-02 11:36:29 +01:00
baa708036c [compat] Fix imports 2014-11-02 11:26:40 +01:00
8c25f81bee [util] Move compatibility functions out of util
utils is large enough without these compatibility functions.

Everything that is present in newer versions of Python (i.e. with dev Python it's just an import) goes into compat.py .
Everything else (i.e. youtube-dl-specific helpers) goes into utils.py .
2014-11-02 11:23:42 +01:00
4c83c96795 [YoutubeDL] Include rtmpdump in exe versions -v output 2014-11-02 10:55:36 +01:00
9580711841 [ffmpeg] Move version detection to utils 2014-11-02 10:50:30 +01:00
c30ae9594c release 2014.11.02.1 2014-11-02 10:28:21 +01:00
ffae28ae18 release 2014.11.02 2014-11-02 09:45:51 +01:00
d9116714f2 [cinemassacre] Fix extraction (Closes #4083) 2014-11-02 08:01:14 +07:00
08965906a8 [README] Update FAQ on Ubuntu (#4078) 2014-11-01 19:24:56 +01:00
ccdd0ffb80 [generic] indicate when a direct video has been extracted
Fixes #4052.
2014-11-01 15:34:00 +01:00
5263cdfcf9 [generic] Improve MLB iframe regex 2014-11-01 04:01:58 +07:00
b2a68d14cf [mlb] Improve _VALID_URL (Closes #4063) 2014-11-01 04:01:18 +07:00
6e1cff9c33 [canalplus] Improve and merge with d8 extractor 2014-10-31 21:54:30 +07:00
72975729c8 [canalplus] Tweak extractor to support piwiplus (Closes #4046) 2014-10-31 20:19:30 +07:00
d319948b6a [funnyordie] Add articles URL test 2014-10-31 19:26:56 +07:00
9a4bf889f9 Merge pull request #4069 from anovicecodemonkey/support_funnyordie_articles_urls
[FunnyOrDie] Add support for "/articles/" URLs
2014-10-31 18:25:22 +05:00
2a834bdb21 [FunnyOrDie] Add support for "/articles/" URLs 2014-10-31 21:20:37 +10:30
0d2c141865 [youtube] Detect formats 298 et al as mp4 (Fixes #4066) 2014-10-31 11:13:02 +01:00
5ec39d8b96 release 2014.10.30 2014-10-30 09:53:48 +01:00
7b6de3728a [youtube] Add format 266 (Fixes #4055) 2014-10-30 09:53:43 +01:00
a51d3aa001 [youtube] Add support for formats 302 and 303 (Fixes #4060) 2014-10-30 09:43:11 +01:00
2c8e03d937 Sort formats by fps as well 2014-10-30 09:40:52 +01:00
fbb21cf528 [youtube] Add formats 298, 299 (Fixes #4056) 2014-10-30 09:34:13 +01:00
b8a618f898 [ro220] Fix broken extractor and modernize (#4054) 2014-10-30 01:42:52 +02:00
feb74960eb release 2014.10.29 2014-10-29 23:29:42 +01:00
d65d628613 [crunchycroll] Fix building of ass subtitles (reported in #4019)
Parse the xml document instead of using regexes, otherwise unicode characters are left unescaped.
2014-10-29 21:19:20 +01:00
ac645ac7d0 [generic] Allow soundcloud embeds with additional attributes 2014-10-29 20:27:58 +01:00
7d11297f3f Merge branch 'master' of github.com:rg3/youtube-dl 2014-10-29 20:10:07 +01:00
6ad4013d40 [drtv] Allow fractional timestamps (Fixes #4059) 2014-10-29 20:10:00 +01:00
dbd1283d31 [naver] Capture and output error message (#4057) 2014-10-29 21:50:37 +07:00
c451d4f553 [trutube] Fix extraction 2014-10-29 21:16:10 +07:00
8abec2c8bb [test_utils] Fix compat_getenv and compat_expanduser tests on python 3.x 2014-10-29 11:13:34 +01:00
a9bad429b3 [niconico] Add extractor for playlists (closes #4043) 2014-10-29 11:04:48 +01:00
50c8266ef0 Merge branch 'master' of github.com:rg3/youtube-dl 2014-10-28 23:40:44 +01:00
00edd4f9be [laola1tv] Mark as broken
When the f4m downloader gets live stream support, I expect this to work magically or with very minor changes.
2014-10-28 17:29:27 +01:00
ee966928af [f4m] Support bootstrap URLs 2014-10-28 17:27:41 +01:00
e5193599ec [laola1tv] Add new extractor
The extractor works fine, but the f4m downloader cannot handle the resulting bootstrap information.
2014-10-28 16:51:34 +01:00
01d663bca3 [auengine] Simplify 2014-10-28 15:51:15 +01:00
e0c51cdadc [vk] Generalize errors 2014-10-28 21:35:25 +07:00
9334f8f17a [vk] Handle deleted videos 2014-10-28 21:06:07 +07:00
632256d9ec [wimp] Update video URL regex 2014-10-28 20:35:02 +07:00
03df7baa6a Start documentation on how to embed youtube-dl 2014-10-28 12:54:39 +01:00
3511266bc3 [YoutubeDL] Simplify API of YoutubeDL
Calling add_default_extractors twice should be harmless since the first set of extractors will match.
2014-10-28 12:54:29 +01:00
9fdece5d34 [srmediathek] Choose variable name more wisely 2014-10-28 10:44:47 +01:00
bbf1092ad0 [fktv] Remove unused import 2014-10-28 10:44:17 +01:00
9ef55c5bbc [quickvid] Add new extractor 2014-10-28 10:41:37 +01:00
48a24ab746 [generic] Fix HTML5 video regexp 2014-10-28 10:41:24 +01:00
68acdbda9d Merge remote-tracking branch 'origin/master' 2014-10-28 09:13:23 +01:00
27c542c06f [iconosquare] Simplify 2014-10-28 09:12:28 +01:00
aaa399d2f6 [sphinx] Fix version import 2014-10-27 18:49:48 +02:00
b2e6a1c14c release 2014.10.27 2014-10-27 02:44:07 +01:00
8cc3eba79a [phoenix] Add new extractor (Fixes #4036) 2014-10-27 02:43:59 +01:00
b0fb6d4db1 [ku6] Modernize 2014-10-27 02:32:44 +01:00
81515ad9f6 [extractor/common] Improve m3u8 output 2014-10-27 02:28:37 +01:00
8112d4b284 [lrt] Modernize 2014-10-27 02:27:49 +01:00
bf7aa6301b [fktv] Modernize 2014-10-27 02:26:05 +01:00
aea856621f [zdf] Simplify 2014-10-27 02:14:07 +01:00
f24a5a2faa Merge remote-tracking branch 'olebowle/ard' 2014-10-27 01:36:50 +01:00
ecfe623422 [heise] Fix extraction
Now they use an XML format instead of JSON.
2014-10-27 01:33:51 +01:00
4a6c94288a [kickstarter] Simplify and fix test case 2014-10-27 01:16:18 +01:00
10e3d73472 [nbc] Fix ThePlatform embedded videos 2014-10-27 01:14:17 +01:00
15956b5aa1 [promptfile] Fix check for deleted videos 2014-10-27 00:50:22 +01:00
586f7082ef [francetv] Remove changing md5sum 2014-10-27 00:46:34 +01:00
d6d9186f0d [generic] Fix test title 2014-10-27 00:45:15 +01:00
2e9ff8f362 [gorillavid] Fix test title 2014-10-27 00:44:27 +01:00
6407432333 [Makefile] remove temporary files in clean target 2014-10-27 00:40:07 +01:00
f744c0f398 [test_download] Improve error message 2014-10-27 00:39:39 +01:00
249efaf44b [pornhub] Modernize and fix test definition 2014-10-27 00:33:35 +01:00
8d32abff9e [ruhd] Simplify 2014-10-27 00:20:54 +01:00
94f052cbf4 [syfy] Remove test checksum
We have the minsize test now.
2014-10-27 00:19:15 +01:00
446a03bd96 [ustream:channel] Change test playlist size (Seems to have been limited that way on the website as well) 2014-10-27 00:18:10 +01:00
6009b69f81 [vgtv] Fix test title 2014-10-27 00:16:01 +01:00
3d6047113c [vgtv] Simplify 2014-10-27 00:14:52 +01:00
9dec99303d [vimeo:review] Fix test title 2014-10-27 00:13:40 +01:00
7706927370 [vine:user] Adapt test to changed list size 2014-10-27 00:11:34 +01:00
3adba6fa2a [xtube] Fix test description 2014-10-27 00:08:37 +01:00
f46a8702cc [youtube:playlist] Fix test title 2014-10-27 00:06:47 +01:00
8d11b59bbb [ynet] Remove test md5sums
These fluctuate regularly.
2014-10-27 00:06:00 +01:00
cf501a23d2 [srmediathek] Correct IE_NAME/IE_DESC 2014-10-26 23:23:53 +01:00
2bcae58d46 [srmediathek] New extractor 2014-10-26 23:23:10 +01:00
c9f08154a3 Remove unused imports 2014-10-26 23:13:42 +01:00
526b276fd7 [faz] Modernize 2014-10-26 23:11:15 +01:00
77ec444d9a release 2014.10.26.2 2014-10-26 21:49:52 +01:00
bfc2bedcfc [youtube] Make confirm_age non-fatal (#4042) 2014-10-26 21:49:29 +01:00
83855f3a1f [livestream:original] Fix RTMP parameters (Fixes #4040) 2014-10-26 21:44:29 +01:00
50b51830fb [ffmpeg] Fix typo 2014-10-26 21:31:51 +01:00
3d6eed9b52 release 2014.10.26.1 2014-10-26 21:03:38 +01:00
1a253e134c [ffmpeg] Fix call to ffprobe (Fixes #4041) 2014-10-26 21:03:16 +01:00
6194bb1419 [ffmpeg] Make downloader optional (Fixes #4039) 2014-10-26 21:00:42 +01:00
37d66e7f1e [generic] Correct call to _webpage_read_full_content 2014-10-26 20:58:09 +01:00
70b7e3fbb6 [generic] Add a test case for direct links with broken HEAD (#4032) 2014-10-26 20:49:51 +01:00
0551a02b82 [Rte] Improve extractor 2014-10-27 01:08:51 +05:30
579657ad87 [soundcloud] Set the 'webpage_url' field for each track
For playlists, YoutubeDL would set it to the playlist url.
2014-10-26 19:08:36 +01:00
5f82b129e0 [ffmpeg] Also look into stderr for extracting the version
At least with avconv 11, it will print 'avconv version 11, ..' to stderr, not stdout.
2014-10-26 18:11:31 +01:00
64269e4d01 Move AUTHORS to root (closes #2985) 2014-10-26 18:01:00 +01:00
d481699a7a release 2014.10.26 2014-10-26 17:29:27 +01:00
5894a4f4ee Credit @gabeos for crunchyroll:playlist (#3988) 2014-10-26 17:29:02 +01:00
09e5d6a6e5 [crunchyroll:playlist] Simplify (#3988) 2014-10-26 17:28:09 +01:00
274b12b5a8 Merge remote-tracking branch 'gabeos/crunchyroll-show-playlist' 2014-10-26 17:06:35 +01:00
23be51d8ce [generic] Handle audio streams that do not implement HEAD (Fixes #4032) 2014-10-26 17:05:44 +01:00
488447455d [ffmpeg] Warn if ffmpeg/avconv version is too old (Fixes #4026) 2014-10-26 16:46:34 +01:00
d28b517154 [YoutubeDL] Output avconv/ffmpeg versions if -v is given 2014-10-26 16:31:52 +01:00
a7e97f6db1 [generic] Allow new SWFObject()-style imports
This embed style is used on http://www.bitburger-open.de/ , but that is not included as a test case since the format is likely to be temporary.
2014-10-26 14:15:49 +01:00
639a422d21 Merge branch 'dstftw-compat-getenv-and-expanduser' 2014-10-26 19:58:39 +07:00
f889cea109 Merge branch 'compat-getenv-and-expanduser' of https://github.com/dstftw/youtube-dl into dstftw-compat-getenv-and-expanduser
Conflicts:
	test/test_utils.py
	youtube_dl/__init__.py
2014-10-26 19:56:52 +07:00
1bdeb7be2e Set '--simulate' if any of the printing options is given (fixes #3036)
That's what the help messages say.
Previously it would only set '--skip-download', which would write thumbnail,
descriptions or subtitles if they were requested (for example you may have set
'--write-thumbnail' in the config file).
2014-10-26 13:01:57 +01:00
699151bcb1 Merge branch 'Dineshs91-belgiannational-ie' 2014-10-26 16:48:55 +07:00
911344e5ac [vrt] Improve extractor 2014-10-26 16:48:11 +07:00
03936f6e6d [BelgianNational] corrected indentation 2014-10-26 13:47:41 +05:30
b13ccb1b87 [BelgianNational] New extractor added 2014-10-26 13:35:00 +05:30
f64f8a4662 [sportbox] Remove view count 2014-10-26 08:00:01 +07:00
681b9caa9c [tumblr] Fix extraction (fixes #4029) 2014-10-25 22:42:56 +02:00
0eb9fb9f24 [soundcloud] Modernize and fix tests 2014-10-25 22:32:01 +02:00
9a76f416ce [lrt] Updated test 2014-10-25 13:24:46 +03:00
603821161f Merge branch 'master' of github.com:rg3/youtube-dl 2014-10-25 09:55:05 +02:00
d3c72db894 [audiomack] Simplify 2014-10-25 08:58:03 +02:00
43d9718fb9 [nhl] Improve video URL extraction (Closes #4013) 2014-10-25 13:56:21 +07:00
7fc54e5262 Merge remote-tracking branch 'xavierbeynon/audiomack' 2014-10-25 08:55:12 +02:00
ec9c978481 Credit @winwon for vidzi (#3989) 2014-10-25 08:42:53 +02:00
d36cae46d8 Not directly calling soundcloud extractor anymore 2014-10-24 21:11:46 -05:00
fdfefa1b9c Made changes per phihag 2014-10-24 21:07:01 -05:00
724d031893 release 2014.10.25 2014-10-25 00:41:06 +02:00
63e0be3415 New option --dump-single-json (#4003) 2014-10-25 00:30:57 +02:00
c64ed2a310 [viddler] Use API 2014-10-25 00:11:12 +02:00
cdc5cb7c2b [hark] Modernize 2014-10-24 22:31:55 +02:00
8efd06aa42 [motherless] Simplify 2014-10-24 19:53:48 +02:00
7f9ced64cb Add group videos support for motherless
Modified motherless regular expression.
Previously it matched only URLs like this:
motherless.com/ID
Now it also matches this:
motherless.com/g/group_name/ID
All tests including the newly added one passed.
2014-10-24 21:44:21 +04:00
7608815cc2 Add another motherless test
This test is for videos posted in groups.
URL looks like this:
motherless.com/g/group_name/ID
which is basically the same as
motherless.com/ID
So far this test fails because I haven't improved the extractor yet.
2014-10-24 21:43:27 +04:00
5823eda139 Fix motherless test 0
Apparently motherless no longer serves flv for that video (or maybe even
other videos). So I changed expected extension from flv to mp4 and
changed expected md5 accordingly.
2014-10-24 21:26:32 +04:00
e82c1e9a6e [YoutubeDL] Do not apply playlist info to videos when extract_flat is set (#4003) 2014-10-24 16:13:45 +02:00
1ede5b2481 [glide] Simplify 2014-10-24 15:34:19 +02:00
964ae0a122 Credit @thornomad for glide (#3944) 2014-10-24 15:29:44 +02:00
98e1d28982 Merge remote-tracking branch 'thornomad/glide' 2014-10-24 15:29:03 +02:00
2c26df763c [vidzi] Use proper test case and simplify (#3989) 2014-10-24 15:27:02 +02:00
018e835594 [vidzi] Simplify 2014-10-24 15:17:17 +02:00
e65e06fbe2 [vidzi] Correct order in imports (#3989) 2014-10-24 15:15:54 +02:00
95ee84421e vidzi Add new extractor 2014-10-24 15:15:13 +02:00
2acfe95f58 Credit @capital-G for bild.de (#3983) 2014-10-24 15:12:29 +02:00
b5a14350b9 [bild] Simplify (#3983) 2014-10-24 15:10:32 +02:00
8d81f872fb Merge remote-tracking branch 'capital-G/master' 2014-10-24 15:02:50 +02:00
36f1c90497 release 2014.10.24 2014-10-24 14:48:19 +02:00
057a5206cc Add --flat-playlist option (Closes #4003) 2014-10-24 14:48:12 +02:00
9e9bc793f3 Finished audiomack extractor 2014-10-23 23:54:59 -05:00
25fadd06d0 [Rte] New extractor added 2014-10-24 09:49:01 +05:30
7a47d07c6d [extractor/common] href attribute added 2014-10-24 09:47:39 +05:30
5c565ac9e7 Added init.py initializer 2014-10-23 16:58:11 -05:00
67500bf939 Initial version of audiomack.py 2014-10-23 16:55:39 -05:00
b1edd7a48a [crunchyroll] Correct parsing (Fixes #4014) 2014-10-23 23:25:02 +02:00
34e48bed3b [extractor/common] Added support for f4m manifest Version 2.0 2014-10-24 02:41:10 +05:30
2c63ccec78 [mitele] Fix on python 2.x 2014-10-23 21:26:48 +02:00
f2f2c0c2c6 [generic] Allow --default-search without colon 2014-10-23 21:13:45 +02:00
4661e243f8 release 2014.10.23 2014-10-23 20:21:38 +02:00
f3cd403c2b [telecino] Add coding declaration 2014-10-23 20:21:27 +02:00
ad5f53ac72 [telecinco] Add extractor (closes #4005)
It uses the same extraction process as mitele.es, but with a few small differences.
2014-10-23 20:08:55 +02:00
75da98e9e1 [funnyordie] Fix extraction (Closes #4011) 2014-10-23 23:07:58 +07:00
281d3f1d68 [generic/wistia] Improve regex 2014-10-23 23:03:07 +07:00
6283c10b1c Merge pull request #4009 from Dineshs91/thoughtworks
[thoughtworks] wistia support added
2014-10-23 22:59:53 +07:00
85d7b76586 [thoughtworks] wistia regex modified 2014-10-23 21:23:56 +05:30
2399535fd1 [francetv] Lower preference of direct links (Closes #4010, closes #3947)
Direct links fail with 403, fallback on f4m for now until further investigations.
2014-10-23 22:30:34 +07:00
52cffcb186 [thoughtworks] wistia support added 2014-10-23 20:28:39 +05:30
8f3b5397a7 [cinemassacre] Fix extraction (Closes #4008) 2014-10-23 21:35:51 +07:00
9bbec55255 [pbs] Add support for tabbed frontline videos (Closes #4006) 2014-10-23 20:41:45 +07:00
6b445558ff [youtube] Only set language when necessary
We only need to set the language when we are using a login; otherwise it will always be English.
2014-10-23 00:44:22 +02:00
bfd91588f3 [ard] make rss match more universal 2014-10-22 14:24:53 +02:00
6bf6962062 [francetv] Force m3u8 formats extension to mp4 (Closes #3997) 2014-10-22 01:24:04 +07:00
40bca5f927 [arte.tv] Remove unused import 2014-10-21 21:06:24 +07:00
74214d35c5 [arte.tv:+7] Improve title extraction (Closes #3995) 2014-10-21 20:08:20 +07:00
1b10a011ec Forgot to reverse extracted video urls so they are in correct order for video selection args 2014-10-20 18:38:42 -07:00
5a000b45b3 Don't use report_warning for reporting warnings
In tests warning is converted to error
2014-10-20 18:53:53 +02:00
40b1cbafac Update fc2.py 2014-10-20 18:53:53 +02:00
4231235cda Fix issues with fc2
Fix issues #2912 and #3171
2014-10-20 18:53:53 +02:00
d24a2b20b4 [arte.tv:+7] Use original format ids to avoid duplicates 2014-10-20 20:27:59 +07:00
8230018c20 Added extractor for crunchyroll 'playlists' i.e. series. so that one can, e.g. download all episodes of a series 2014-10-19 22:47:05 -07:00
cc98a3f096 [cnn] Fix for urls ending in '.cnn-ap' (fixes #3985) 2014-10-19 15:14:37 +02:00
ce519b747e added "bild.de" as extractor 2014-10-18 22:15:47 +02:00
16efb3695f [sexykarma] Add support for watchindianporn 2014-10-19 01:53:15 +07:00
4510d14f0a [twitch] Update tests 2014-10-19 01:35:53 +07:00
0f175a932f release 2014.10.18 2014-10-18 20:22:23 +02:00
849b269273 Merge branch 'CkuT-sexykarma' 2014-10-19 00:48:30 +07:00
95fa5fb569 [sexykarma] Improve and simplify 2014-10-19 00:48:05 +07:00
77c3c5c5ed Merge branch 'sexykarma' of https://github.com/CkuT/youtube-dl into CkuT-sexykarma 2014-10-19 00:06:53 +07:00
159444a668 [twitch] Remove superfluous comma 2014-10-18 21:57:24 +07:00
f9befee1f5 [arte.tv:+7] Append media type to format_id (Closes #3967) 2014-10-18 18:14:49 +07:00
9471c44405 [generic] Make sure Wistia embed URLs contain the protocol (Closes #3977)
Also, improve detection (Addresses #3662)
2014-10-18 01:55:21 +03:00
013bfdd84c [twitch] Update tests and minor improvements 2014-10-17 23:46:53 +07:00
46fd0dd5a5 [twitch] Rename extractor and support channel videos 2014-10-17 22:58:18 +07:00
4698f0d858 [vimeo] Improve regex for the config js dict (fixes #3955, fixes #3974)
The javascript code contains assignments of empty dicts with the same variable name
2014-10-17 15:55:58 +02:00
355d074ff9 [twitch] Adapt to new API (Fixes #3946, Fixes #3949, Fixes #3965)
Work in progress
2014-10-16 22:23:35 +07:00
7da224c907 Add categories 2014-10-15 22:34:35 +02:00
1723edb1a5 Few improvements 2014-10-15 20:17:07 +02:00
4740864508 [SexyKarma] Add new extractor 2014-10-15 18:24:32 +02:00
09a42738fc [generic] Correct handling of embedded vimeo players (#3955) 2014-10-15 13:50:53 +02:00
df928d500f release 2014.10.15 2014-10-15 12:39:30 +02:00
a72cbfacf0 [ted] Add support for external videos (fixes #3948) 2014-10-15 12:24:11 +02:00
net
7b61ac3ddf Fix #2310. Play by the 8tracks rules 2014-10-15 06:46:47 +03:00
62a164e713 [mixcloud] Output downloading progress 2014-10-15 00:53:54 +02:00
5f58165def [extractor/common] Fix dumping requests with long file abspath on Windows 2014-10-14 21:43:48 +07:00
a86c73cf80 [glide] Add new extractor
Added an extractor for glide.me shared messages.  Glide is a movile video messaging services.  You can share the link to the messages easily enough and this would allow you to download and save the actual video.
2014-10-13 14:08:29 -07:00
bd4e40df1a [brightcove] Add a test for playlists 2014-10-13 16:26:53 +02:00
1419fafd36 [condenast] Add support for embedded videos (Closes #3929) 2014-10-13 19:59:35 +07:00
9b36dcbd65 release 2014.10.13 2014-10-13 10:12:51 +02:00
2aefb886fa [ffmpeg] Improve format merging (Closes #3935) 2014-10-13 10:12:43 +02:00
72961c2a8a Merge remote-tracking branch 'Dineshs91/KontrTube-fix' 2014-10-13 10:09:57 +02:00
4c1ce987b0 [huffpost] Modernize 2014-10-13 10:08:59 +02:00
8a2300a597 [kontrtube] Fix video title extraction 2014-10-13 10:03:55 +05:30
1cc887cbf0 [youtube] Add support for format 278 2014-10-13 00:09:19 +02:00
203fb43f36 [youtube] Download DASH manifest by default (Closes #3887) 2014-10-13 00:03:08 +02:00
4d7b03f1f2 [zsh-completion] Ignore generated file (#3890) 2014-10-12 23:31:15 +02:00
72ebb5e4b4 Merge remote-tracking branch 'xu-cheng/zsh-completion' 2014-10-12 23:30:17 +02:00
8450c15c25 [options] Consistent formatting and general niceness 2014-10-12 23:10:11 +02:00
b88b45e46c [options] Mention login in --username documentation (#3753) 2014-10-12 22:45:36 +02:00
2417dc1715 release 2014.10.12 2014-10-12 22:25:18 +02:00
23d83ad4d5 [niconico] Fix ignored --netrc flag
See issue #3753
2014-10-12 23:18:42 +03:00
772ece3571 Merge pull request #3932 from Dineshs91/ndr-fix
Description changed
2014-10-12 17:54:51 +02:00
2c9f31188b Description changed 2014-10-12 20:09:12 +05:30
d18be55533 [theonion] Fix a small mistake in string formatting 2014-10-12 15:47:31 +03:00
ac20fc047a [theonion] Add new extractor (closes #3928) 2014-10-12 15:42:35 +03:00
b4c3c8c172 [mixcloud] Fix metadata extraction (fixes #3930) 2014-10-12 13:06:31 +02:00
b1c3a49fff apply ratelimit to f4m 2014-10-12 08:32:26 +08:00
3357110a4c [vimeo] Make the protocol mandatory in the url (fixes #3926)
If it's missing, it will be correctly handled by the generic IE.
2014-10-11 22:26:26 +02:00
e29fdedb45 Merge pull request #3923 from Dineshs91/howstuffworks-fix
Replace 404 url
2014-10-12 01:48:11 +07:00
4828703f14 [googleplus] Modernize and extract all formats 2014-10-12 01:44:13 +07:00
afe08e0d4a Merge pull request #3924 from Dineshs91/googleplus-fix
Fix download error in GooglePlus
2014-10-12 00:48:58 +07:00
071420e136 Fix download error in GooglePlus 2014-10-11 21:10:53 +05:30
f4cf848d1d Replace 404 url 2014-10-11 15:59:42 +05:30
b7b2ca6e2b Merge pull request #3921 from Dineshs91/ndr-fix
Fix ndr.de outdated test url
2014-10-11 16:07:39 +07:00
1409704afa Fix ndr.de outdated test url 2014-10-11 12:20:13 +05:30
3741302a10 [ard] Add rss support 2014-10-10 20:35:34 +02:00
c8e390c2b0 Merge pull request #3911 from Dockheas23/master
KeyError on initialising YoutubeDL in python3 #3910
2014-10-10 22:28:18 +07:00
823f1e015a [yahoo] Wipe out yahoo news extractor 2014-10-10 22:18:37 +07:00
3c06d3715e [yahoo] Generalize, support arbitrary subdomains, support iframe videos, capture error message (Closes #2470) 2014-10-10 22:11:30 +07:00
762958d5af [yahoo] Add support for regional subdomains and extract duration (Closes #3915) 2014-10-10 19:50:29 +07:00
53d9009bdb KeyError on initialising YoutubeDL in python3 #3910 2014-10-10 10:03:24 +01:00
1b725173a5 Fixed typo 2014-10-10 09:35:41 +01:00
0ca41c3d9c [walla] Fix typo 2014-10-09 21:10:07 +07:00
fc6861b175 [sportbox] Add extractor (Closes #3906) 2014-10-09 21:05:39 +07:00
b097b5f246 [mlb] Remove unused import 2014-10-09 20:07:34 +07:00
385009fc44 [mlb] Fix thumbnails extraction (Closes #3905) 2014-10-09 19:56:55 +07:00
ced659bb4d [generic] Ignore some non-video file extensions during generic extraction (Closes #3900) 2014-10-09 19:26:23 +07:00
842cca7d56 [pornhd] Fix formats extraction (Closes #3898) 2014-10-08 20:08:29 +07:00
b3826f6c8d Merge branch 'lenaten-walla' 2014-10-07 22:23:22 +07:00
7bc8780c57 [walla] Fix extractor and add subtitle tests 2014-10-07 22:23:05 +07:00
c59c3c84ed Merge branch 'walla' of https://github.com/lenaten/youtube-dl into lenaten-walla 2014-10-07 20:24:52 +07:00
24f7fb5e1e add zsh completion support into Makefile 2014-10-07 13:19:59 +08:00
3b700f8d43 support zsh completion 2014-10-07 13:19:59 +08:00
net
31d06400ec add missed init file 2014-10-06 03:03:05 +03:00
642b76ac15 release 2014.10.05.2 2014-10-05 22:04:02 +02:00
4c4de296d4 release 2014.10.05.1 2014-10-05 22:00:16 +02:00
b10609d98c [dailymotion] Alternative title search (Fixes #3882) 2014-10-05 21:59:53 +02:00
3ae165aa10 [gorillavid] Add check for non existing videos 2014-10-06 01:48:01 +07:00
e4b85e35d0 [gorillavid] Fix title extraction and make thumbnail optional (Closes #3884) 2014-10-06 01:47:22 +07:00
bb0c206f59 release 2014.10.05 2014-10-05 07:53:25 +02:00
b81f484b60 [gorillavid] Add support for movpod.in (Fixes #3881) 2014-10-05 07:53:02 +02:00
5e69192ef7 [thesixtyone] Add new extractor (closes #3781) 2014-10-04 22:40:36 +03:00
e9be9a6acd [utils] Add additional format to unified_strdate 2014-10-04 22:38:23 +03:00
f47754f061 [globo] Initial extractor implementation (Closes #3823) 2014-10-04 18:56:36 +07:00
d838b1bd4a [utils] Default age_limit to None
If we can't parse it, it means we don't have any information, not that the content is unrestricted.
2014-10-03 20:17:12 +02:00
fe506288bd [planetaplay] Add new extractor (closes #3839) 2014-10-03 19:43:36 +03:00
d397c0b3dd [breakcom] Extract all formats 2014-10-03 19:37:47 +07:00
146c80e256 [utils] Add parse_age_limit 2014-10-03 19:37:25 +07:00
f78c01f68b [breakcom] Cover more URLs with _VALID_URL (Closes #3876) 2014-10-03 18:57:18 +07:00
8489578df4 [generic] Support embedded Dailymotion playlists (fixes #3822) 2014-10-02 21:42:45 +03:00
10606050bc release 2014.10.02 2014-10-02 15:30:19 +02:00
d9bf465200 [bandcamp] Fix id extraction and modernize (Closes #3866) 2014-10-02 20:22:46 +07:00
01384d6e4b [jpopsuki] More modernize 2014-10-02 19:58:28 +07:00
08d5230945 [jpopsukitv] Improve _VALID_URL and modernize 2014-10-02 19:43:25 +07:00
852f8641e8 Merge pull request #3865 from diffycat/jpopsuki
[jpopsuki] Support category links
2014-10-02 19:38:29 +07:00
18937a50a4 [dropbox] Fix the video url query (fixes #3868)
Previously it would convert urls without a query into 'something.mp4&dl=1'
2014-10-01 23:19:56 +02:00
net
e4d6cca0c1 [walla] Add new extractor 2014-10-01 23:45:35 +03:00
d5feab9aaa [jpopsuki] Support category links 2014-10-01 23:24:23 +04:00
9e77c60c9a [tapely] Catch SoundCloud tracks 2014-10-01 21:53:45 +03:00
1414df5ce2 [izlesene] Prepend scheme to thumbnails 2014-10-01 21:11:38 +03:00
e80f40e5ca [tapely] Add new extractor (closes #3861) 2014-10-01 17:26:09 +03:00
d3c9af84fc [spankwire] Fix extraction 2014-10-01 20:53:58 +07:00
59d206ca2d [sunporno] Fix duration extraction and make more robust 2014-10-01 20:44:43 +07:00
fc66e4a0d5 [utils] Add posix expanduser implementation and clarify the original source 2014-10-01 19:48:55 +07:00
e7b6d12254 [utils] Improve and test js_to_json 2014-10-01 00:08:34 +02:00
410f3e73ab [utils] Fix js_to_json 2014-10-01 00:08:28 +02:00
07e764439a [generic] Delete test case
The page is not available any more.
2014-10-01 00:08:19 +02:00
f8fb4a7ca8 [nfl] Use compatible urlparse 2014-09-30 20:01:37 +03:00
4644ac5527 [core] Decode environment variables with filesystem encoding (Fixes #3854, Fixes #3217, Fixes #2918)
Introduces compat versions of os.getenv and os.path.expanduser
2014-09-30 22:27:53 +07:00
e497a7f2ca [tvigle] Extract format file sizes 2014-09-30 20:00:21 +07:00
a3b6be104d [tvigle] Replace 404 test 2014-09-30 19:55:30 +07:00
b7bb0df21e [vgtv] Fix tests' exts 2014-09-30 19:50:14 +07:00
4dc19c0982 [lrt] Add new extractor 2014-09-30 02:26:16 +03:00
58ea7ec81e [vimeo] Fix description extraction 2014-09-29 22:23:21 +02:00
c0f64ac689 [test/helper] Improve output of missing test definition dictionaries 2014-09-29 22:19:11 +02:00
7a08ad7d59 [test/helper] Modernize 2014-09-29 22:11:24 +02:00
2d29ac4f23 [vuclip] Fix regexp 2014-09-29 21:48:44 +02:00
a7a747d687 [vuclip] Remove test code 2014-09-29 21:47:57 +02:00
net
c816336cbd [karaoketv] Add new extractor 2014-09-29 21:58:42 +03:00
fdb4d278bf [spankwire] Fix extraction and modernize 2014-09-29 20:11:51 +07:00
59c03a9bfb [vuclip] Fix extraction 2014-09-29 13:07:58 +02:00
e7db973328 [yahoo] Remove test case
This video seems to have been removed entirely
2014-09-29 12:45:57 +02:00
99b67fecc5 [arte] Fix upload date extraction 2014-09-29 12:45:18 +02:00
89294b5f50 [README] Explain updating in detail (Fixes #3850) 2014-09-29 12:33:29 +02:00
72d53356f6 [internetvideoarchive] Fix test case 2014-09-29 12:24:48 +02:00
9e1e67fc15 [internetvideoarchive] Modernize 2014-09-29 12:23:52 +02:00
1e60e5546e [funnyordie] Fix test case md5 2014-09-29 12:20:25 +02:00
457749a703 [prosiebensat1] Fix test case 2014-09-29 12:18:49 +02:00
937f935db0 [jukebox] Remove md5 sum, it fluctuates 2014-09-29 12:15:49 +02:00
80bcefcd77 [cliphunter] Remove duration 2014-09-29 06:22:54 +02:00
8c23945c72 [eporner] Adapt to changed default format 2014-09-29 06:19:18 +02:00
989b4b2b86 [utils:YoutubeDLHandler] Work around brain-dead Python 2.6 httplib
In 2.6, the httplib sends fragments! Remove those (fixes generic_26 on 2.6).
2014-09-29 06:15:46 +02:00
2a7b4681c6 [godtube] Fix on Python 2.6 2014-09-29 05:51:41 +02:00
8157ae3904 [golem] Fix under 2.6
It's a sad story; 2.6 does not support any non-trivial xpaths.
2014-09-29 05:48:56 +02:00
e50e2fcd4d [br] fix test case 2014-09-29 05:40:20 +02:00
6be451f422 [youtube] Remove swf signature test cases
These files are now 0 Bytes
2014-09-29 05:24:00 +02:00
5e4f06197f [facebook] Fix test case 2014-09-29 05:19:56 +02:00
761e1645e0 [generic] Remove unstable test checksum 2014-09-29 05:18:45 +02:00
8ff14175e2 [sportdeutschland] Fix testcase 2014-09-29 05:17:16 +02:00
dbe3043cd6 [ynet] Fix test checksums 2014-09-29 05:15:42 +02:00
a8eb5a8e61 [generic] Fix testcases 2014-09-29 05:12:57 +02:00
6043f1df4e [ign] Return proper playlist object 2014-09-29 05:05:06 +02:00
12548cd933 [worldstarhiphop] Correct title extraction 2014-09-29 05:02:58 +02:00
2593039522 [vimeo] Use regexps to find description
This fixes descriptions on 2.6 and makes the code simpler.
2014-09-29 04:58:31 +02:00
35d3e63d24 release 2014.09.29.2 2014-09-29 04:49:11 +02:00
27aede9074 [pbs] Add support for series/jwplayer type video (Fixes #3849) 2014-09-29 04:48:50 +02:00
f5b7e6a842 release 2014.09.29.1 2014-09-29 02:04:28 +02:00
a1f934b171 [youtube] Correct language cookie handling 2014-09-29 02:04:16 +02:00
a43ee88c6f release 2014.09.29 2014-09-29 01:51:53 +02:00
e2dce53781 [youtube] Always request webpage in English (Fixes #3844) 2014-09-29 01:39:26 +02:00
1770ed9e86 [thvideo] Simplify (#3848) 2014-09-29 00:38:37 +02:00
457ac58cc7 Merge remote-tracking branch 'diffycat/thvideo-update' 2014-09-29 00:36:55 +02:00
9c44d2429b [vimeo:likes] Support large like lists (Fixes #3847) 2014-09-29 00:36:06 +02:00
d2e32f7df5 Do not use HTML characters in output
This messes up the format when people paste it outside of code tags.
2014-09-29 00:23:43 +02:00
67077b182b [thvideo] Add support for playlists 2014-09-28 23:36:55 +04:00
5f4c318844 [nfl] Support team micro-sites (fixes #3831) 2014-09-28 21:48:26 +03:00
dfee83234b [nfl] Prefer progressive downloads 2014-09-28 19:25:28 +03:00
7f5c0c4a19 [README] Clarify test's md5 filesize (#3846) 2014-09-28 22:10:20 +07:00
4bc77c8417 [README] Use _match_id helper function 2014-09-28 13:52:21 +02:00
22dd3fad86 release 2014.09.28.1 2014-09-28 12:14:25 +02:00
d6e6a42256 [vimeo:likes] Add new extractor (Fixes #3835) 2014-09-28 12:14:16 +02:00
76e7d1e74b [played] Remove unused import 2014-09-28 10:56:36 +02:00
38c4d41b74 [played] Simplify (#3798) 2014-09-28 10:55:27 +02:00
f0b8e3607d Merge remote-tracking branch 'r4mos/played' 2014-09-28 10:52:23 +02:00
51ee08c4bb Remove unused imports 2014-09-28 10:50:43 +02:00
c841789772 [muenchentv] Add thumbnail 2014-09-28 10:49:58 +02:00
c121a75b36 [heise] Add support for description 2014-09-28 10:49:12 +02:00
5a8b77551d [heise] Simplify (#3842) 2014-09-28 10:47:25 +02:00
0217aee154 Merge remote-tracking branch 'd912e3/heise' 2014-09-28 10:36:44 +02:00
b14f3a4c1d [golem] Simplify (#3828) 2014-09-28 10:35:19 +02:00
92f7963f6e Merge remote-tracking branch 'd912e3/golem' 2014-09-28 10:10:34 +02:00
88fbe4c2cc release 2014.09.28 2014-09-28 09:49:42 +02:00
394599f422 [oktoberfesttv] Add new extractor (Fixes #3845) 2014-09-28 09:49:21 +02:00
ed9266db90 [common] Add new helper function _match_id 2014-09-28 09:31:58 +02:00
f4b1c7adb8 [muenchentv] Move live title generation to common 2014-09-28 08:53:52 +02:00
c95eeb7b80 [eitb] Modernize 2014-09-28 08:49:03 +02:00
5e43e3803c Credit @lenaten for ynet (#3840) and sport5 (#3841) 2014-09-28 03:45:15 +07:00
a89435a7a8 [ynet] Improve _VALID_URL 2014-09-28 03:30:41 +07:00
a0a90b3ba1 Merge branch 'lenaten-ynet' 2014-09-28 03:26:47 +07:00
c664182323 [ynet] Remove unused stuff, simplify and improve 2014-09-28 03:26:38 +07:00
6be1cd4ddb Merge branch 'ynet' of https://github.com/lenaten/youtube-dl into lenaten-ynet 2014-09-28 02:56:51 +07:00
ee0d90707a [YoutubeDL] Fix string check for python3 2014-09-28 02:48:41 +07:00
f776d8f608 [sport5] Keep alphanumeric order 2014-09-28 02:35:46 +07:00
b3ac3a51ac Merge branch 'lenaten-sport5' 2014-09-28 02:32:52 +07:00
0b75c2a88b [sport5] Capture error message and improve 2014-09-28 02:31:14 +07:00
7b7518124e [heise] Don't check string type
Before Python 3 could be unicode, so don't check at all.
2014-09-27 21:12:23 +02:00
68b0973046 [YoutubeDL] Expect all kind of strings in urlopen
Now it doesn't fail if req is python2's str
2014-09-28 02:07:42 +07:00
3a203b8bfa Merge branch 'sport5' of https://github.com/lenaten/youtube-dl into lenaten-sport5 2014-09-28 00:55:16 +07:00
70752ccefd [golem] Don't omit positional argument specifiers
Required by Python 2.6.
2014-09-27 19:35:55 +02:00
0155549d6c [heise] Add new extractor 2014-09-27 19:28:01 +02:00
net
b66745288e [sport5] Add new extractor 2014-09-27 20:21:46 +03:00
net
2a1325fdde [ynet] Add new extractor 2014-09-27 20:11:22 +03:00
2f9e8776df [extremetube] Fix extraction 2014-09-27 22:36:53 +07:00
497339fa0e [anysex] Fix extraction 2014-09-27 22:29:27 +07:00
8e6f8051f0 [vbox7] Don't set the extension to 'flv' (fixes #3836) 2014-09-27 10:53:02 +02:00
11b3ce8509 [crunchyroll] Allow to list subtitles (fixes #3805) 2014-09-25 17:57:38 +03:00
6a5af6acb9 [golem] Add new extractor 2014-09-25 16:25:53 +02:00
9a0d98bb40 [vube] Update tests 2014-09-25 20:57:18 +07:00
fbd3162e49 [vube] Add DMCA notice 2014-09-25 20:48:54 +07:00
54e9a4af95 [wat] Skip test 2014-09-25 20:33:11 +07:00
8a32b82e46 [youku] Modernize somewhat 2014-09-25 09:58:09 +02:00
fec02bcc90 [hlsnative] Correct handling when remaining_bytes is None 2014-09-25 09:21:45 +02:00
c6e90caaa6 Merge remote-tracking branch 'naglis/wistia' 2014-09-25 02:03:49 +02:00
4bbf157794 release 2014.09.25 2014-09-25 01:59:45 +02:00
6b08cdf626 [youtube] Support for embedded /p players (Fixes #3821) 2014-09-25 01:59:02 +02:00
b686fc18da [hlsnative] Support test parameter 2014-09-24 14:38:40 +02:00
0b97f3a936 release 2014.09.24.1 2014-09-24 14:17:42 +02:00
eb73f2649f [vevo] Skip SMIL download 2014-09-24 14:17:33 +02:00
f0b5d6af74 [vevo] Support 1080p videos (Fixes #3656) 2014-09-24 14:16:56 +02:00
2f771f6c99 release 2014.09.24 2014-09-24 11:06:46 +02:00
3b2f933b01 [generic] Allow embedded YoutubePlaylists (Fixes #3821) 2014-09-24 11:05:14 +02:00
cc746841e7 [flickr] Modernize 2014-09-24 10:46:33 +02:00
ac7553d031 [youtube] Support embed/videoseries URLs (#3821) 2014-09-24 10:34:29 +02:00
cdc628a498 [youtube] Move more tests to extractors 2014-09-24 10:25:47 +02:00
69ea8ca42c [youtube] Remove superfluous unicode specifiers 2014-09-24 09:51:45 +02:00
4bc3a23ec5 [youtube] Modernize 2014-09-24 09:49:53 +02:00
bd5650ac64 [nfl] Fix test case - download, but don't check md5 2014-09-23 20:42:28 +03:00
86916dae4b [wat] Capture and output error message 2014-09-23 19:58:35 +07:00
f7d159cf95 [noco] Encode before passing to hashlib.md5 (Closes #3816) 2014-09-23 19:13:11 +07:00
632e5684ce [nfl] Add new extractor. (Closes #3815) 2014-09-23 00:28:19 +03:00
094d42fe44 release 2014.09.22.1 2014-09-22 18:15:07 +02:00
63cddb6477 [sbs] Recognize urls with format 'http://www.sbs.com.au/ondemand/video/<id>' (#3811) 2014-09-22 14:11:08 +02:00
273dea4248 [playfm] Fix view count and add comment count 2014-09-22 18:58:22 +07:00
f90d95edeb release 2014.09.22 2014-09-22 13:07:23 +02:00
45c85d7ba1 Merge remote-tracking branch 'origin/master' 2014-09-22 12:53:41 +02:00
d0df92928b [npo] Add extractor for tegenlicht.vpro.nl (closes #3778) 2014-09-21 16:53:00 +02:00
df8f53f752 [thvideo] Support mobile URLs as well 2014-09-21 16:32:38 +02:00
e35cb78c40 [theplatform] Correctly extract videos that don't use f4m or rtmp (reported in #3176) 2014-09-21 16:08:38 +02:00
3ef7d11acd Merge remote-tracking branch 'diffycat/thvideo' 2014-09-21 15:50:01 +02:00
224ce0d872 [nbc] Update test 2014-09-21 15:49:04 +02:00
dd41e8c82b [theplatform] Extract all formats for f4m videos 2014-09-21 15:47:58 +02:00
b509a4b176 [downloader/f4m] If <pv-2.0> is in the manifest, add it to the fragments urls query (fixes #3176)
It's used in some akamai videos (for example for theplatform.com).
2014-09-21 15:43:09 +02:00
b28c8403b2 [yourupload] Add new extractor. Fixes #3085 2014-09-21 15:13:35 +03:00
7bd4b4229a [dropbox] Recognize 'https://www.dropbox.com/sh/*' urls (fixes #3795)
And extract the title from the url last path component.
2014-09-21 13:42:08 +02:00
72e450c555 [thvideo] Add support for THVideo 2014-09-21 13:21:29 +04:00
522c55b7f2 [mgoon] Add new extractor 2014-09-21 03:26:12 +03:00
58e7071a2c [tube8] Improve _VALID_URL and add display_id 2014-09-21 06:37:11 +07:00
516812df41 Merge pull request #3801 from megustamucho/master
[tube8] Improved regex
2014-09-21 06:29:13 +07:00
752297631f [noco] Adapt to API v1.1 (Closes #3797) 2014-09-21 06:20:42 +07:00
34e14a9beb Merge remote-tracking branch 'upstream/master' 2014-09-20 22:41:45 +10:00
ffb5b05db1 Merge pull request #3800 from msmuenchen/patch-1
[divxstage] added .to TLD
2014-09-20 16:59:14 +07:00
3e8fcd9fa1 [divxstage] added .to TLD
Example video "http://www.divxstage.eu/video/930c52709d2" which gets redirected to .to TLD
2014-09-20 02:32:41 +02:00
746c67d72f [wistia] Use API and make more generic 2014-09-20 03:02:11 +03:00
5aa38e75b2 [played] Add new extractor 2014-09-19 22:46:57 +02:00
532f5bff70 [franceinter] Fix extraction and modernize 2014-09-19 20:58:50 +07:00
f566d9f1d5 release 2014.09.19 2014-09-19 09:58:01 +02:00
7267bd536f [muenchentv] Add support (Fixes #3507) 2014-09-19 09:57:53 +02:00
589d3d7c7a [moniker] rename from hypestat (#3788) 2014-09-18 21:37:09 +02:00
46f74bcf5c [soundcloud] Fix non-secret playlists 2014-09-18 18:57:04 +02:00
37bfe8ace4 [hypestat] Match URLs with www. and https:// 2014-09-18 18:56:02 +02:00
0529eef5a4 [hypestat] Unify allmyvideos and vidspot (Closes #3788) 2014-09-18 18:54:03 +02:00
fd78a4d3e6 release 2014.09.18 2014-09-18 18:43:59 +02:00
1de33fafd9 [YoutubeDL] Allow downloading multiple formats with , 2014-09-18 18:43:49 +02:00
e2e5dae64d Add -f m4a 2014-09-18 18:40:19 +02:00
09b23c902b [1up.com] Urls end now with '.html' 2014-09-18 17:02:53 +02:00
109a540e7a [ign] Fix extraction 2014-09-18 16:57:34 +02:00
2914e5f00f [drtuber] Fix categories 2014-09-18 20:56:54 +07:00
2f834e9381 [soundcloud] Secret playlists and sets
Closes #3707 again.

No test cases because I don't know what urls to use that won't be turned
into public eventually (as it happened with the first one in that
ticket)
2014-09-18 06:35:11 -03:00
9296738f20 [soundcloud] Support api urls with secret_token, Closes #3707 2014-09-18 03:02:03 -03:00
0e59b9fffb [videomega] Simplify (#3786)
* Use raw strings (r'foo') for regular expressions (enables highlighting and avoids some errors).
* title is always true-ish
2014-09-18 00:19:08 +02:00
67abbe9527 [videomega] Add new extractor. Closes #3775 2014-09-17 22:57:01 +03:00
944a3de278 [npo] Improve formats extraction (Closes #3782) 2014-09-17 19:02:57 +07:00
5a13fe9ed2 Credit @m5moufl for behindkink (#3740) 2014-09-17 12:50:47 +02:00
6b6096d0b7 release 2014.09.16.1 2014-09-16 23:35:15 +02:00
d0246d07f1 Merge remote-tracking branch 'origin/master' 2014-09-16 23:35:08 +02:00
727a98c3ee Merge remote-tracking branch 'r4mos/master' 2014-09-16 23:33:26 +02:00
997987d568 Credit @r4mos for allmyvideos 2014-09-16 23:33:13 +02:00
c001f939e4 [Allmyvideos] Fixed md5. Only 10KiB of the video file 2014-09-16 23:23:54 +02:00
e825c38082 [allmyvideos] Support multiple formats 2014-09-16 22:58:31 +02:00
a04aa7a9e6 [generic] The url in the <meta redirect> doesn't need to be enclosed in single quotes (fixes #3770)
See the examples in https://en.wikipedia.org/wiki/Meta_refresh or the shortened urls from https://t.co/.
2014-09-16 22:53:48 +02:00
7cdd5339b3 Change tabs to spaces 2014-09-16 21:05:50 +02:00
38349518f1 Added new host: allmyvideos.net 2014-09-16 20:48:53 +02:00
64892c0b79 [francetv] Adapt to new API (Closes #3751, closes #3769) 2014-09-16 23:47:59 +07:00
dc9f356846 [ard] Remove URL path escaping
This is now implemented generic in YoutubeDL.urlopen
2014-09-16 18:47:06 +07:00
ed86ee3b4a release 2014.09.16 2014-09-16 10:09:02 +02:00
7bb5df1cda [nhl] Match videos without catid (Fixes #3764) 2014-09-16 10:08:34 +02:00
37a81dff04 [behindkink] Remove call to report_extraction 2014-09-15 23:37:22 +02:00
fc96eb4e21 Merge remote-tracking branch '5moufl/behindkink' 2014-09-15 23:36:21 +02:00
ae369738b0 Credit @haricharan for einthusan (#3755) 2014-09-15 23:35:00 +02:00
e2037b3f7d [einthusan] Add description and beautify 2014-09-15 23:33:47 +02:00
5419033935 Fixed tests 2014-09-15 23:27:18 +02:00
2eebf060af Merge commit '98703c7fbfcf06348220aa63f9422cdd792cfe1a' 2014-09-15 23:26:54 +02:00
acd9db5902 Merge remote-tracking branch 'naglis/nosvideo' 2014-09-15 16:10:52 +02:00
d0e8b3d59b [nosvideo] Make more robust against missing metadata 2014-09-15 16:59:03 +03:00
c15dd15388 Merge remote-tracking branch 'naglis/turbo' 2014-09-15 15:48:48 +02:00
0003a5c416 Merge remote-tracking branch 'dstftw/escape-non-ascii-in-urls'
Conflicts:
	test/test_utils.py
2014-09-15 15:40:10 +02:00
21f2927f70 release 2014.09.15.1 2014-09-15 15:28:41 +02:00
e5a79071a5 [daum] Support non-numeric video IDs (Fixes #3749) 2014-09-15 15:25:35 +02:00
ca0e7a2b17 release 2014.09.15 2014-09-15 15:11:22 +02:00
b523bb71ab Merge remote-tracking branch 'origin/master' 2014-09-15 15:11:15 +02:00
a020a0dc20 [facebook] Fix support for untitled videos (Fixes #3757) 2014-09-15 15:10:24 +02:00
6d1f2431bd [BehindKink] Minor fixes
- fix _VALID_URL regex
 - remove unnecessary variable
 - remove second call of report_extraction
2014-09-15 15:09:17 +02:00
fdea3abdf8 [turbo] Add new extractor 2014-09-15 16:08:20 +03:00
59d284c316 [vporn] Make video URL regex more strict
There is a garbage instead of proper URL for some HD videos
2014-09-15 19:19:37 +07:00
98703c7fbf Einthusan Add new extractor 2014-09-14 23:14:00 -05:00
b04c8f7358 release 2014.09.14.3 2014-09-14 16:48:15 +02:00
56d1912f1d Add a completion script generator for the fish shell 2014-09-14 14:07:33 +02:00
eb3bd7ba8d [cloudy] Retry extraction on 410 status code (#3743 #3744) 2014-09-14 19:04:16 +07:00
3b11e86eeb release 2014.09.14.2 2014-09-14 12:56:04 +02:00
2bca84e345 [BehindKink] Add new extractor 2014-09-13 17:47:19 +02:00
984e8e14ea [utils] Remove debug garbage 2014-09-13 21:08:04 +07:00
d05cfe0600 [YoutubeDL/utils] Clarify rationale for URL escaping in comment, move escape routines to utils and add some tests 2014-09-13 20:59:16 +07:00
37419b4f99 [YoutubeDL] Escape non-ASCII characters in URLs
urllib chokes on URLs with non-ASCII characters (see http://bugs.python.org/issue3991)
Working around by replacing request's original URL with escaped one
2014-09-12 23:20:17 +07:00
a8aa99442f Merge branch 'master' of https://github.com/megustamucho/youtube-dl 2014-09-09 13:07:16 +10:00
94b539d155 [tube8] Improved regex 2014-09-09 13:05:14 +10:00
b8874d4d4e [tube8] Improved regex 2014-09-09 12:48:56 +10:00
ca7a9c1bf7 Merge remote-tracking branch 'upstream/master' 2014-08-19 07:15:33 -07:00
247a5da704 Merge remote-tracking branch 'upstream/master' 2014-08-16 03:51:51 -07:00
d1b4617e1d Merge remote-tracking branch 'upstream/master' 2014-08-15 09:52:06 -07:00
74dcf42a85 Merge remote-tracking branch 'upstream/master' 2014-08-13 16:07:58 -07:00
a42c921598 Removed sleep and sleep output when interval is zero 2014-08-13 04:38:40 -07:00
f96252b913 Merge remote-tracking branch 'upstream/master' 2014-08-13 04:22:45 -07:00
04b89c9026 Merge remote-tracking branch 'upstream/master' 2014-08-08 07:14:54 -07:00
0c72eb9060 Merge remote-tracking branch 'upstream/master' 2014-08-06 16:43:21 -07:00
00cf122d7a [downloader/common] Fix possible negative sleep time in slow_down() 2014-08-06 20:53:04 +02:00
f9f86b0c64 Merge remote-tracking branch 'upstream/master' 2014-08-05 12:43:30 -07:00
0aed8df2bf Merge remote-tracking branch 'upstream/master' 2014-08-03 15:23:01 -07:00
2f61fe4ccc Removed unneccesary changes to utils.py 2014-08-03 07:38:04 -07:00
03359e9864 Added --sleep-interval option 2014-08-03 07:34:04 -07:00
c7667c2d7f [downloader/(common/http)] Changes calculation of the rate-limit. (Fix #2297, fix #2140, fix #595, fix #2370) 2014-07-31 03:08:24 +02:00
484 changed files with 21518 additions and 6788 deletions

4
.gitignore vendored
View File

@ -11,6 +11,7 @@ MANIFEST
README.txt
youtube-dl.1
youtube-dl.bash-completion
youtube-dl.fish
youtube-dl
youtube-dl.exe
youtube-dl.tar.gz
@ -29,3 +30,6 @@ updates_key.pem
*.swp
test/testdata
.tox
youtube-dl.zsh
.idea
.idea/*

View File

@ -4,12 +4,14 @@ python:
- "2.7"
- "3.3"
- "3.4"
before_install:
- sudo apt-get update -qq
- sudo apt-get install -yqq rtmpdump
script: nosetests test --verbose
notifications:
email:
- filippo.valsorda@gmail.com
- phihag@phihag.de
- jaime.marquinez.ferrandiz+travis@gmail.com
- yasoob.khld@gmail.com
# irc:
# channels:

110
AUTHORS Normal file
View File

@ -0,0 +1,110 @@
Ricardo Garcia Gonzalez
Danny Colligan
Benjamin Johnson
Vasyl' Vavrychuk
Witold Baryluk
Paweł Paprota
Gergely Imreh
Rogério Brito
Philipp Hagemeister
Sören Schulze
Kevin Ngo
Ori Avtalion
shizeeg
Filippo Valsorda
Christian Albrecht
Dave Vasilevsky
Jaime Marquínez Ferrándiz
Jeff Crouse
Osama Khalid
Michael Walter
M. Yasoob Ullah Khalid
Julien Fraichard
Johny Mo Swag
Axel Noack
Albert Kim
Pierre Rudloff
Huarong Huo
Ismael Mejía
Steffan 'Ruirize' James
Andras Elso
Jelle van der Waa
Marcin Cieślak
Anton Larionov
Takuya Tsuchida
Sergey M.
Michael Orlitzky
Chris Gahan
Saimadhav Heblikar
Mike Col
Oleg Prutz
pulpe
Andreas Schmitz
Michael Kaiser
Niklas Laxström
David Triendl
Anthony Weems
David Wagner
Juan C. Olivares
Mattias Harrysson
phaer
Sainyam Kapoor
Nicolas Évrard
Jason Normore
Hoje Lee
Adam Thalhammer
Georg Jähnig
Ralf Haring
Koki Takahashi
Ariset Llerena
Adam Malcontenti-Wilson
Tobias Bell
Naglis Jonaitis
Charles Chen
Hassaan Ali
Dobrosław Żybort
David Fabijan
Sebastian Haas
Alexander Kirk
Erik Johnson
Keith Beckman
Ole Ernst
Aaron McDaniel (mcd1992)
Magnus Kolstad
Hari Padmanaban
Carlos Ramos
5moufl
lenaten
Dennis Scheiba
Damon Timm
winwon
Xavier Beynon
Gabriel Schubiner
xantares
Jan Matějka
Mauroy Sébastien
William Sewell
Dao Hoang Son
Oskar Jauch
Matthew Rayfield
t0mm0
Tithen-Firion
Zack Fernandes
cryptonaut
Adrian Kretz
Mathias Rav
Petr Kutalek
Will Glynn
Max Reimann
Cédric Luthi
Thijs Vermeir
Joel Leclerc
Christopher Krooss
Ondřej Caletka
Dinesh S
Johan K. Jensen
Yen Chi Hsuan
Enam Mijbah Noor
David Luhmer
Shaya Goldberg
Paul Hartmann

136
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,136 @@
Please include the full output of the command when run with `--verbose`. The output (including the first lines) contain important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever.
Please re-read your issue once again to avoid a couple of common mistakes (you can and should use this as a checklist):
### Is the description of the issue itself sufficient?
We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources. Many contributors, including myself, are also not native speakers, so we may misread some parts.
So please elaborate on what feature you are requesting, or what bug you want to be fixed. Make sure that it's obvious
- What the problem is
- How it could be fixed
- How your proposed solution would look like
If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. As a commiter myself, I often get frustrated by these issues, since the only possible way for me to move forward on them is to ask for clarification over and over.
For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the -v flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information.
Site support requests **must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL.
### Are you using the latest version?
Before reporting any issue, type youtube-dl -U. This should report that you're up-to-date. About 20% of the reports we receive are already fixed, but people are using outdated versions. This goes for feature requests as well.
### Is the issue already documented?
Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or at https://github.com/rg3/youtube-dl/search?type=Issues . If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
### Why are existing options not enough?
Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#synopsis). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
### Is there enough context in your bug report?
People want to solve problems, and often think they do us a favor by breaking down their larger problems (e.g. wanting to skip already downloaded files) to a specific request (e.g. requesting us to look whether the file exists before downloading the info page). However, what often happens is that they break down the problem into two steps: One simple, and one impossible (or extremely complicated one).
We are then presented with a very complicated request when the original problem could be solved far easier, e.g. by recording the downloaded video IDs in a separate file. To avoid this, you must include the greater context where it is non-obvious. In particular, every feature request that does not consist of adding support for a new site should contain a use case scenario that explains in what situation the missing feature would be useful.
### Does the issue involve one problem, and one problem only?
Some of our users seem to think there is a limit of issues they can or should open. There is no limit of issues they can or should open. While it may seem appealing to be able to dump all your issues into one ticket, that means that someone who solves one of your issues cannot mark the issue as closed. Typically, reporting a bunch of issues leads to the ticket lingering since nobody wants to attack that behemoth, until someone mercifully splits the issue into multiple ones.
In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, Whitehouse podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of youtube-dl that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service.
### Is anyone going to need the feature?
Only post features that you (or an incapacitated friend you can personally talk to) require. Do not post features because they seem like a good idea. If they are really useful, they will be requested by someone who requires them.
### Is your question about youtube-dl?
It may sound strange, but some bug reports we receive are completely unrelated to youtube-dl and relate to a different or even the reporter's own application. Please make sure that you are actually using youtube-dl. If you are using a UI for youtube-dl, report the bug to the maintainer of the actual application providing the UI. On the other hand, if your UI for youtube-dl fails in some way you believe is related to youtube-dl, by all means, go ahead and report the bug.
# DEVELOPER INSTRUCTIONS
Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
To run youtube-dl as a developer, you don't need to build anything either. Simply execute
python -m youtube_dl
To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work:
python -m unittest discover
python test/test_download.py
nosetests
If you want to create a build of youtube-dl yourself, you'll need
* python
* make
* pandoc
* zip
* nosetests
### Adding support for a new site
If you want to add support for a new site, you can follow this quick list (assuming your service is called `yourextractor`):
1. [Fork this repository](https://github.com/rg3/youtube-dl/fork)
2. Check out the source code with `git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git`
3. Start a new git branch with `cd youtube-dl; git checkout -b yourextractor`
4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`:
```python
# coding: utf-8
from __future__ import unicode_literals
from .common import InfoExtractor
class YourExtractorIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)'
_TEST = {
'url': 'http://yourextractor.com/watch/42',
'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
'info_dict': {
'id': '42',
'ext': 'mp4',
'title': 'Video title goes here',
'thumbnail': 're:^https?://.*\.jpg$',
# TODO more properties, either as:
# * A value
# * MD5 checksum; start the string with md5:
# * A regular expression; start the string with re:
# * Any Python type (for example int or float)
}
}
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
# TODO more code goes here, for example ...
title = self._html_search_regex(r'<h1>(.*?)</h1>', webpage, 'title')
return {
'id': video_id,
'title': title,
'description': self._og_search_description(webpage),
# TODO more properties (see youtube_dl/extractor/common.py)
}
```
5. Add an import in [`youtube_dl/extractor/__init__.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/__init__.py).
6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will be then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
7. Have a look at [`youtube_dl/common/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L38). Add tests and code for as many as you want.
8. If you can, check the code with [pyflakes](https://pypi.python.org/pypi/pyflakes) (a good idea) and [pep8](https://pypi.python.org/pypi/pep8) (optional, ignore E501).
9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
$ git add youtube_dl/extractor/__init__.py
$ git add youtube_dl/extractor/yourextractor.py
$ git commit -m '[yourextractor] Add new extractor'
$ git push origin yourextractor
10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
In any case, thank you very much for your contributions!

View File

@ -2,5 +2,6 @@ include README.md
include test/*.py
include test/*.json
include youtube-dl.bash-completion
include youtube-dl.fish
include youtube-dl.1
recursive-include docs Makefile conf.py *.rst

View File

@ -1,7 +1,7 @@
all: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-completion
all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish supportedsites
clean:
rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz
rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish *.dump *.part *.info.json CONTRIBUTING.md.tmp
cleanall: clean
rm -f youtube-dl youtube-dl.exe
@ -9,6 +9,7 @@ cleanall: clean
PREFIX ?= /usr/local
BINDIR ?= $(PREFIX)/bin
MANDIR ?= $(PREFIX)/man
SHAREDIR ?= $(PREFIX)/share
PYTHON ?= /usr/bin/env python
# set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
@ -22,23 +23,36 @@ else
endif
endif
install: youtube-dl youtube-dl.1 youtube-dl.bash-completion
install: youtube-dl youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
install -d $(DESTDIR)$(BINDIR)
install -m 755 youtube-dl $(DESTDIR)$(BINDIR)
install -d $(DESTDIR)$(MANDIR)/man1
install -m 644 youtube-dl.1 $(DESTDIR)$(MANDIR)/man1
install -d $(DESTDIR)$(SYSCONFDIR)/bash_completion.d
install -m 644 youtube-dl.bash-completion $(DESTDIR)$(SYSCONFDIR)/bash_completion.d/youtube-dl
install -d $(DESTDIR)$(SHAREDIR)/zsh/site-functions
install -m 644 youtube-dl.zsh $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_youtube-dl
install -d $(DESTDIR)$(SYSCONFDIR)/fish/completions
install -m 644 youtube-dl.fish $(DESTDIR)$(SYSCONFDIR)/fish/completions/youtube-dl.fish
codetest:
flake8 .
test:
#nosetests --with-coverage --cover-package=youtube_dl --cover-html --verbose --processes 4 test
nosetests --verbose test
$(MAKE) codetest
ot: offlinetest
offlinetest: codetest
nosetests --verbose test --exclude test_download --exclude test_age_restriction --exclude test_subtitles --exclude test_write_annotations --exclude test_youtube_lists
tar: youtube-dl.tar.gz
.PHONY: all clean install test tar bash-completion pypi-files
.PHONY: all clean install test tar bash-completion pypi-files zsh-completion fish-completion ot offlinetest codetest supportedsites
pypi-files: youtube-dl.bash-completion README.txt youtube-dl.1
pypi-files: youtube-dl.bash-completion README.txt youtube-dl.1 youtube-dl.fish
youtube-dl: youtube_dl/*.py youtube_dl/*/*.py
zip --quiet youtube-dl youtube_dl/*.py youtube_dl/*/*.py
@ -49,7 +63,13 @@ youtube-dl: youtube_dl/*.py youtube_dl/*/*.py
chmod a+x youtube-dl
README.md: youtube_dl/*.py youtube_dl/*/*.py
COLUMNS=80 python -m youtube_dl --help | python devscripts/make_readme.py
COLUMNS=80 python youtube_dl/__main__.py --help | python devscripts/make_readme.py
CONTRIBUTING.md: README.md
python devscripts/make_contributing.py README.md CONTRIBUTING.md
supportedsites:
python devscripts/make_supportedsites.py docs/supportedsites.md
README.txt: README.md
pandoc -f markdown -t plain README.md -o README.txt
@ -64,7 +84,17 @@ youtube-dl.bash-completion: youtube_dl/*.py youtube_dl/*/*.py devscripts/bash-co
bash-completion: youtube-dl.bash-completion
youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-completion
youtube-dl.zsh: youtube_dl/*.py youtube_dl/*/*.py devscripts/zsh-completion.in
python devscripts/zsh-completion.py
zsh-completion: youtube-dl.zsh
youtube-dl.fish: youtube_dl/*.py youtube_dl/*/*.py devscripts/fish-completion.in
python devscripts/fish-completion.py
fish-completion: youtube-dl.fish
youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
@tar -czf youtube-dl.tar.gz --transform "s|^|youtube-dl/|" --owner 0 --group 0 \
--exclude '*.DS_Store' \
--exclude '*.kate-swp' \
@ -78,5 +108,6 @@ youtube-dl.tar.gz: youtube-dl README.md README.txt youtube-dl.1 youtube-dl.bash-
-- \
bin devscripts test youtube_dl docs \
LICENSE README.md README.txt \
Makefile MANIFEST.in youtube-dl.1 youtube-dl.bash-completion setup.py \
Makefile MANIFEST.in youtube-dl.1 youtube-dl.bash-completion \
youtube-dl.zsh youtube-dl.fish setup.py \
youtube-dl

299
README.md
View File

@ -1,7 +1,15 @@
youtube-dl - download videos from youtube.com or other video platforms
# SYNOPSIS
**youtube-dl** [OPTIONS] URL [URL...]
- [INSTALLATION](#installation)
- [DESCRIPTION](#description)
- [OPTIONS](#options)
- [CONFIGURATION](#configuration)
- [OUTPUT TEMPLATE](#output-template)
- [VIDEO SELECTION](#video-selection)
- [FAQ](#faq)
- [DEVELOPER INSTRUCTIONS](#developer-instructions)
- [BUGS](#bugs)
- [COPYRIGHT](#copyright)
# INSTALLATION
@ -30,10 +38,12 @@ Alternatively, refer to the developer instructions below for how to check out an
# DESCRIPTION
**youtube-dl** is a small command-line program to download videos from
YouTube.com and a few more sites. It requires the Python interpreter, version
2.6, 2.7, or 3.3+, and it is not platform specific. It should work on
2.6, 2.7, or 3.2+, and it is not platform specific. It should work on
your Unix box, on Windows or on Mac OS X. It is released to the public domain,
which means you can modify it, redistribute it or use it however you like.
youtube-dl [OPTIONS] URL [URL...]
# OPTIONS
-h, --help print this help text and exit
--version print program version and exit
@ -50,10 +60,6 @@ which means you can modify it, redistribute it or use it however you like.
they would handle
--extractor-descriptions Output descriptions of all supported
extractors
--proxy URL Use the specified HTTP/HTTPS proxy. Pass in
an empty string (--proxy "") for direct
connection
--socket-timeout None Time to wait before giving up, in seconds
--default-search PREFIX Use this prefix for unqualified URLs. For
example "gvsearch2:" downloads two videos
from google videos for youtube-dl "large
@ -65,14 +71,36 @@ which means you can modify it, redistribute it or use it however you like.
this is not possible instead of searching.
--ignore-config Do not read configuration files. When given
in the global configuration file /etc
/youtube-dl.conf: do not read the user
configuration in ~/.config/youtube-dl.conf
(%APPDATA%/youtube-dl/config.txt on
Windows)
/youtube-dl.conf: Do not read the user
configuration in ~/.config/youtube-
dl/config (%APPDATA%/youtube-dl/config.txt
on Windows)
--flat-playlist Do not extract the videos of a playlist,
only list them.
## Network Options:
--proxy URL Use the specified HTTP/HTTPS proxy. Pass in
an empty string (--proxy "") for direct
connection
--socket-timeout SECONDS Time to wait before giving up, in seconds
--source-address IP Client-side IP address to bind to
(experimental)
-4, --force-ipv4 Make all connections via IPv4
(experimental)
-6, --force-ipv6 Make all connections via IPv6
(experimental)
## Video Selection:
--playlist-start NUMBER playlist video to start at (default is 1)
--playlist-end NUMBER playlist video to end at (default is last)
--playlist-items ITEM_SPEC playlist video items to download. Specify
indices of the videos in the playlist
seperated by commas like: "--playlist-items
1,2,5,8" if you want to download videos
indexed 1, 2, 5, 8 in the playlist. You can
specify range: "--playlist-items
1-3,7,10-13", it will download the videos
at index 1, 2, 3, 7, 10, 11, 12 and 13.
--match-title REGEX download only matching titles (regex or
caseless sub-string)
--reject-title REGEX skip download for matching titles (regex or
@ -91,7 +119,8 @@ which means you can modify it, redistribute it or use it however you like.
COUNT views
--max-views COUNT Do not download any videos with more than
COUNT views
--no-playlist download only the currently playing video
--no-playlist If the URL refers to a video and a
playlist, download only the video.
--age-limit YEARS download only videos suitable for the given
age
--download-archive FILE Download only videos not listed in the
@ -99,25 +128,29 @@ which means you can modify it, redistribute it or use it however you like.
downloaded videos in it.
--include-ads Download advertisements as well
(experimental)
--youtube-include-dash-manifest Try to download the DASH manifest on
YouTube videos (experimental)
## Download Options:
-r, --rate-limit LIMIT maximum download rate in bytes per second
(e.g. 50K or 4.2M)
-R, --retries RETRIES number of retries (default is 10)
-R, --retries RETRIES number of retries (default is 10), or
"infinite".
--buffer-size SIZE size of download buffer (e.g. 1024 or 16K)
(default is 1024)
--no-resize-buffer do not automatically adjust the buffer
size. By default, the buffer size is
automatically resized from an initial value
of SIZE.
--playlist-reverse Download playlist videos in reverse order
--xattr-set-filesize (experimental) set file xattribute
ytdl.filesize with expected filesize
--external-downloader COMMAND (experimental) Use the specified external
downloader. Currently supports
aria2c,curl,wget
## Filesystem Options:
-a, --batch-file FILE file containing URLs to download ('-' for
stdin)
--id use only video ID in file name
-A, --auto-number number downloaded files starting from 00000
-o, --output TEMPLATE output filename template. Use %(title)s to
get the title, %(uploader)s for the
uploader name, %(uploader_id)s for the
@ -131,17 +164,19 @@ which means you can modify it, redistribute it or use it however you like.
%(upload_date)s for the upload date
(YYYYMMDD), %(extractor)s for the provider
(youtube, metacafe, etc), %(id)s for the
video id, %(playlist)s for the playlist the
video id, %(playlist_title)s,
%(playlist_id)s, or %(playlist)s (=title if
present, ID otherwise) for the playlist the
video is in, %(playlist_index)s for the
position in the playlist and %% for a
literal percent. %(height)s and %(width)s
for the width and height of the video
format. %(resolution)s for a textual
position in the playlist. %(height)s and
%(width)s for the width and height of the
video format. %(resolution)s for a textual
description of the resolution of the video
format. Use - to output to stdout. Can also
be used to download to a different
directory, for example with -o '/my/downloa
ds/%(uploader)s/%(title)s-%(id)s.%(ext)s' .
format. %% for a literal percent. Use - to
output to stdout. Can also be used to
download to a different directory, for
example with -o '/my/downloads/%(uploader)s
/%(title)s-%(id)s.%(ext)s' .
--autonumber-size NUMBER Specifies the number of digits in
%(autonumber)s when it is present in output
filename template or --auto-number option
@ -149,6 +184,9 @@ which means you can modify it, redistribute it or use it however you like.
--restrict-filenames Restrict filenames to only ASCII
characters, and avoid "&" and spaces in
filenames
-A, --auto-number [deprecated; use -o
"%(autonumber)s-%(title)s.%(ext)s" ] number
downloaded files starting from 00000
-t, --title [deprecated] use title in file name
(default)
-l, --literal [deprecated] alias of --title
@ -158,7 +196,8 @@ which means you can modify it, redistribute it or use it however you like.
downloads if possible.
--no-continue do not resume partially downloaded files
(restart from beginning)
--no-part do not use .part files
--no-part do not use .part files - write directly
into output file
--no-mtime do not use the Last-modified header to set
the file modification time
--write-description write video description to a .description
@ -166,7 +205,6 @@ which means you can modify it, redistribute it or use it however you like.
--write-info-json write video metadata to a .info.json file
--write-annotations write video annotations to a .annotation
file
--write-thumbnail write thumbnail image to disk
--load-info FILE json file containing the video information
(created with the "--write-json" option)
--cookies FILE file to read cookies from and dump cookie
@ -181,6 +219,12 @@ which means you can modify it, redistribute it or use it however you like.
--no-cache-dir Disable filesystem caching
--rm-cache-dir Delete all filesystem cache files
## Thumbnail images:
--write-thumbnail write thumbnail image to disk
--write-all-thumbnails write all thumbnail image formats to disk
--list-thumbnails Simulate and list all available thumbnail
formats
## Verbosity / Simulation Options:
-q, --quiet activates quiet mode
--no-warnings Ignore warnings
@ -198,6 +242,12 @@ which means you can modify it, redistribute it or use it however you like.
-j, --dump-json simulate, quiet but print JSON information.
See --output for a description of available
keys.
-J, --dump-single-json simulate, quiet but print JSON information
for each command-line argument. If the URL
refers to a playlist, dump the whole
playlist information in a single line.
--print-json Be quiet and print the video information as
JSON (video is still being downloaded).
--newline output progress bar as new lines
--no-progress do not print progress bar
--console-title display progress in console titlebar
@ -208,6 +258,10 @@ which means you can modify it, redistribute it or use it however you like.
files in the current directory to debug
problems
--print-traffic Display sent and read HTTP traffic
-C, --call-home Contact the youtube-dl server for
debugging.
--no-call-home Do NOT contact the youtube-dl server for
debugging.
## Workarounds:
--encoding ENCODING Force the specified encoding (experimental)
@ -216,7 +270,7 @@ which means you can modify it, redistribute it or use it however you like.
information about the video. (Currently
supported only for YouTube)
--user-agent UA specify a custom user agent
--referer REF specify a custom referer, use if the video
--referer URL specify a custom referer, use if the video
access is restricted to one domain
--add-header FIELD:VALUE specify a custom HTTP header and its value,
separated by a colon ':'. You can use this
@ -224,20 +278,49 @@ which means you can modify it, redistribute it or use it however you like.
--bidi-workaround Work around terminals that lack
bidirectional text support. Requires bidiv
or fribidi executable in PATH
--sleep-interval SECONDS Number of seconds to sleep before each
download.
## Video Format Options:
-f, --format FORMAT video format code, specify the order of
preference using slashes: "-f 22/17/18".
"-f mp4" and "-f flv" are also supported.
You can also use the special names "best",
"bestvideo", "bestaudio", "worst",
"worstvideo" and "worstaudio". By default,
youtube-dl will pick the best quality.
preference using slashes, as in -f 22/17/18
. Instead of format codes, you can select
by extension for the extensions aac, m4a,
mp3, mp4, ogg, wav, webm. You can also use
the special names "best", "bestvideo",
"bestaudio", "worst". You can filter the
video results by putting a condition in
brackets, as in -f "best[height=720]" (or
-f "[filesize>10M]"). This works for
filesize, height, width, tbr, abr, vbr, and
fps and the comparisons <, <=, >, >=, =, !=
. Formats for which the value is not known
are excluded unless you put a question mark
(?) after the operator. You can combine
format filters, so -f "[height <=?
720][tbr>500]" selects up to 720p videos
(or videos where the height is not known)
with a bitrate of at least 500 KBit/s. By
default, youtube-dl will pick the best
quality. Use commas to download multiple
audio formats, such as -f
136/137/mp4/bestvideo,140/m4a/bestaudio.
You can merge the video and audio of two
formats into a single file using -f <video-
format>+<audio-format> (requires ffmpeg or
avconv), for example -f
bestvideo+bestaudio.
--all-formats download all available video formats
--prefer-free-formats prefer free video formats unless a specific
one is requested
--max-quality FORMAT highest quality format to download
-F, --list-formats list all available formats
--youtube-skip-dash-manifest Do not download the DASH manifest on
YouTube videos
--merge-output-format FORMAT If a merge is required (e.g.
bestvideo+bestaudio), output to given
container format. One of mkv, mp4, ogg,
webm, flv.Ignored if no merge is required
## Subtitle Options:
--write-sub write subtitle file
@ -253,8 +336,9 @@ which means you can modify it, redistribute it or use it however you like.
language tags like 'en,pt'
## Authentication Options:
-u, --username USERNAME account username
-p, --password PASSWORD account password
-u, --username USERNAME login with this account ID
-p, --password PASSWORD account password. If this option is left
out, youtube-dl will ask interactively.
-2, --twofactor TWOFACTOR two-factor auth code
-n, --netrc use .netrc authentication data
--video-password PASSWORD video password (vimeo, smotri)
@ -264,7 +348,7 @@ which means you can modify it, redistribute it or use it however you like.
(requires ffmpeg or avconv and ffprobe or
avprobe)
--audio-format FORMAT "best", "aac", "vorbis", "mp3", "m4a",
"opus", or "wav"; best by default
"opus", or "wav"; "best" by default
--audio-quality QUALITY ffmpeg/avconv audio quality specification,
insert a value between 0 (better) and 9
(worse) for VBR or a specific bitrate like
@ -284,6 +368,11 @@ which means you can modify it, redistribute it or use it however you like.
--add-metadata write metadata to the video file
--xattrs write metadata to the video file's xattrs
(using dublin core and xdg standards)
--fixup POLICY Automatically correct known faults of the
file. One of never (do nothing), warn (only
emit a warning), detect_or_warn(the
default; fix file if we can, warn
otherwise)
--prefer-avconv Prefer avconv over ffmpeg for running the
postprocessors (default)
--prefer-ffmpeg Prefer ffmpeg over avconv for running the
@ -295,7 +384,7 @@ which means you can modify it, redistribute it or use it however you like.
# CONFIGURATION
You can configure youtube-dl by placing default arguments (such as `--extract-audio --no-mtime` to always extract the audio and not copy the mtime) into `/etc/youtube-dl.conf` and/or `~/.config/youtube-dl/config`. On Windows, the configuration file locations are `%APPDATA%\youtube-dl\config.txt` and `C:\Users\<Yourname>\youtube-dl.conf`.
You can configure youtube-dl by placing default arguments (such as `--extract-audio --no-mtime` to always extract the audio and not copy the mtime) into `/etc/youtube-dl.conf` and/or `~/.config/youtube-dl/config`. On Windows, the configuration file locations are `%APPDATA%\youtube-dl\config.txt` and `C:\Users\<user name>\youtube-dl.conf`.
# OUTPUT TEMPLATE
@ -345,21 +434,34 @@ $ youtube-dl --dateafter 20000101 --datebefore 20091231
# FAQ
### I'm getting an error `Unable to extract OpenGraph title` on YouTube playlists
### How do I update youtube-dl?
YouTube changed their playlist format in March 2014 and later on, so you'll need at least youtube-dl 2014.07.25 to download all YouTube videos.
If you've followed [our manual installation instructions](http://rg3.github.io/youtube-dl/download.html), you can simply run `youtube-dl -U` (or, on Linux, `sudo youtube-dl -U`).
If you have installed youtube-dl with a package manager, pip, setup.py or a tarball, please use that to update. Note that Ubuntu packages do not seem to get updated anymore. Since we are not affiliated with Ubuntu, there is little we can do. Feel free to report bugs to the Ubuntu packaging guys - all they have to do is update the package to a somewhat recent version.
If you have used pip, a simple `sudo pip install -U youtube-dl` is sufficient to update.
Alternatively, uninstall the youtube-dl package and follow [our manual installation instructions](http://rg3.github.io/youtube-dl/download.html). In a pinch, this should do if you used `apt-get` before to install youtube-dl:
If you have installed youtube-dl using a package manager like *apt-get* or *yum*, use the standard system update mechanism to update. Note that distribution packages are often outdated. As a rule of thumb, youtube-dl releases at least once a month, and often weekly or even daily. Simply go to http://yt-dl.org/ to find out the current version. Unfortunately, there is nothing we youtube-dl developers can do if your distributions serves a really outdated version. You can (and should) complain to your distribution in their bugtracker or support forum.
As a last resort, you can also uninstall the version installed by your package manager and follow our manual installation instructions. For that, remove the distribution's package, with a line like
sudo apt-get remove -y youtube-dl
Afterwards, simply follow [our manual installation instructions](http://rg3.github.io/youtube-dl/download.html):
```
sudo apt-get remove -y youtube-dl
sudo wget https://yt-dl.org/latest/youtube-dl -O /usr/local/bin/youtube-dl
sudo chmod a+x /usr/local/bin/youtube-dl
hash -r
```
Again, from then on you'll be able to update with `sudo youtube-dl -U`.
### I'm getting an error `Unable to extract OpenGraph title` on YouTube playlists
YouTube changed their playlist format in March 2014 and later on, so you'll need at least youtube-dl 2014.07.25 to download all YouTube videos.
If you have installed youtube-dl with a package manager, pip, setup.py or a tarball, please use that to update. Note that Ubuntu packages do not seem to get updated anymore. Since we are not affiliated with Ubuntu, there is little we can do. Feel free to [report bugs](https://bugs.launchpad.net/ubuntu/+source/youtube-dl/+filebug) to the [Ubuntu packaging guys](mailto:ubuntu-motu@lists.ubuntu.com?subject=outdated%20version%20of%20youtube-dl) - all they have to do is update the package to a somewhat recent version. See above for a way to update.
### Do I always have to pass in `--max-quality FORMAT`, or `-citw`?
By default, youtube-dl intends to have the best options (incidentally, if you have a convincing case that these should be different, [please file an issue where you explain that](https://yt-dl.org/bug)). Therefore, it is unnecessary and sometimes harmful to copy long option strings from webpages. In particular, `--max-quality` *limits* the video quality (so if you want the best quality, do NOT pass it in), and the only option out of `-citw` that is regularly useful is `-i`.
@ -376,9 +478,15 @@ Apparently YouTube requires you to pass a CAPTCHA test if you download too much.
Once the video is fully downloaded, use any video player, such as [vlc](http://www.videolan.org) or [mplayer](http://www.mplayerhq.hu/).
### The links provided by youtube-dl -g are not working anymore
### I extracted a video URL with -g, but it does not play on another machine / in my webbrowser.
The URLs youtube-dl outputs require the downloader to have the correct cookies. Use the `--cookies` option to write the required cookies into a file, and advise your downloader to read cookies from that file. Some sites also require a common user agent to be used, use `--dump-user-agent` to see the one in use by youtube-dl.
It depends a lot on the service. In many cases, requests for the video (to download/play it) must come from the same IP address and with the same cookies. Use the `--cookies` option to write the required cookies into a file, and advise your downloader to read cookies from that file. Some sites also require a common user agent to be used, use `--dump-user-agent` to see the one in use by youtube-dl.
It may be beneficial to use IPv6; in some cases, the restrictions are only applied to IPv4. Some services (sometimes only for a subset of videos) do not restrict the video URL by IP address, cookie, or user-agent, but these are the exception rather than the rule.
Please bear in mind that some URL protocols are **not** supported by browsers out of the box, including RTMP. If you are using -g, your own downloader must support these as well.
If you want to play the video on a machine that is not running youtube-dl, you can relay the video content from the machine that runs youtube-dl. You can use `-o -` to let youtube-dl stream a video to stdout, or simply allow the player to download the files written by youtube-dl in turn.
### ERROR: no fmt_url_map or conn information found in video info
@ -405,6 +513,33 @@ Since June 2012 (#342) youtube-dl is packed as an executable zipfile, simply unz
To run the exe you need to install first the [Microsoft Visual C++ 2008 Redistributable Package](http://www.microsoft.com/en-us/download/details.aspx?id=29).
### On Windows, how should I set up ffmpeg and youtube-dl? Where should I put the exe files?
If you put youtube-dl and ffmpeg in the same directory that you're running the command from, it will work, but that's rather cumbersome.
To make a different directory work - either for ffmpeg, or for youtube-dl, or for both - simply create the directory (say, `C:\bin`, or `C:\Users\<User name>\bin`), put all the executables directly in there, and then [set your PATH environment variable](https://www.java.com/en/download/help/path.xml) to include that directory.
From then on, after restarting your shell, you will be able to access both youtube-dl and ffmpeg (and youtube-dl will be able to find ffmpeg) by simply typing `youtube-dl` or `ffmpeg`, no matter what directory you're in.
### How do I put downloads into a specific folder?
Use the `-o` to specify an [output template](#output-template), for example `-o "/home/user/videos/%(title)s-%(id)s.%(ext)s"`. If you want this for all of your downloads, put the option into your [configuration file](#configuration).
### How do I download a video starting with a `-` ?
Either prepend `http://www.youtube.com/watch?v=` or separate the ID from the options with `--`:
youtube-dl -- -wNyEUrxzFU
youtube-dl "http://www.youtube.com/watch?v=-wNyEUrxzFU"
### How can I detect whether a given URL is supported by youtube-dl?
For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from http://example.com/v/1234567 to http://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug.
It is *not* possible to detect whether a URL is supported or not. That's because youtube-dl contains a generic extractor which matches **all** URLs. You may be tempted to disable, exclude, or remove the generic extractor, but the generic extractor not only allows users to extract videos from lots of websites that embed a video from another service, but may also be used to extract video from a service that it's hosting itself. Therefore, we neither recommend nor support disabling, excluding, or removing the generic extractor.
If you want to find out whether a given URL is supported, simply call youtube-dl with it. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run youtube-dl on the console) or catching an `UnsupportedError` exception if you run it from a Python program.
# DEVELOPER INSTRUCTIONS
Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
@ -439,8 +574,6 @@ If you want to add support for a new site, you can follow this quick list (assum
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
@ -448,7 +581,7 @@ If you want to add support for a new site, you can follow this quick list (assum
_VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)'
_TEST = {
'url': 'http://yourextractor.com/watch/42',
'md5': 'TODO: md5 sum of the first 10KiB of the video file',
'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
'info_dict': {
'id': '42',
'ext': 'mp4',
@ -463,24 +596,24 @@ If you want to add support for a new site, you can follow this quick list (assum
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
# TODO more code goes here, for example ...
webpage = self._download_webpage(url, video_id)
title = self._html_search_regex(r'<h1>(.*?)</h1>', webpage, 'title')
return {
'id': video_id,
'title': title,
'description': self._og_search_description(webpage),
# TODO more properties (see youtube_dl/extractor/common.py)
}
```
5. Add an import in [`youtube_dl/extractor/__init__.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/__init__.py).
6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will be then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
7. Have a look at [`youtube_dl/common/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L38). Add tests and code for as many as you want.
8. If you can, check the code with [pyflakes](https://pypi.python.org/pypi/pyflakes) (a good idea) and [pep8](https://pypi.python.org/pypi/pep8) (optional, ignore E501).
9. When the tests pass, [add](https://www.kernel.org/pub/software/scm/git/docs/git-add.html) the new files and [commit](https://www.kernel.org/pub/software/scm/git/docs/git-commit.html) them and [push](https://www.kernel.org/pub/software/scm/git/docs/git-push.html) the result, like this:
8. If you can, check the code with [flake8](https://pypi.python.org/pypi/flake8).
9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
$ git add youtube_dl/extractor/__init__.py
$ git add youtube_dl/extractor/yourextractor.py
@ -491,15 +624,67 @@ If you want to add support for a new site, you can follow this quick list (assum
In any case, thank you very much for your contributions!
# EMBEDDING YOUTUBE-DL
youtube-dl makes the best effort to be a good command-line program, and thus should be callable from any programming language. If you encounter any problems parsing its output, feel free to [create a report](https://github.com/rg3/youtube-dl/issues/new).
From a Python program, you can embed youtube-dl in a more powerful fashion, like this:
```python
import youtube_dl
ydl_opts = {}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
```
Most likely, you'll want to use various options. For a list of what can be done, have a look at [youtube_dl/YoutubeDL.py](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L69). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file:
```python
import youtube_dl
class MyLogger(object):
def debug(self, msg):
pass
def warning(self, msg):
pass
def error(self, msg):
print(msg)
def my_hook(d):
if d['status'] == 'finished':
print('Done downloading, now converting ...')
ydl_opts = {
'format': 'bestaudio/best',
'postprocessors': [{
'key': 'FFmpegExtractAudio',
'preferredcodec': 'mp3',
'preferredquality': '192',
}],
'logger': MyLogger(),
'progress_hooks': [my_hook],
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
```
# BUGS
Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues> . Unless you were prompted so or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email.
Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues> . Unless you were prompted so or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the irc channel #youtube-dl on freenode.
Please include the full output of the command when run with `--verbose`. The output (including the first lines) contain important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever.
**Please include the full output of youtube-dl when run with `-v`**.
For discussions, join us in the irc channel #youtube-dl on freenode.
The output (including the first lines) contain important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever.
When you submit a request, please re-read it once to avoid a couple of mistakes (you can and should use this as a checklist):
Please re-read your issue once again to avoid a couple of common mistakes (you can and should use this as a checklist):
### Is the description of the issue itself sufficient?

View File

@ -1,4 +1,6 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import os
from os.path import dirname as dirn
import sys
@ -9,6 +11,7 @@ import youtube_dl
BASH_COMPLETION_FILE = "youtube-dl.bash-completion"
BASH_COMPLETION_TEMPLATE = "devscripts/bash-completion.in"
def build_completion(opt_parser):
opts_flag = []
for group in opt_parser.option_groups:

View File

@ -233,6 +233,7 @@ def rmtree(path):
#==============================================================================
class BuildError(Exception):
def __init__(self, output, code=500):
self.output = output

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
"""
This script employs a VERY basic heuristic ('porn' in webpage.lower()) to check

View File

@ -0,0 +1,5 @@
{{commands}}
complete --command youtube-dl --arguments ":ytfavorites :ytrecommended :ytsubscriptions :ytwatchlater :ythistory"

48
devscripts/fish-completion.py Executable file
View File

@ -0,0 +1,48 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import optparse
import os
from os.path import dirname as dirn
import sys
sys.path.append(dirn(dirn((os.path.abspath(__file__)))))
import youtube_dl
from youtube_dl.utils import shell_quote
FISH_COMPLETION_FILE = 'youtube-dl.fish'
FISH_COMPLETION_TEMPLATE = 'devscripts/fish-completion.in'
EXTRA_ARGS = {
'recode-video': ['--arguments', 'mp4 flv ogg webm mkv', '--exclusive'],
# Options that need a file parameter
'download-archive': ['--require-parameter'],
'cookies': ['--require-parameter'],
'load-info': ['--require-parameter'],
'batch-file': ['--require-parameter'],
}
def build_completion(opt_parser):
commands = []
for group in opt_parser.option_groups:
for option in group.option_list:
long_option = option.get_opt_string().strip('-')
complete_cmd = ['complete', '--command', 'youtube-dl', '--long-option', long_option]
if option._short_opts:
complete_cmd += ['--short-option', option._short_opts[0].strip('-')]
if option.help != optparse.SUPPRESS_HELP:
complete_cmd += ['--description', option.help]
complete_cmd.extend(EXTRA_ARGS.get(long_option, []))
commands.append(shell_quote(complete_cmd))
with open(FISH_COMPLETION_TEMPLATE) as f:
template = f.read()
filled_template = template.replace('{{commands}}', '\n'.join(commands))
with open(FISH_COMPLETION_FILE, 'w') as f:
f.write(filled_template)
parser = youtube_dl.parseOpts()[0]
build_completion(parser)

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python3
from __future__ import unicode_literals
import json
import sys

View File

@ -1,8 +1,7 @@
#!/usr/bin/env python3
from __future__ import unicode_literals
import hashlib
import shutil
import subprocess
import tempfile
import urllib.request
import json

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python3
from __future__ import unicode_literals, with_statement
import rsa
import json
@ -29,4 +30,5 @@ signature = hexlify(rsa.pkcs1.sign(json.dumps(versions_info, sort_keys=True).enc
print('signature: ' + signature)
versions_info['signature'] = signature
json.dump(versions_info, open('update/versions.json', 'w'), indent=4, sort_keys=True)
with open('update/versions.json', 'w') as versionsf:
json.dump(versions_info, versionsf, indent=4, sort_keys=True)

View File

@ -1,7 +1,7 @@
#!/usr/bin/env python
# coding: utf-8
from __future__ import with_statement
from __future__ import with_statement, unicode_literals
import datetime
import glob
@ -13,7 +13,7 @@ year = str(datetime.datetime.now().year)
for fn in glob.glob('*.html*'):
with io.open(fn, encoding='utf-8') as f:
content = f.read()
newc = re.sub(u'(?P<copyright>Copyright © 2006-)(?P<year>[0-9]{4})', u'Copyright © 2006-' + year, content)
newc = re.sub(r'(?P<copyright>Copyright © 2006-)(?P<year>[0-9]{4})', 'Copyright © 2006-' + year, content)
if content != newc:
tmpFn = fn + '.part'
with io.open(tmpFn, 'wt', encoding='utf-8') as outf:

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python3
from __future__ import unicode_literals
import datetime
import io
@ -73,4 +74,3 @@ atom_template = atom_template.replace('@ENTRIES@', entries_str)
with io.open('update/releases.atom', 'w', encoding='utf-8') as atom_file:
atom_file.write(atom_template)

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python3
from __future__ import unicode_literals
import sys
import os
@ -9,19 +10,20 @@ sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(
import youtube_dl
def main():
with open('supportedsites.html.in', 'r', encoding='utf-8') as tmplf:
template = tmplf.read()
ie_htmls = []
for ie in sorted(youtube_dl.gen_extractors(), key=lambda i: i.IE_NAME.lower()):
for ie in youtube_dl.list_extractors(age_limit=None):
ie_html = '<b>{}</b>'.format(ie.IE_NAME)
ie_desc = getattr(ie, 'IE_DESC', None)
if ie_desc is False:
continue
elif ie_desc is not None:
ie_html += ': {}'.format(ie.IE_DESC)
if ie.working() == False:
if not ie.working():
ie_html += ' (Currently broken)'
ie_htmls.append('<li>{}</li>'.format(ie_html))

32
devscripts/make_contributing.py Executable file
View File

@ -0,0 +1,32 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import io
import optparse
import re
def main():
parser = optparse.OptionParser(usage='%prog INFILE OUTFILE')
options, args = parser.parse_args()
if len(args) != 2:
parser.error('Expected an input and an output filename')
infile, outfile = args
with io.open(infile, encoding='utf-8') as inf:
readme = inf.read()
bug_text = re.search(
r'(?s)#\s*BUGS\s*[^\n]*\s*(.*?)#\s*COPYRIGHT', readme).group(1)
dev_text = re.search(
r'(?s)(#\s*DEVELOPER INSTRUCTIONS.*?)#\s*EMBEDDING YOUTUBE-DL',
readme).group(1)
out = bug_text + dev_text
with io.open(outfile, 'w', encoding='utf-8') as outf:
outf.write(out)
if __name__ == '__main__':
main()

View File

@ -1,3 +1,5 @@
from __future__ import unicode_literals
import io
import sys
import re

View File

@ -0,0 +1,45 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import io
import optparse
import os
import sys
# Import youtube_dl
ROOT_DIR = os.path.join(os.path.dirname(__file__), '..')
sys.path.append(ROOT_DIR)
import youtube_dl
def main():
parser = optparse.OptionParser(usage='%prog OUTFILE.md')
options, args = parser.parse_args()
if len(args) != 1:
parser.error('Expected an output filename')
outfile, = args
def gen_ies_md(ies):
for ie in ies:
ie_md = '**{0}**'.format(ie.IE_NAME)
ie_desc = getattr(ie, 'IE_DESC', None)
if ie_desc is False:
continue
if ie_desc is not None:
ie_md += ': {0}'.format(ie.IE_DESC)
if not ie.working():
ie_md += ' (Currently broken)'
yield ie_md
ies = sorted(youtube_dl.gen_extractors(), key=lambda i: i.IE_NAME.lower())
out = '# Supported sites\n' + ''.join(
' - ' + md + '\n'
for md in gen_ies_md(ies))
with io.open(outfile, 'w', encoding='utf-8') as outf:
outf.write(out)
if __name__ == '__main__':
main()

View File

@ -1,3 +1,4 @@
from __future__ import unicode_literals
import io
import os.path
@ -10,8 +11,19 @@ README_FILE = os.path.join(ROOT_DIR, 'README.md')
with io.open(README_FILE, encoding='utf-8') as f:
readme = f.read()
PREFIX = '%YOUTUBE-DL(1)\n\n# NAME\n'
readme = re.sub(r'(?s)# INSTALLATION.*?(?=# DESCRIPTION)', '', readme)
PREFIX = '''%YOUTUBE-DL(1)
# NAME
youtube\-dl \- download videos from youtube.com or other video platforms
# SYNOPSIS
**youtube-dl** \[OPTIONS\] URL [URL...]
'''
readme = re.sub(r'(?s)^.*?(?=# DESCRIPTION)', '', readme)
readme = re.sub(r'\s+youtube-dl \[OPTIONS\] URL \[URL\.\.\.\]', '', readme)
readme = PREFIX + readme
if sys.version_info < (3, 0):

View File

@ -1,40 +0,0 @@
#!/usr/bin/env python
import sys, os
try:
import urllib.request as compat_urllib_request
except ImportError: # Python 2
import urllib2 as compat_urllib_request
sys.stderr.write(u'Hi! We changed distribution method and now youtube-dl needs to update itself one more time.\n')
sys.stderr.write(u'This will only happen once. Simply press enter to go on. Sorry for the trouble!\n')
sys.stderr.write(u'The new location of the binaries is https://github.com/rg3/youtube-dl/downloads, not the git repository.\n\n')
try:
raw_input()
except NameError: # Python 3
input()
filename = sys.argv[0]
API_URL = "https://api.github.com/repos/rg3/youtube-dl/downloads"
BIN_URL = "https://github.com/downloads/rg3/youtube-dl/youtube-dl"
if not os.access(filename, os.W_OK):
sys.exit('ERROR: no write permissions on %s' % filename)
try:
urlh = compat_urllib_request.urlopen(BIN_URL)
newcontent = urlh.read()
urlh.close()
except (IOError, OSError) as err:
sys.exit('ERROR: unable to download latest version')
try:
with open(filename, 'wb') as outf:
outf.write(newcontent)
except (IOError, OSError) as err:
sys.exit('ERROR: unable to overwrite current version')
sys.stderr.write(u'Done! Now you can run youtube-dl.\n')

View File

@ -1,12 +0,0 @@
from distutils.core import setup
import py2exe
py2exe_options = {
"bundle_files": 1,
"compressed": 1,
"optimize": 2,
"dist_dir": '.',
"dll_excludes": ['w9xpopen.exe']
}
setup(console=['youtube-dl.py'], options={ "py2exe": py2exe_options }, zipfile=None)

View File

@ -1,102 +0,0 @@
#!/usr/bin/env python
import sys, os
import urllib2
import json, hashlib
def rsa_verify(message, signature, key):
from struct import pack
from hashlib import sha256
from sys import version_info
def b(x):
if version_info[0] == 2: return x
else: return x.encode('latin1')
assert(type(message) == type(b('')))
block_size = 0
n = key[0]
while n:
block_size += 1
n >>= 8
signature = pow(int(signature, 16), key[1], key[0])
raw_bytes = []
while signature:
raw_bytes.insert(0, pack("B", signature & 0xFF))
signature >>= 8
signature = (block_size - len(raw_bytes)) * b('\x00') + b('').join(raw_bytes)
if signature[0:2] != b('\x00\x01'): return False
signature = signature[2:]
if not b('\x00') in signature: return False
signature = signature[signature.index(b('\x00'))+1:]
if not signature.startswith(b('\x30\x31\x30\x0D\x06\x09\x60\x86\x48\x01\x65\x03\x04\x02\x01\x05\x00\x04\x20')): return False
signature = signature[19:]
if signature != sha256(message).digest(): return False
return True
sys.stderr.write(u'Hi! We changed distribution method and now youtube-dl needs to update itself one more time.\n')
sys.stderr.write(u'This will only happen once. Simply press enter to go on. Sorry for the trouble!\n')
sys.stderr.write(u'From now on, get the binaries from http://rg3.github.com/youtube-dl/download.html, not from the git repository.\n\n')
raw_input()
filename = sys.argv[0]
UPDATE_URL = "http://rg3.github.io/youtube-dl/update/"
VERSION_URL = UPDATE_URL + 'LATEST_VERSION'
JSON_URL = UPDATE_URL + 'versions.json'
UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537)
if not os.access(filename, os.W_OK):
sys.exit('ERROR: no write permissions on %s' % filename)
exe = os.path.abspath(filename)
directory = os.path.dirname(exe)
if not os.access(directory, os.W_OK):
sys.exit('ERROR: no write permissions on %s' % directory)
try:
versions_info = urllib2.urlopen(JSON_URL).read().decode('utf-8')
versions_info = json.loads(versions_info)
except:
sys.exit(u'ERROR: can\'t obtain versions info. Please try again later.')
if not 'signature' in versions_info:
sys.exit(u'ERROR: the versions file is not signed or corrupted. Aborting.')
signature = versions_info['signature']
del versions_info['signature']
if not rsa_verify(json.dumps(versions_info, sort_keys=True), signature, UPDATES_RSA_KEY):
sys.exit(u'ERROR: the versions file signature is invalid. Aborting.')
version = versions_info['versions'][versions_info['latest']]
try:
urlh = urllib2.urlopen(version['exe'][0])
newcontent = urlh.read()
urlh.close()
except (IOError, OSError) as err:
sys.exit('ERROR: unable to download latest version')
newcontent_hash = hashlib.sha256(newcontent).hexdigest()
if newcontent_hash != version['exe'][1]:
sys.exit(u'ERROR: the downloaded file hash does not match. Aborting.')
try:
with open(exe + '.new', 'wb') as outf:
outf.write(newcontent)
except (IOError, OSError) as err:
sys.exit(u'ERROR: unable to write the new version')
try:
bat = os.path.join(directory, 'youtube-dl-updater.bat')
b = open(bat, 'w')
b.write("""
echo Updating youtube-dl...
ping 127.0.0.1 -n 5 -w 1000 > NUL
move /Y "%s.new" "%s"
del "%s"
\n""" %(exe, exe, bat))
b.close()
os.startfile(bat)
except (IOError, OSError) as err:
sys.exit('ERROR: unable to overwrite current version')
sys.stderr.write(u'Done! Now you can run youtube-dl.\n')

View File

@ -0,0 +1,28 @@
#compdef youtube-dl
__youtube_dl() {
local curcontext="$curcontext" fileopts diropts cur prev
typeset -A opt_args
fileopts="{{fileopts}}"
diropts="{{diropts}}"
cur=$words[CURRENT]
case $cur in
:)
_arguments '*: :(::ytfavorites ::ytrecommended ::ytsubscriptions ::ytwatchlater ::ythistory)'
;;
*)
prev=$words[CURRENT-1]
if [[ ${prev} =~ ${fileopts} ]]; then
_path_files
elif [[ ${prev} =~ ${diropts} ]]; then
_path_files -/
elif [[ ${prev} == "--recode-video" ]]; then
_arguments '*: :(mp4 flv ogg webm mkv)'
else
_arguments '*: :({{flags}})'
fi
;;
esac
}
__youtube_dl

48
devscripts/zsh-completion.py Executable file
View File

@ -0,0 +1,48 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import os
from os.path import dirname as dirn
import sys
sys.path.append(dirn(dirn((os.path.abspath(__file__)))))
import youtube_dl
ZSH_COMPLETION_FILE = "youtube-dl.zsh"
ZSH_COMPLETION_TEMPLATE = "devscripts/zsh-completion.in"
def build_completion(opt_parser):
opts = [opt for group in opt_parser.option_groups
for opt in group.option_list]
opts_file = [opt for opt in opts if opt.metavar == "FILE"]
opts_dir = [opt for opt in opts if opt.metavar == "DIR"]
fileopts = []
for opt in opts_file:
if opt._short_opts:
fileopts.extend(opt._short_opts)
if opt._long_opts:
fileopts.extend(opt._long_opts)
diropts = []
for opt in opts_dir:
if opt._short_opts:
diropts.extend(opt._short_opts)
if opt._long_opts:
diropts.extend(opt._long_opts)
flags = [opt.get_opt_string() for opt in opts]
with open(ZSH_COMPLETION_TEMPLATE) as f:
template = f.read()
template = template.replace("{{fileopts}}", "|".join(fileopts))
template = template.replace("{{diropts}}", "|".join(diropts))
template = template.replace("{{flags}}", " ".join(flags))
with open(ZSH_COMPLETION_FILE, "w") as f:
f.write(template)
parser = youtube_dl.parseOpts()[0]
build_completion(parser)

View File

@ -44,8 +44,8 @@ copyright = u'2014, Ricardo Garcia Gonzalez'
# built documents.
#
# The short X.Y version.
import youtube_dl
version = youtube_dl.__version__
from youtube_dl.version import __version__
version = __version__
# The full version, including alpha/beta/rc tags.
release = version

500
docs/supportedsites.md Normal file
View File

@ -0,0 +1,500 @@
# Supported sites
- **1up.com**
- **220.ro**
- **24video**
- **3sat**
- **4tube**
- **56.com**
- **5min**
- **8tracks**
- **9gag**
- **abc.net.au**
- **AcademicEarth:Course**
- **AddAnime**
- **AdobeTV**
- **AdultSwim**
- **Aftonbladet**
- **AlJazeera**
- **Allocine**
- **anitube.se**
- **AnySex**
- **Aparat**
- **AppleTrailers**
- **archive.org**: archive.org videos
- **ARD**
- **ARD:mediathek**
- **arte.tv**
- **arte.tv:+7**
- **arte.tv:concert**
- **arte.tv:creative**
- **arte.tv:ddc**
- **arte.tv:embed**
- **arte.tv:future**
- **audiomack**
- **AUEngine**
- **Azubu**
- **bambuser**
- **bambuser:channel**
- **Bandcamp**
- **Bandcamp:album**
- **bbc.co.uk**: BBC iPlayer
- **Beeg**
- **BehindKink**
- **Bet**
- **Bild**: Bild.de
- **BiliBili**
- **blinkx**
- **blip.tv:user**
- **BlipTV**
- **Bloomberg**
- **Bpb**: Bundeszentrale für politische Bildung
- **BR**: Bayerischer Rundfunk Mediathek
- **Break**
- **Brightcove**
- **BuzzFeed**
- **BYUtv**
- **Canal13cl**
- **canalc2.tv**
- **Canalplus**: canalplus.fr, piwiplus.fr and d8.tv
- **CBS**
- **CBSNews**: CBS News
- **CeskaTelevize**
- **channel9**: Channel 9
- **Chilloutzone**
- **Cinchcast**
- **Cinemassacre**
- **clipfish**
- **cliphunter**
- **Clipsyndicate**
- **Cloudy**
- **Clubic**
- **cmt.com**
- **CNET**
- **CNN**
- **CNNBlogs**
- **CollegeHumor**
- **ComCarCoff**
- **ComedyCentral**
- **ComedyCentralShows**: The Daily Show / The Colbert Report
- **CondeNast**: Condé Nast media group: Condé Nast, GQ, Glamour, Vanity Fair, Vogue, W Magazine, WIRED
- **Cracked**
- **Criterion**
- **Crunchyroll**
- **crunchyroll:playlist**
- **CSpan**: C-SPAN
- **culturebox.francetvinfo.fr**
- **dailymotion**
- **dailymotion:playlist**
- **dailymotion:user**
- **daum.net**
- **DBTV**
- **DeezerPlaylist**
- **defense.gouv.fr**
- **Discovery**
- **divxstage**: DivxStage
- **Dotsub**
- **Dropbox**
- **DrTuber**
- **DRTV**
- **Dump**
- **dvtv**: http://video.aktualne.cz/
- **EbaumsWorld**
- **eHow**
- **Einthusan**
- **eitb.tv**
- **EllenTV**
- **EllenTV:clips**
- **ElPais**: El País
- **EMPFlix**
- **Engadget**
- **Eporner**
- **Escapist**
- **EveryonesMixtape**
- **exfm**: ex.fm
- **ExpoTV**
- **ExtremeTube**
- **facebook**
- **faz.net**
- **fc2**
- **fernsehkritik.tv**
- **fernsehkritik.tv:postecke**
- **Firedrive**
- **Firstpost**
- **firsttv**: Видеоархив - Первый канал
- **Flickr**
- **Folketinget**: Folketinget (ft.dk; Danish parliament)
- **Foxgay**
- **FoxNews**
- **france2.fr:generation-quoi**
- **FranceCulture**
- **FranceInter**
- **francetv**: France 2, 3, 4, 5 and Ô
- **francetvinfo.fr**
- **Freesound**
- **freespeech.org**
- **FreeVideo**
- **FunnyOrDie**
- **Gamekings**
- **GameOne**
- **gameone:playlist**
- **GameSpot**
- **GameStar**
- **Gametrailers**
- **GDCVault**
- **generic**: Generic downloader that works on some sites
- **GiantBomb**
- **Glide**: Glide mobile video messages (glide.me)
- **Globo**
- **GodTube**
- **GoldenMoustache**
- **Golem**
- **GorillaVid**: GorillaVid.in, daclips.in, movpod.in and fastvideo.in
- **Goshgay**
- **Grooveshark**
- **Groupon**
- **Hark**
- **Heise**
- **Helsinki**: helsinki.fi
- **HentaiStigma**
- **HornBunny**
- **HostingBulk**
- **HotNewHipHop**
- **Howcast**
- **HowStuffWorks**
- **HuffPost**: Huffington Post
- **Hypem**
- **Iconosquare**
- **ign.com**
- **imdb**: Internet Movie Database trailers
- **imdb:list**: Internet Movie Database lists
- **Ina**
- **InfoQ**
- **Instagram**
- **instagram:user**: Instagram user profile
- **InternetVideoArchive**
- **IPrima**
- **ivi**: ivi.ru
- **ivi:compilation**: ivi.ru compilations
- **Izlesene**
- **JadoreCettePub**
- **JeuxVideo**
- **Jove**
- **jpopsuki.tv**
- **Jukebox**
- **Kankan**
- **keek**
- **KeezMovies**
- **KhanAcademy**
- **KickStarter**
- **kontrtube**: KontrTube.ru - Труба зовёт
- **KrasView**: Красвью
- **Ku6**
- **la7.tv**
- **Laola1Tv**
- **lifenews**: LIFE | NEWS
- **LiveLeak**
- **livestream**
- **livestream:original**
- **lrt.lt**
- **lynda**: lynda.com videos
- **lynda:course**: lynda.com online courses
- **m6**
- **macgamestore**: MacGameStore trailers
- **mailru**: Видео@Mail.Ru
- **Malemotion**
- **MDR**
- **metacafe**
- **Metacritic**
- **Mgoon**
- **Minhateca**
- **MinistryGrid**
- **mitele.es**
- **mixcloud**
- **MLB**
- **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
- **Mofosex**
- **Mojvideo**
- **Moniker**: allmyvideos.net and vidspot.net
- **mooshare**: Mooshare.biz
- **Morningstar**: morningstar.com
- **Motherless**
- **Motorsport**: motorsport.com
- **MovieClips**
- **Moviezine**
- **movshare**: MovShare
- **MPORA**
- **MTV**
- **mtviggy.com**
- **mtvservices:embedded**
- **MuenchenTV**: münchen.tv
- **MusicPlayOn**
- **MusicVault**
- **muzu.tv**
- **MySpace**
- **MySpace:album**
- **MySpass**
- **myvideo**
- **MyVidster**
- **Naver**
- **NBA**
- **NBC**
- **NBCNews**
- **ndr**: NDR.de - Mediathek
- **NDTV**
- **NerdCubedFeed**
- **Newgrounds**
- **Newstube**
- **nfb**: National Film Board of Canada
- **nfl.com**
- **nhl.com**
- **nhl.com:videocenter**: NHL videocenter category
- **niconico**: ニコニコ動画
- **NiconicoPlaylist**
- **Noco**
- **Normalboots**
- **NosVideo**
- **novamov**: NovaMov
- **Nowness**
- **nowvideo**: NowVideo
- **npo.nl**
- **NRK**
- **NRKTV**
- **NTV**
- **Nuvid**
- **NYTimes**
- **ocw.mit.edu**
- **OktoberfestTV**
- **on.aol.com**
- **Ooyala**
- **orf:oe1**: Radio Österreich 1
- **orf:tvthek**: ORF TVthek
- **ORFFM4**: radio FM4
- **parliamentlive.tv**: UK parliament videos
- **Patreon**
- **PBS**
- **Phoenix**
- **Photobucket**
- **PlanetaPlay**
- **play.fm**
- **played.to**
- **Playvid**
- **plus.google**: Google Plus
- **pluzz.francetv.fr**
- **podomatic**
- **PornHd**
- **PornHub**
- **Pornotube**
- **PornoXO**
- **PromptFile**
- **prosiebensat1**: ProSiebenSat.1 Digital
- **Pyvideo**
- **QuickVid**
- **radio.de**
- **radiofrance**
- **Rai**
- **RBMARadio**
- **RedTube**
- **Restudy**
- **ReverbNation**
- **RingTV**
- **RottenTomatoes**
- **Roxwel**
- **RTBF**
- **RTLnow**
- **rtlxl.nl**
- **RTP**
- **RTS**: RTS.ch
- **rtve.es:alacarta**: RTVE a la carta
- **rtve.es:live**: RTVE.es live streams
- **RUHD**
- **rutube**: Rutube videos
- **rutube:channel**: Rutube channels
- **rutube:movie**: Rutube movies
- **rutube:person**: Rutube person videos
- **RUTV**: RUTV.RU
- **Sapo**: SAPO Vídeos
- **savefrom.net**
- **SBS**: sbs.com.au
- **SciVee**
- **screen.yahoo:search**: Yahoo screen search
- **Screencast**
- **ScreencastOMatic**
- **ScreenwaveMedia**
- **ServingSys**
- **Sexu**
- **SexyKarma**: Sexy Karma and Watch Indian Porn
- **Shared**
- **ShareSix**
- **Sina**
- **Slideshare**
- **Slutload**
- **smotri**: Smotri.com
- **smotri:broadcast**: Smotri.com broadcasts
- **smotri:community**: Smotri.com community videos
- **smotri:user**: Smotri.com user videos
- **Snotr**
- **Sockshare**
- **Sohu**
- **soundcloud**
- **soundcloud:playlist**
- **soundcloud:set**
- **soundcloud:user**
- **Soundgasm**
- **southpark.cc.com**
- **southpark.de**
- **Space**
- **Spankwire**
- **Spiegel**
- **Spiegel:Article**: Articles on spiegel.de
- **Spiegeltv**
- **Spike**
- **Sport5**
- **SportBox**
- **SportDeutschland**
- **SRMediathek**: Süddeutscher Rundfunk
- **stanfordoc**: Stanford Open ClassRoom
- **Steam**
- **streamcloud.eu**
- **StreamCZ**
- **SunPorno**
- **SWRMediathek**
- **Syfy**
- **SztvHu**
- **Tagesschau**
- **Tapely**
- **Tass**
- **teachertube**: teachertube.com videos
- **teachertube:user:collection**: teachertube.com user and collection videos
- **TeachingChannel**
- **Teamcoco**
- **TeamFour**
- **TechTalks**
- **techtv.mit.edu**
- **TED**
- **tegenlicht.vpro.nl**
- **TeleBruxelles**
- **telecinco.es**
- **TeleMB**
- **TenPlay**
- **TF1**
- **TheOnion**
- **ThePlatform**
- **TheSixtyOne**
- **ThisAV**
- **THVideo**
- **THVideoPlaylist**
- **tinypic**: tinypic.com videos
- **tlc.com**
- **tlc.de**
- **TMZ**
- **TNAFlix**
- **tou.tv**
- **Toypics**: Toypics user profile
- **ToypicsUser**: Toypics user profile
- **TrailerAddict** (Currently broken)
- **Trilulilu**
- **TruTube**
- **Tube8**
- **Tudou**
- **Tumblr**
- **TuneIn**
- **Turbo**
- **Tutv**
- **tv.dfb.de**
- **tvigle**: Интернет-телевидение Tvigle.ru
- **tvp.pl**
- **TVPlay**: TV3Play and related services
- **Twitch**
- **Ubu**
- **udemy**
- **udemy:course**
- **Unistra**
- **Urort**: NRK P3 Urørt
- **ustream**
- **ustream:channel**
- **Vbox7**
- **VeeHD**
- **Veoh**
- **Vesti**: Вести.Ru
- **Vevo**
- **VGTV**
- **vh1.com**
- **Vice**
- **Viddler**
- **video.google:search**: Google Video search
- **video.mit.edu**
- **VideoBam**
- **VideoDetective**
- **videofy.me**
- **videolectures.net**
- **VideoMega**
- **VideoPremium**
- **VideoTt**: video.tt - Your True Tube
- **videoweed**: VideoWeed
- **Vidme**
- **Vidzi**
- **viki**
- **vimeo**
- **vimeo:album**
- **vimeo:channel**
- **vimeo:group**
- **vimeo:likes**: Vimeo user likes
- **vimeo:review**: Review pages on vimeo
- **vimeo:user**
- **vimeo:watchlater**: Vimeo watch later list, "vimeowatchlater" keyword (requires authentication)
- **Vimple**: Vimple.ru
- **Vine**
- **vine:user**
- **vk.com**
- **vk.com:user-videos**: vk.com:All of a user's videos
- **Vodlocker**
- **Vporn**
- **VRT**
- **vube**: Vube.com
- **VuClip**
- **vulture.com**
- **Walla**
- **WashingtonPost**
- **wat.tv**
- **WayOfTheMaster**
- **WDR**
- **wdr:mobile**
- **WDRMaus**: Sendung mit der Maus
- **Weibo**
- **Wimp**
- **Wistia**
- **WorldStarHipHop**
- **wrzuta.pl**
- **XBef**
- **XboxClips**
- **XHamster**
- **XMinus**
- **XNXX**
- **XTube**
- **XTubeUser**: XTube user profile
- **XVideos**
- **Yahoo**: Yahoo screen and movies
- **YesJapan**
- **Ynet**
- **YouJizz**
- **Youku**
- **YouPorn**
- **YourUpload**
- **youtube**: YouTube.com
- **youtube:channel**: YouTube.com channels
- **youtube:favorites**: YouTube.com favourite videos, ":ytfav" for short (requires authentication)
- **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
- **youtube:playlist**: YouTube.com playlists
- **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
- **youtube:search**: YouTube.com searches
- **youtube:search:date**: YouTube.com searches, newest videos first
- **youtube:search_url**: YouTube.com search URLs
- **youtube:show**: YouTube.com (multi-season) shows
- **youtube:subscriptions**: YouTube.com subscriptions feed, "ytsubs" keyword (requires authentication)
- **youtube:toplist**: YouTube.com top lists, "yttoplist:{channel}:{list title}" (Example: "yttoplist:music:Top Tracks")
- **youtube:user**: YouTube.com user videos (URL or "ytuser" keyword)
- **youtube:watch_later**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
- **ZDF**
- **ZDFChannel**
- **zingmp3:album**: mp3.zing.vn albums
- **zingmp3:song**: mp3.zing.vn songs

View File

@ -1,2 +1,6 @@
[wheel]
universal = True
[flake8]
exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,setup.py,build,.git
ignore = E501

View File

@ -4,7 +4,6 @@
from __future__ import print_function
import os.path
import pkg_resources
import warnings
import sys
@ -48,6 +47,7 @@ if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
else:
files_spec = [
('etc/bash_completion.d', ['youtube-dl.bash-completion']),
('etc/fish/completions', ['youtube-dl.fish']),
('share/doc/youtube_dl', ['README.txt']),
('share/man/man1', ['youtube-dl.1'])
]
@ -102,7 +102,9 @@ setup(
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.3"
"Programming Language :: Python :: 3.2",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
],
**params

View File

@ -1,3 +1,5 @@
from __future__ import unicode_literals
import errno
import io
import hashlib
@ -12,6 +14,7 @@ from youtube_dl import YoutubeDL
from youtube_dl.utils import (
compat_str,
preferredencoding,
write_string,
)
@ -40,10 +43,10 @@ def report_warning(message):
If stderr is a tty file the 'WARNING:' will be colored
'''
if sys.stderr.isatty() and os.name != 'nt':
_msg_header = u'\033[0;33mWARNING:\033[0m'
_msg_header = '\033[0;33mWARNING:\033[0m'
else:
_msg_header = u'WARNING:'
output = u'%s %s\n' % (_msg_header, message)
_msg_header = 'WARNING:'
output = '%s %s\n' % (_msg_header, message)
if 'b' in getattr(sys.stderr, 'mode', '') or sys.version_info[0] < 3:
output = output.encode(preferredencoding())
sys.stderr.write(output)
@ -54,7 +57,7 @@ class FakeYDL(YoutubeDL):
# Different instances of the downloader can't share the same dictionary
# some test set the "sublang" parameter, which would break the md5 checks.
params = get_params(override=override)
super(FakeYDL, self).__init__(params)
super(FakeYDL, self).__init__(params, auto_init=False)
self.result = []
def to_screen(self, s, skip_eol=None):
@ -69,32 +72,24 @@ class FakeYDL(YoutubeDL):
def expect_warning(self, regex):
# Silence an expected warning matching a regex
old_report_warning = self.report_warning
def report_warning(self, message):
if re.match(regex, message): return
if re.match(regex, message):
return
old_report_warning(message)
self.report_warning = types.MethodType(report_warning, self)
def gettestcases(include_onlymatching=False):
for ie in youtube_dl.extractor.gen_extractors():
t = getattr(ie, '_TEST', None)
if t:
assert not hasattr(ie, '_TESTS'), \
'%s has _TEST and _TESTS' % type(ie).__name__
tests = [t]
else:
tests = getattr(ie, '_TESTS', [])
for t in tests:
if not include_onlymatching and t.get('only_matching', False):
continue
t['name'] = type(ie).__name__[:-len('IE')]
yield t
for tc in ie.get_testcases(include_onlymatching):
yield tc
md5 = lambda s: hashlib.md5(s.encode('utf-8')).hexdigest()
def expect_info_dict(self, expected_dict, got_dict):
def expect_info_dict(self, got_dict, expected_dict):
for info_field, expected in expected_dict.items():
if isinstance(expected, compat_str) and expected.startswith('re:'):
got = got_dict.get(info_field)
@ -103,22 +98,36 @@ def expect_info_dict(self, expected_dict, got_dict):
self.assertTrue(
isinstance(got, compat_str),
u'Expected a %s object, but got %s for field %s' % (
'Expected a %s object, but got %s for field %s' % (
compat_str.__name__, type(got).__name__, info_field))
self.assertTrue(
match_rex.match(got),
u'field %s (value: %r) should match %r' % (info_field, got, match_str))
'field %s (value: %r) should match %r' % (info_field, got, match_str))
elif isinstance(expected, type):
got = got_dict.get(info_field)
self.assertTrue(isinstance(got, expected),
u'Expected type %r for field %s, but got value %r of type %r' % (expected, info_field, got, type(got)))
'Expected type %r for field %s, but got value %r of type %r' % (expected, info_field, got, type(got)))
else:
if isinstance(expected, compat_str) and expected.startswith('md5:'):
got = 'md5:' + md5(got_dict.get(info_field))
elif isinstance(expected, compat_str) and expected.startswith('mincount:'):
got = got_dict.get(info_field)
self.assertTrue(
isinstance(got, list),
'Expected field %s to be a list, but it is of type %s' % (
info_field, type(got).__name__))
expected_num = int(expected.partition(':')[2])
assertGreaterEqual(
self, len(got), expected_num,
'Expected %d items in field %s, but only got %d' % (
expected_num, info_field, len(got)
)
)
continue
else:
got = got_dict.get(info_field)
self.assertEqual(expected, got,
u'invalid value for field %s, expected %r, got %r' % (info_field, expected, got))
'invalid value for field %s, expected %r, got %r' % (info_field, expected, got))
# Check for the presence of mandatory fields
if got_dict.get('_type') != 'playlist':
@ -126,15 +135,30 @@ def expect_info_dict(self, expected_dict, got_dict):
self.assertTrue(got_dict.get(key), 'Missing mandatory field %s' % key)
# Check for mandatory fields that are automatically set by YoutubeDL
for key in ['webpage_url', 'extractor', 'extractor_key']:
self.assertTrue(got_dict.get(key), u'Missing field: %s' % key)
self.assertTrue(got_dict.get(key), 'Missing field: %s' % key)
# Are checkable fields missing from the test case definition?
test_info_dict = dict((key, value if not isinstance(value, compat_str) or len(value) < 250 else 'md5:' + md5(value))
for key, value in got_dict.items()
if value and key in ('title', 'description', 'uploader', 'upload_date', 'timestamp', 'uploader_id', 'location'))
if value and key in ('id', 'title', 'description', 'uploader', 'upload_date', 'timestamp', 'uploader_id', 'location'))
missing_keys = set(test_info_dict.keys()) - set(expected_dict.keys())
if missing_keys:
sys.stderr.write(u'\n"info_dict": ' + json.dumps(test_info_dict, ensure_ascii=False, indent=4) + u'\n')
def _repr(v):
if isinstance(v, compat_str):
return "'%s'" % v.replace('\\', '\\\\').replace("'", "\\'").replace('\n', '\\n')
else:
return repr(v)
info_dict_str = ''
if len(missing_keys) != len(expected_dict):
info_dict_str += ''.join(
' %s: %s,\n' % (_repr(k), _repr(v))
for k, v in test_info_dict.items() if k not in missing_keys)
info_dict_str += '\n'
info_dict_str += ''.join(
' %s: %s,\n' % (_repr(k), _repr(test_info_dict[k]))
for k in missing_keys)
write_string(
'\n\'info_dict\': {\n' + info_dict_str + '}\n', out=sys.stderr)
self.assertFalse(
missing_keys,
'Missing keys in test definition: %s' % (
@ -147,7 +171,9 @@ def assertRegexpMatches(self, text, regexp, msg=None):
else:
m = re.match(regexp, text)
if not m:
note = 'Regexp didn\'t match: %r not found in %r' % (regexp, text)
note = 'Regexp didn\'t match: %r not found' % (regexp)
if len(text) < 1000:
note += ' in %r' % text
if msg is None:
msg = note
else:
@ -160,3 +186,13 @@ def assertGreaterEqual(self, got, expected, msg=None):
if msg is None:
msg = '%r not greater than or equal to %r' % (got, expected)
self.assertTrue(got >= expected, msg)
def expect_warnings(ydl, warnings_re):
real_warning = ydl.report_warning
def _report_warning(w):
if not any(re.search(w_re, w) for w_re in warnings_re):
real_warning(w)
ydl.report_warning = _report_warning

View File

@ -0,0 +1,18 @@
// input: []
// output: 4
package {
public class ConstArrayAccess {
private static const x:int = 2;
private static const ar:Array = ["42", "3411"];
public static function main():int{
var c:ConstArrayAccess = new ConstArrayAccess();
return c.f();
}
public function f(): int {
return ar[1].length;
}
}
}

View File

@ -0,0 +1,12 @@
// input: []
// output: 2
package {
public class ConstantInt {
private static const x:int = 2;
public static function main():int{
return x;
}
}
}

10
test/swftests/DictCall.as Normal file
View File

@ -0,0 +1,10 @@
// input: [{"x": 1, "y": 2}]
// output: 3
package {
public class DictCall {
public static function main(d:Object):int{
return d.x + d.y;
}
}
}

View File

@ -0,0 +1,10 @@
// input: []
// output: false
package {
public class EqualsOperator {
public static function main():Boolean{
return 1 == 2;
}
}
}

View File

@ -0,0 +1,22 @@
// input: [1]
// output: 2
package {
public class MemberAssignment {
public var v:int;
public function g():int {
return this.v;
}
public function f(a:int):int{
this.v = a;
return this.v + this.g();
}
public static function main(a:int): int {
var v:MemberAssignment = new MemberAssignment();
return v.f(a);
}
}
}

View File

@ -0,0 +1,24 @@
// input: []
// output: 123
package {
public class NeOperator {
public static function main(): int {
var res:int = 0;
if (1 != 2) {
res += 3;
} else {
res += 4;
}
if (2 != 2) {
res += 10;
} else {
res += 20;
}
if (9 == 9) {
res += 100;
}
return res;
}
}
}

View File

@ -0,0 +1,22 @@
// input: []
// output: 9
package {
public class PrivateVoidCall {
public static function main():int{
var f:OtherClass = new OtherClass();
f.func();
return 9;
}
}
}
class OtherClass {
private function pf():void {
;
}
public function func():void {
this.pf();
}
}

View File

@ -0,0 +1,11 @@
// input: []
// output: 3
package {
public class StringBasics {
public static function main():int{
var s:String = "abc";
return s.length;
}
}
}

View File

@ -0,0 +1,11 @@
// input: []
// output: 9897
package {
public class StringCharCodeAt {
public static function main():int{
var s:String = "abc";
return s.charCodeAt(1) * 100 + s.charCodeAt();
}
}
}

View File

@ -0,0 +1,11 @@
// input: []
// output: 2
package {
public class StringConversion {
public static function main():int{
var s:String = String(99);
return s.length;
}
}
}

View File

@ -40,5 +40,23 @@ class TestInfoExtractor(unittest.TestCase):
self.assertEqual(ie._og_search_description(html), 'Some video\'s description ')
self.assertEqual(ie._og_search_thumbnail(html), 'http://domain.com/pic.jpg?key1=val1&key2=val2')
def test_html_search_meta(self):
ie = self.ie
html = '''
<meta name="a" content="1" />
<meta name='b' content='2'>
<meta name="c" content='3'>
<meta name=d content='4'>
<meta property="e" content='5' >
<meta content="6" name="f">
'''
self.assertEqual(ie._html_search_meta('a', html), '1')
self.assertEqual(ie._html_search_meta('b', html), '2')
self.assertEqual(ie._html_search_meta('c', html), '3')
self.assertEqual(ie._html_search_meta('d', html), '4')
self.assertEqual(ie._html_search_meta('e', html), '5')
self.assertEqual(ie._html_search_meta('f', html), '6')
if __name__ == '__main__':
unittest.main()

View File

@ -8,6 +8,8 @@ import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import copy
from test.helper import FakeYDL, assertRegexpMatches
from youtube_dl import YoutubeDL
from youtube_dl.extractor import YoutubeIE
@ -192,6 +194,37 @@ class TestFormatSelection(unittest.TestCase):
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'vid-high')
def test_format_selection_audio_exts(self):
formats = [
{'format_id': 'mp3-64', 'ext': 'mp3', 'abr': 64, 'url': 'http://_', 'vcodec': 'none'},
{'format_id': 'ogg-64', 'ext': 'ogg', 'abr': 64, 'url': 'http://_', 'vcodec': 'none'},
{'format_id': 'aac-64', 'ext': 'aac', 'abr': 64, 'url': 'http://_', 'vcodec': 'none'},
{'format_id': 'mp3-32', 'ext': 'mp3', 'abr': 32, 'url': 'http://_', 'vcodec': 'none'},
{'format_id': 'aac-32', 'ext': 'aac', 'abr': 32, 'url': 'http://_', 'vcodec': 'none'},
]
info_dict = _make_result(formats)
ydl = YDL({'format': 'best'})
ie = YoutubeIE(ydl)
ie._sort_formats(info_dict['formats'])
ydl.process_ie_result(copy.deepcopy(info_dict))
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'aac-64')
ydl = YDL({'format': 'mp3'})
ie = YoutubeIE(ydl)
ie._sort_formats(info_dict['formats'])
ydl.process_ie_result(copy.deepcopy(info_dict))
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'mp3-64')
ydl = YDL({'prefer_free_formats': True})
ie = YoutubeIE(ydl)
ie._sort_formats(info_dict['formats'])
ydl.process_ie_result(copy.deepcopy(info_dict))
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'ogg-64')
def test_format_selection_video(self):
formats = [
{'format_id': 'dash-video-low', 'ext': 'mp4', 'preference': 1, 'acodec': 'none', 'url': '_'},
@ -218,7 +251,7 @@ class TestFormatSelection(unittest.TestCase):
# 3D
'85', '84', '102', '83', '101', '82', '100',
# Dash video
'138', '137', '248', '136', '247', '135', '246',
'137', '248', '136', '247', '135', '246',
'245', '244', '134', '243', '133', '242', '160',
# Dash audio
'141', '172', '140', '171', '139',
@ -248,6 +281,61 @@ class TestFormatSelection(unittest.TestCase):
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], f1id)
def test_format_filtering(self):
formats = [
{'format_id': 'A', 'filesize': 500, 'width': 1000},
{'format_id': 'B', 'filesize': 1000, 'width': 500},
{'format_id': 'C', 'filesize': 1000, 'width': 400},
{'format_id': 'D', 'filesize': 2000, 'width': 600},
{'format_id': 'E', 'filesize': 3000},
{'format_id': 'F'},
{'format_id': 'G', 'filesize': 1000000},
]
for f in formats:
f['url'] = 'http://_/'
f['ext'] = 'unknown'
info_dict = _make_result(formats)
ydl = YDL({'format': 'best[filesize<3000]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'D')
ydl = YDL({'format': 'best[filesize<=3000]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'E')
ydl = YDL({'format': 'best[filesize <= ? 3000]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'F')
ydl = YDL({'format': 'best [filesize = 1000] [width>450]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'B')
ydl = YDL({'format': 'best [filesize = 1000] [width!=450]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'C')
ydl = YDL({'format': '[filesize>?1]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'G')
ydl = YDL({'format': '[filesize<1M]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'E')
ydl = YDL({'format': '[filesize<1MiB]'})
ydl.process_ie_result(info_dict)
downloaded = ydl.downloaded_info_dicts[0]
self.assertEqual(downloaded['format_id'], 'G')
def test_add_extra_info(self):
test_dict = {
'extractor': 'Foo',
@ -266,6 +354,7 @@ class TestFormatSelection(unittest.TestCase):
'ext': 'mp4',
'width': None,
}
def fname(templ):
ydl = YoutubeDL({'outtmpl': templ})
return ydl.prepare_filename(info)

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
@ -19,7 +20,7 @@ def _download_restricted(url, filename, age):
'age_limit': age,
'skip_download': True,
'writeinfojson': True,
"outtmpl": "%(id)s.%(ext)s",
'outtmpl': '%(id)s.%(ext)s',
}
ydl = YoutubeDL(params)
ydl.add_default_info_extractors()
@ -44,11 +45,6 @@ class TestAgeRestriction(unittest.TestCase):
'http://www.youporn.com/watch/505835/sex-ed-is-it-safe-to-masturbate-daily/',
'505835.mp4', 2, old_age=25)
def test_pornotube(self):
self._assert_restricted(
'http://pornotube.com/c/173/m/1689755/Marilyn-Monroe-Bathing',
'1689755.flv', 13)
if __name__ == '__main__':
unittest.main()

View File

@ -14,7 +14,6 @@ from test.helper import gettestcases
from youtube_dl.extractor import (
FacebookIE,
gen_extractors,
JustinTVIE,
YoutubeIE,
)
@ -72,22 +71,6 @@ class TestAllURLsMatching(unittest.TestCase):
self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
def test_justin_tv_channelid_matching(self):
self.assertTrue(JustinTVIE.suitable('justin.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('twitch.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('www.justin.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('www.twitch.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('http://www.justin.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('http://www.twitch.tv/vanillatv'))
self.assertTrue(JustinTVIE.suitable('http://www.justin.tv/vanillatv/'))
self.assertTrue(JustinTVIE.suitable('http://www.twitch.tv/vanillatv/'))
def test_justintv_videoid_matching(self):
self.assertTrue(JustinTVIE.suitable('http://www.twitch.tv/vanillatv/b/328087483'))
def test_justin_tv_chapterid_matching(self):
self.assertTrue(JustinTVIE.suitable('http://www.twitch.tv/tsm_theoddone/c/2349361'))
def test_youtube_extract(self):
assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)
assertExtractId('http://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
@ -119,8 +102,6 @@ class TestAllURLsMatching(unittest.TestCase):
self.assertMatch(':ythistory', ['youtube:history'])
self.assertMatch(':thedailyshow', ['ComedyCentralShows'])
self.assertMatch(':tds', ['ComedyCentralShows'])
self.assertMatch(':colbertreport', ['ComedyCentralShows'])
self.assertMatch(':cr', ['ComedyCentralShows'])
def test_vimeo_matching(self):
self.assertMatch('http://vimeo.com/channels/tributes', ['vimeo:channel'])

46
test/test_compat.py Normal file
View File

@ -0,0 +1,46 @@
#!/usr/bin/env python
# coding: utf-8
from __future__ import unicode_literals
# Allow direct execution
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from youtube_dl.utils import get_filesystem_encoding
from youtube_dl.compat import (
compat_getenv,
compat_expanduser,
)
class TestCompat(unittest.TestCase):
def test_compat_getenv(self):
test_str = 'тест'
os.environ['YOUTUBE-DL-TEST'] = (
test_str if sys.version_info >= (3, 0)
else test_str.encode(get_filesystem_encoding()))
self.assertEqual(compat_getenv('YOUTUBE-DL-TEST'), test_str)
def test_compat_expanduser(self):
old_home = os.environ.get('HOME')
test_str = 'C:\Documents and Settings\тест\Application Data'
os.environ['HOME'] = (
test_str if sys.version_info >= (3, 0)
else test_str.encode(get_filesystem_encoding()))
self.assertEqual(compat_expanduser('~'), test_str)
os.environ['HOME'] = old_home
def test_all_present(self):
import youtube_dl.compat
all_names = youtube_dl.compat.__all__
present_names = set(filter(
lambda c: '_' in c and not c.startswith('_'),
dir(youtube_dl.compat))) - set(['unicode_literals'])
self.assertEqual(all_names, sorted(present_names))
if __name__ == '__main__':
unittest.main()

View File

@ -1,5 +1,7 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
import sys
@ -8,6 +10,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from test.helper import (
assertGreaterEqual,
expect_warnings,
get_params,
gettestcases,
expect_info_dict,
@ -22,10 +25,12 @@ import json
import socket
import youtube_dl.YoutubeDL
from youtube_dl.utils import (
from youtube_dl.compat import (
compat_http_client,
compat_urllib_error,
compat_HTTPError,
)
from youtube_dl.utils import (
DownloadError,
ExtractorError,
format_bytes,
@ -35,18 +40,22 @@ from youtube_dl.extractor import get_info_extractor
RETRIES = 3
class YoutubeDL(youtube_dl.YoutubeDL):
def __init__(self, *args, **kwargs):
self.to_stderr = self.to_screen
self.processed_info_dicts = []
super(YoutubeDL, self).__init__(*args, **kwargs)
def report_warning(self, message):
# Don't accept warnings during tests
raise ExtractorError(message)
def process_info(self, info_dict):
self.processed_info_dicts.append(info_dict)
return super(YoutubeDL, self).process_info(info_dict)
def _file_md5(fn):
with open(fn, 'rb') as f:
return hashlib.md5(f.read()).hexdigest()
@ -56,10 +65,13 @@ defs = gettestcases()
class TestDownload(unittest.TestCase):
maxDiff = None
def setUp(self):
self.defs = defs
### Dynamically generate tests
# Dynamically generate tests
def generator(test_case):
def test_template(self):
@ -77,7 +89,7 @@ def generator(test_case):
for tc in test_cases:
info_dict = tc.get('info_dict', {})
if not tc.get('file') and not (info_dict.get('id') and info_dict.get('ext')):
if not (info_dict.get('id') and info_dict.get('ext')):
raise Exception('Test definition incorrect. The output file cannot be known. Are both \'id\' and \'ext\' keys present?')
if 'skip' in test_case:
@ -85,7 +97,7 @@ def generator(test_case):
return
for other_ie in other_ies:
if not other_ie.working():
print_skipping(u'test depends on %sIE, marked as not WORKING' % other_ie.ie_key())
print_skipping('test depends on %sIE, marked as not WORKING' % other_ie.ie_key())
return
params = get_params(test_case.get('params', {}))
@ -93,18 +105,21 @@ def generator(test_case):
params.setdefault('extract_flat', True)
params.setdefault('skip_download', True)
ydl = YoutubeDL(params)
ydl = YoutubeDL(params, auto_init=False)
ydl.add_default_info_extractors()
finished_hook_called = set()
def _hook(status):
if status['status'] == 'finished':
finished_hook_called.add(status['filename'])
ydl.add_progress_hook(_hook)
expect_warnings(ydl, test_case.get('expected_warnings', []))
def get_tc_filename(tc):
return tc.get('file') or ydl.prepare_filename(tc.get('info_dict', {}))
return ydl.prepare_filename(tc.get('info_dict', {}))
res_dict = None
def try_rm_tcs_files(tcs=None):
if tcs is None:
tcs = test_cases
@ -128,7 +143,7 @@ def generator(test_case):
raise
if try_num == RETRIES:
report_warning(u'Failed due to network errors, skipping...')
report_warning('Failed due to network errors, skipping...')
return
print('Retrying: {0} failed tries\n\n##########\n\n'.format(try_num))
@ -139,7 +154,9 @@ def generator(test_case):
if is_playlist:
self.assertEqual(res_dict['_type'], 'playlist')
expect_info_dict(self, test_case.get('info_dict', {}), res_dict)
self.assertTrue('entries' in res_dict)
expect_info_dict(self, res_dict, test_case.get('info_dict', {}))
if 'playlist_mincount' in test_case:
assertGreaterEqual(
self,
@ -181,14 +198,16 @@ def generator(test_case):
md5_for_file = _file_md5(tc_filename)
self.assertEqual(md5_for_file, tc['md5'])
info_json_fn = os.path.splitext(tc_filename)[0] + '.info.json'
self.assertTrue(os.path.exists(info_json_fn))
self.assertTrue(
os.path.exists(info_json_fn),
'Missing info file %s' % info_json_fn)
with io.open(info_json_fn, encoding='utf-8') as infof:
info_dict = json.load(infof)
expect_info_dict(self, tc.get('info_dict', {}), info_dict)
expect_info_dict(self, info_dict, tc.get('info_dict', {}))
finally:
try_rm_tcs_files()
if is_playlist and res_dict is not None:
if is_playlist and res_dict is not None and res_dict.get('entries'):
# Remove all other files that may have been extracted if the
# extractor returns full results even with extract_flat
res_tcs = [{'info_dict': e} for e in res_dict['entries']]
@ -196,15 +215,15 @@ def generator(test_case):
return test_template
### And add them to TestDownload
# And add them to TestDownload
for n, test_case in enumerate(defs):
test_method = generator(test_case)
tname = 'test_' + str(test_case['name'])
i = 1
while hasattr(TestDownload, tname):
tname = 'test_' + str(test_case['name']) + '_' + str(i)
tname = 'test_%s_%d' % (test_case['name'], i)
i += 1
test_method.__name__ = tname
test_method.__name__ = str(tname)
setattr(TestDownload, test_method.__name__, test_method)
del test_method

View File

@ -1,3 +1,6 @@
#!/usr/bin/env python
from __future__ import unicode_literals
import unittest
import sys
@ -6,11 +9,13 @@ import subprocess
rootDir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
try:
_DEV_NULL = subprocess.DEVNULL
except AttributeError:
_DEV_NULL = open(os.devnull, 'wb')
class TestExecution(unittest.TestCase):
def test_import(self):
subprocess.check_call([sys.executable, '-c', 'import youtube_dl'], cwd=rootDir)

72
test/test_http.py Normal file
View File

@ -0,0 +1,72 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from youtube_dl import YoutubeDL
from youtube_dl.compat import compat_http_server
import ssl
import threading
TEST_DIR = os.path.dirname(os.path.abspath(__file__))
class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
def log_message(self, format, *args):
pass
def do_GET(self):
if self.path == '/video.html':
self.send_response(200)
self.send_header('Content-Type', 'text/html; charset=utf-8')
self.end_headers()
self.wfile.write(b'<html><video src="/vid.mp4" /></html>')
elif self.path == '/vid.mp4':
self.send_response(200)
self.send_header('Content-Type', 'video/mp4')
self.end_headers()
self.wfile.write(b'\x00\x00\x00\x00\x20\x66\x74[video]')
else:
assert False
class FakeLogger(object):
def debug(self, msg):
pass
def warning(self, msg):
pass
def error(self, msg):
pass
class TestHTTP(unittest.TestCase):
def setUp(self):
certfn = os.path.join(TEST_DIR, 'testcert.pem')
self.httpd = compat_http_server.HTTPServer(
('localhost', 0), HTTPTestRequestHandler)
self.httpd.socket = ssl.wrap_socket(
self.httpd.socket, certfile=certfn, server_side=True)
self.port = self.httpd.socket.getsockname()[1]
self.server_thread = threading.Thread(target=self.httpd.serve_forever)
self.server_thread.daemon = True
self.server_thread.start()
def test_nocheckcertificate(self):
if sys.version_info >= (2, 7, 9): # No certificate checking anyways
ydl = YoutubeDL({'logger': FakeLogger()})
self.assertRaises(
Exception,
ydl.extract_info, 'https://localhost:%d/video.html' % self.port)
ydl = YoutubeDL({'logger': FakeLogger(), 'nocheckcertificate': True})
r = ydl.extract_info('https://localhost:%d/video.html' % self.port)
self.assertEqual(r['url'], 'https://localhost:%d/vid.mp4' % self.port)
if __name__ == '__main__':
unittest.main()

95
test/test_jsinterp.py Normal file
View File

@ -0,0 +1,95 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from youtube_dl.jsinterp import JSInterpreter
class TestJSInterpreter(unittest.TestCase):
def test_basic(self):
jsi = JSInterpreter('function x(){;}')
self.assertEqual(jsi.call_function('x'), None)
jsi = JSInterpreter('function x3(){return 42;}')
self.assertEqual(jsi.call_function('x3'), 42)
def test_calc(self):
jsi = JSInterpreter('function x4(a){return 2*a+1;}')
self.assertEqual(jsi.call_function('x4', 3), 7)
def test_empty_return(self):
jsi = JSInterpreter('function f(){return; y()}')
self.assertEqual(jsi.call_function('f'), None)
def test_morespace(self):
jsi = JSInterpreter('function x (a) { return 2 * a + 1 ; }')
self.assertEqual(jsi.call_function('x', 3), 7)
jsi = JSInterpreter('function f () { x = 2 ; return x; }')
self.assertEqual(jsi.call_function('f'), 2)
def test_strange_chars(self):
jsi = JSInterpreter('function $_xY1 ($_axY1) { var $_axY2 = $_axY1 + 1; return $_axY2; }')
self.assertEqual(jsi.call_function('$_xY1', 20), 21)
def test_operators(self):
jsi = JSInterpreter('function f(){return 1 << 5;}')
self.assertEqual(jsi.call_function('f'), 32)
jsi = JSInterpreter('function f(){return 19 & 21;}')
self.assertEqual(jsi.call_function('f'), 17)
jsi = JSInterpreter('function f(){return 11 >> 2;}')
self.assertEqual(jsi.call_function('f'), 2)
def test_array_access(self):
jsi = JSInterpreter('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2] = 7; return x;}')
self.assertEqual(jsi.call_function('f'), [5, 2, 7])
def test_parens(self):
jsi = JSInterpreter('function f(){return (1) + (2) * ((( (( (((((3)))))) )) ));}')
self.assertEqual(jsi.call_function('f'), 7)
jsi = JSInterpreter('function f(){return (1 + 2) * 3;}')
self.assertEqual(jsi.call_function('f'), 9)
def test_assignments(self):
jsi = JSInterpreter('function f(){var x = 20; x = 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), 31)
jsi = JSInterpreter('function f(){var x = 20; x += 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), 51)
jsi = JSInterpreter('function f(){var x = 20; x -= 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), -11)
def test_comments(self):
jsi = JSInterpreter('''
function x() {
var x = /* 1 + */ 2;
var y = /* 30
* 40 */ 50;
return x + y;
}
''')
self.assertEqual(jsi.call_function('x'), 52)
def test_precedence(self):
jsi = JSInterpreter('''
function x() {
var a = [10, 20, 30, 40, 50];
var b = 6;
a[0]=a[b%a.length];
return a;
}''')
self.assertEqual(jsi.call_function('x'), [20, 20, 30, 40, 50])
if __name__ == '__main__':
unittest.main()

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
@ -15,12 +16,15 @@ from youtube_dl.extractor import (
DailymotionIE,
TEDIE,
VimeoIE,
WallaIE,
CeskaTelevizeIE,
)
class BaseTestSubtitles(unittest.TestCase):
url = None
IE = None
def setUp(self):
self.DL = FakeYDL()
self.ie = self.IE(self.DL)
@ -73,7 +77,7 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
self.assertEqual(md5(subtitles['en']), '3cb210999d3e021bd6c7f0ea751eab06')
def test_youtube_list_subtitles(self):
self.DL.expect_warning(u'Video doesn\'t have automatic captions')
self.DL.expect_warning('Video doesn\'t have automatic captions')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
@ -85,8 +89,16 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
subtitles = self.getSubtitles()
self.assertTrue(subtitles['it'] is not None)
def test_youtube_translated_subtitles(self):
# This video has a subtitles track, which can be translated
self.url = 'Ky9eprVWzlI'
self.DL.params['writeautomaticsub'] = True
self.DL.params['subtitleslangs'] = ['it']
subtitles = self.getSubtitles()
self.assertTrue(subtitles['it'] is not None)
def test_youtube_nosubtitles(self):
self.DL.expect_warning(u'video doesn\'t have subtitles')
self.DL.expect_warning('video doesn\'t have subtitles')
self.url = 'n5BB19UTcdA'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
@ -100,7 +112,7 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
self.DL.params['subtitleslangs'] = langs
subtitles = self.getSubtitles()
for lang in langs:
self.assertTrue(subtitles.get(lang) is not None, u'Subtitles for \'%s\' not extracted' % lang)
self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
class TestDailymotionSubtitles(BaseTestSubtitles):
@ -129,20 +141,20 @@ class TestDailymotionSubtitles(BaseTestSubtitles):
self.assertEqual(len(subtitles.keys()), 5)
def test_list_subtitles(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_automatic_captions(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writeautomaticsub'] = True
self.DL.params['subtitleslang'] = ['en']
subtitles = self.getSubtitles()
self.assertTrue(len(subtitles.keys()) == 0)
def test_nosubtitles(self):
self.DL.expect_warning(u'video doesn\'t have subtitles')
self.DL.expect_warning('video doesn\'t have subtitles')
self.url = 'http://www.dailymotion.com/video/x12u166_le-zapping-tele-star-du-08-aout-2013_tv'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
@ -155,7 +167,7 @@ class TestDailymotionSubtitles(BaseTestSubtitles):
self.DL.params['subtitleslangs'] = langs
subtitles = self.getSubtitles()
for lang in langs:
self.assertTrue(subtitles.get(lang) is not None, u'Subtitles for \'%s\' not extracted' % lang)
self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
class TestTedSubtitles(BaseTestSubtitles):
@ -184,13 +196,13 @@ class TestTedSubtitles(BaseTestSubtitles):
self.assertTrue(len(subtitles.keys()) >= 28)
def test_list_subtitles(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_automatic_captions(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writeautomaticsub'] = True
self.DL.params['subtitleslang'] = ['en']
subtitles = self.getSubtitles()
@ -202,7 +214,7 @@ class TestTedSubtitles(BaseTestSubtitles):
self.DL.params['subtitleslangs'] = langs
subtitles = self.getSubtitles()
for lang in langs:
self.assertTrue(subtitles.get(lang) is not None, u'Subtitles for \'%s\' not extracted' % lang)
self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
class TestBlipTVSubtitles(BaseTestSubtitles):
@ -210,13 +222,13 @@ class TestBlipTVSubtitles(BaseTestSubtitles):
IE = BlipTVIE
def test_list_subtitles(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_allsubtitles(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
@ -235,7 +247,7 @@ class TestVimeoSubtitles(BaseTestSubtitles):
def test_subtitles(self):
self.DL.params['writesubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(md5(subtitles['en']), '8062383cf4dec168fc40a088aa6d5888')
self.assertEqual(md5(subtitles['en']), '26399116d23ae3cf2c087cea94bc43b4')
def test_subtitles_lang(self):
self.DL.params['writesubtitles'] = True
@ -250,20 +262,20 @@ class TestVimeoSubtitles(BaseTestSubtitles):
self.assertEqual(set(subtitles.keys()), set(['de', 'en', 'es', 'fr']))
def test_list_subtitles(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_automatic_captions(self):
self.DL.expect_warning(u'Automatic Captions not supported by this server')
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writeautomaticsub'] = True
self.DL.params['subtitleslang'] = ['en']
subtitles = self.getSubtitles()
self.assertTrue(len(subtitles.keys()) == 0)
def test_nosubtitles(self):
self.DL.expect_warning(u'video doesn\'t have subtitles')
self.DL.expect_warning('video doesn\'t have subtitles')
self.url = 'http://vimeo.com/56015672'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
@ -276,7 +288,61 @@ class TestVimeoSubtitles(BaseTestSubtitles):
self.DL.params['subtitleslangs'] = langs
subtitles = self.getSubtitles()
for lang in langs:
self.assertTrue(subtitles.get(lang) is not None, u'Subtitles for \'%s\' not extracted' % lang)
self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
class TestWallaSubtitles(BaseTestSubtitles):
url = 'http://vod.walla.co.il/movie/2705958/the-yes-men'
IE = WallaIE
def test_list_subtitles(self):
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_allsubtitles(self):
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(set(subtitles.keys()), set(['heb']))
self.assertEqual(md5(subtitles['heb']), 'e758c5d7cb982f6bef14f377ec7a3920')
def test_nosubtitles(self):
self.DL.expect_warning('video doesn\'t have subtitles')
self.url = 'http://vod.walla.co.il/movie/2642630/one-direction-all-for-one'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(len(subtitles), 0)
class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
url = 'http://www.ceskatelevize.cz/ivysilani/10600540290-u6-uzasny-svet-techniky'
IE = CeskaTelevizeIE
def test_list_subtitles(self):
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['listsubtitles'] = True
info_dict = self.getInfoDict()
self.assertEqual(info_dict, None)
def test_allsubtitles(self):
self.DL.expect_warning('Automatic Captions not supported by this server')
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(set(subtitles.keys()), set(['cs']))
self.assertEqual(md5(subtitles['cs']), '9bf52d9549533c32c427e264bf0847d4')
def test_nosubtitles(self):
self.DL.expect_warning('video doesn\'t have subtitles')
self.url = 'http://www.ceskatelevize.cz/ivysilani/ivysilani/10441294653-hyde-park-civilizace/214411058091220'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(len(subtitles), 0)
if __name__ == '__main__':

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
@ -37,7 +38,9 @@ def _make_testfunc(testfile):
or os.path.getmtime(swf_file) < os.path.getmtime(as_file)):
# Recompile
try:
subprocess.check_call(['mxmlc', '-output', swf_file, as_file])
subprocess.check_call([
'mxmlc', '-output', swf_file,
'-static-link-runtime-shared-libraries', as_file])
except OSError as ose:
if ose.errno == errno.ENOENT:
print('mxmlc not found! Skipping test.')

View File

@ -1,22 +1,28 @@
from __future__ import unicode_literals
import io
# Allow direct execution
import os
import re
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import io
import re
rootDir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
IGNORED_FILES = [
'setup.py', # http://bugs.python.org/issue13943
'conf.py',
'buildserver.py',
]
from test.helper import assertRegexpMatches
class TestUnicodeLiterals(unittest.TestCase):
def test_all_files(self):
print('Skipping this test (not yet fully implemented)')
return
for dirpath, _, filenames in os.walk(rootDir):
for basename in filenames:
if not basename.endswith('.py'):
@ -30,10 +36,11 @@ class TestUnicodeLiterals(unittest.TestCase):
if "'" not in code and '"' not in code:
continue
imps = 'from __future__ import unicode_literals'
self.assertTrue(
imps in code,
' %s missing in %s' % (imps, fn))
assertRegexpMatches(
self,
code,
r'(?:(?:#.*?|\s*)\n)*from __future__ import (?:[a-z_]+,\s*)*unicode_literals',
'unicode_literals import missing in %s' % fn)
m = re.search(r'(?<=\s)u[\'"](?!\)|,|$)', code)
if m is not None:

View File

@ -16,30 +16,43 @@ import json
import xml.etree.ElementTree
from youtube_dl.utils import (
age_restricted,
args_to_str,
clean_html,
DateRange,
detect_exe_version,
encodeFilename,
escape_rfc3986,
escape_url,
find_xpath_attr,
fix_xml_ampersands,
get_meta_content,
InAdvancePagedList,
intlist_to_bytes,
is_html,
js_to_json,
limit_length,
OnDemandPagedList,
orderedSet,
PagedList,
parse_duration,
parse_filesize,
parse_iso8601,
read_batch_urls,
sanitize_filename,
shell_quote,
smuggle_url,
str_to_int,
strip_jsonp,
struct_unpack,
timeconvert,
unescapeHTML,
unified_strdate,
unsmuggle_url,
uppercase_escape,
url_basename,
urlencode_postdata,
version_tuple,
xpath_with_ns,
parse_iso8601,
strip_jsonp,
uppercase_escape,
render_table,
)
@ -68,6 +81,10 @@ class TestUtil(unittest.TestCase):
tests = '\u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430'
self.assertEqual(sanitize_filename(tests), tests)
self.assertEqual(
sanitize_filename('New World record at 0:12:34'),
'New World record at 0_12_34')
forbidden = '"\0\\/'
for fc in forbidden:
for fbc in forbidden:
@ -133,7 +150,12 @@ class TestUtil(unittest.TestCase):
self.assertEqual(unified_strdate('8/7/2009'), '20090708')
self.assertEqual(unified_strdate('Dec 14, 2012'), '20121214')
self.assertEqual(unified_strdate('2012/10/11 01:56:38 +0000'), '20121011')
self.assertEqual(unified_strdate('1968 12 10'), '19681210')
self.assertEqual(unified_strdate('1968-12-10'), '19681210')
self.assertEqual(unified_strdate('28/01/2014 21:00:00 +0100'), '20140128')
self.assertEqual(
unified_strdate('11/26/2014 11:30:00 AM PST', day_first=False),
'20141126')
def test_find_xpath_attr(self):
testxml = '''<root>
@ -148,17 +170,6 @@ class TestUtil(unittest.TestCase):
self.assertEqual(find_xpath_attr(doc, './/node', 'x', 'a'), doc[1])
self.assertEqual(find_xpath_attr(doc, './/node', 'y', 'c'), doc[2])
def test_meta_parser(self):
testhtml = '''
<head>
<meta name="description" content="foo &amp; bar">
<meta content='Plato' name='author'/>
</head>
'''
get_meta = lambda name: get_meta_content(name, testhtml)
self.assertEqual(get_meta('description'), 'foo & bar')
self.assertEqual(get_meta('author'), 'Plato')
def test_xpath_with_ns(self):
testxml = '''<root xmlns:media="http://example.com/">
<media:song>
@ -173,7 +184,7 @@ class TestUtil(unittest.TestCase):
self.assertEqual(find('media:song/url').text, 'http://server.com/download.mp3')
def test_smuggle_url(self):
data = {u"ö": u"ö", u"abc": [3]}
data = {"ö": "ö", "abc": [3]}
url = 'https://foo.bar/baz?x=y#a'
smug_url = smuggle_url(url, data)
unsmug_url, unsmug_data = unsmuggle_url(smug_url)
@ -204,6 +215,8 @@ class TestUtil(unittest.TestCase):
def test_parse_duration(self):
self.assertEqual(parse_duration(None), None)
self.assertEqual(parse_duration(False), None)
self.assertEqual(parse_duration('invalid'), None)
self.assertEqual(parse_duration('1'), 1)
self.assertEqual(parse_duration('1337:12'), 80232)
self.assertEqual(parse_duration('9:12:43'), 33163)
@ -221,6 +234,12 @@ class TestUtil(unittest.TestCase):
self.assertEqual(parse_duration('0m0s'), 0)
self.assertEqual(parse_duration('0s'), 0)
self.assertEqual(parse_duration('01:02:03.05'), 3723.05)
self.assertEqual(parse_duration('T30M38S'), 1838)
self.assertEqual(parse_duration('5 s'), 5)
self.assertEqual(parse_duration('3 min'), 180)
self.assertEqual(parse_duration('2.5 hours'), 9000)
self.assertEqual(parse_duration('02:03:04'), 7384)
self.assertEqual(parse_duration('01:02:03:04'), 93784)
def test_fix_xml_ampersands(self):
self.assertEqual(
@ -243,10 +262,14 @@ class TestUtil(unittest.TestCase):
for i in range(firstid, upto):
yield i
pl = PagedList(get_page, pagesize)
pl = OnDemandPagedList(get_page, pagesize)
got = pl.getslice(*sliceargs)
self.assertEqual(got, expected)
iapl = InAdvancePagedList(get_page, size // pagesize + 1, pagesize)
got = iapl.getslice(*sliceargs)
self.assertEqual(got, expected)
testPL(5, 2, (), [0, 1, 2, 3, 4])
testPL(5, 2, (1,), [1, 2, 3, 4])
testPL(5, 2, (2,), [2, 3, 4])
@ -276,15 +299,163 @@ class TestUtil(unittest.TestCase):
self.assertEqual(parse_iso8601('2014-03-23T23:04:26+0100'), 1395612266)
self.assertEqual(parse_iso8601('2014-03-23T22:04:26+0000'), 1395612266)
self.assertEqual(parse_iso8601('2014-03-23T22:04:26Z'), 1395612266)
self.assertEqual(parse_iso8601('2014-03-23T22:04:26.1234Z'), 1395612266)
def test_strip_jsonp(self):
stripped = strip_jsonp('cb ([ {"id":"532cb",\n\n\n"x":\n3}\n]\n);')
d = json.loads(stripped)
self.assertEqual(d, [{"id": "532cb", "x": 3}])
stripped = strip_jsonp('parseMetadata({"STATUS":"OK"})\n\n\n//epc')
d = json.loads(stripped)
self.assertEqual(d, {'STATUS': 'OK'})
def test_uppercase_escape(self):
self.assertEqual(uppercase_escape(''), '')
self.assertEqual(uppercase_escape('\\U0001d550'), '𝕐')
def test_limit_length(self):
self.assertEqual(limit_length(None, 12), None)
self.assertEqual(limit_length('foo', 12), 'foo')
self.assertTrue(
limit_length('foo bar baz asd', 12).startswith('foo bar'))
self.assertTrue('...' in limit_length('foo bar baz asd', 12))
def test_escape_rfc3986(self):
reserved = "!*'();:@&=+$,/?#[]"
unreserved = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~'
self.assertEqual(escape_rfc3986(reserved), reserved)
self.assertEqual(escape_rfc3986(unreserved), unreserved)
self.assertEqual(escape_rfc3986('тест'), '%D1%82%D0%B5%D1%81%D1%82')
self.assertEqual(escape_rfc3986('%D1%82%D0%B5%D1%81%D1%82'), '%D1%82%D0%B5%D1%81%D1%82')
self.assertEqual(escape_rfc3986('foo bar'), 'foo%20bar')
self.assertEqual(escape_rfc3986('foo%20bar'), 'foo%20bar')
def test_escape_url(self):
self.assertEqual(
escape_url('http://wowza.imust.org/srv/vod/telemb/new/UPLOAD/UPLOAD/20224_IncendieHavré_FD.mp4'),
'http://wowza.imust.org/srv/vod/telemb/new/UPLOAD/UPLOAD/20224_IncendieHavre%CC%81_FD.mp4'
)
self.assertEqual(
escape_url('http://www.ardmediathek.de/tv/Sturm-der-Liebe/Folge-2036-Zu-Mann-und-Frau-erklärt/Das-Erste/Video?documentId=22673108&bcastId=5290'),
'http://www.ardmediathek.de/tv/Sturm-der-Liebe/Folge-2036-Zu-Mann-und-Frau-erkl%C3%A4rt/Das-Erste/Video?documentId=22673108&bcastId=5290'
)
self.assertEqual(
escape_url('http://тест.рф/фрагмент'),
'http://тест.рф/%D1%84%D1%80%D0%B0%D0%B3%D0%BC%D0%B5%D0%BD%D1%82'
)
self.assertEqual(
escape_url('http://тест.рф/абв?абв=абв#абв'),
'http://тест.рф/%D0%B0%D0%B1%D0%B2?%D0%B0%D0%B1%D0%B2=%D0%B0%D0%B1%D0%B2#%D0%B0%D0%B1%D0%B2'
)
self.assertEqual(escape_url('http://vimeo.com/56015672#at=0'), 'http://vimeo.com/56015672#at=0')
def test_js_to_json_realworld(self):
inp = '''{
'clip':{'provider':'pseudo'}
}'''
self.assertEqual(js_to_json(inp), '''{
"clip":{"provider":"pseudo"}
}''')
json.loads(js_to_json(inp))
inp = '''{
'playlist':[{'controls':{'all':null}}]
}'''
self.assertEqual(js_to_json(inp), '''{
"playlist":[{"controls":{"all":null}}]
}''')
def test_js_to_json_edgecases(self):
on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
on = js_to_json('{"abc": true}')
self.assertEqual(json.loads(on), {'abc': True})
# Ignore JavaScript code as well
on = js_to_json('''{
"x": 1,
y: "a",
z: some.code
}''')
d = json.loads(on)
self.assertEqual(d['x'], 1)
self.assertEqual(d['y'], 'a')
def test_clean_html(self):
self.assertEqual(clean_html('a:\nb'), 'a: b')
self.assertEqual(clean_html('a:\n "b"'), 'a: "b"')
def test_intlist_to_bytes(self):
self.assertEqual(
intlist_to_bytes([0, 1, 127, 128, 255]),
b'\x00\x01\x7f\x80\xff')
def test_args_to_str(self):
self.assertEqual(
args_to_str(['foo', 'ba/r', '-baz', '2 be', '']),
'foo ba/r -baz \'2 be\' \'\''
)
def test_parse_filesize(self):
self.assertEqual(parse_filesize(None), None)
self.assertEqual(parse_filesize(''), None)
self.assertEqual(parse_filesize('91 B'), 91)
self.assertEqual(parse_filesize('foobar'), None)
self.assertEqual(parse_filesize('2 MiB'), 2097152)
self.assertEqual(parse_filesize('5 GB'), 5000000000)
self.assertEqual(parse_filesize('1.2Tb'), 1200000000000)
self.assertEqual(parse_filesize('1,24 KB'), 1240)
def test_version_tuple(self):
self.assertEqual(version_tuple('1'), (1,))
self.assertEqual(version_tuple('10.23.344'), (10, 23, 344))
self.assertEqual(version_tuple('10.1-6'), (10, 1, 6)) # avconv style
def test_detect_exe_version(self):
self.assertEqual(detect_exe_version('''ffmpeg version 1.2.1
built on May 27 2013 08:37:26 with gcc 4.7 (Debian 4.7.3-4)
configuration: --prefix=/usr --extra-'''), '1.2.1')
self.assertEqual(detect_exe_version('''ffmpeg version N-63176-g1fb4685
built on May 15 2014 22:09:06 with gcc 4.8.2 (GCC)'''), 'N-63176-g1fb4685')
self.assertEqual(detect_exe_version('''X server found. dri2 connection failed!
Trying to open render node...
Success at /dev/dri/renderD128.
ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
def test_age_restricted(self):
self.assertFalse(age_restricted(None, 10)) # unrestricted content
self.assertFalse(age_restricted(1, None)) # unrestricted policy
self.assertFalse(age_restricted(8, 10))
self.assertTrue(age_restricted(18, 14))
self.assertFalse(age_restricted(18, 18))
def test_is_html(self):
self.assertFalse(is_html(b'\x49\x44\x43<html'))
self.assertTrue(is_html(b'<!DOCTYPE foo>\xaaa'))
self.assertTrue(is_html( # UTF-8 with BOM
b'\xef\xbb\xbf<!DOCTYPE foo>\xaaa'))
self.assertTrue(is_html( # UTF-16-LE
b'\xff\xfe<\x00h\x00t\x00m\x00l\x00>\x00\xe4\x00'
))
self.assertTrue(is_html( # UTF-16-BE
b'\xfe\xff\x00<\x00h\x00t\x00m\x00l\x00>\x00\xe4'
))
self.assertTrue(is_html( # UTF-32-BE
b'\x00\x00\xFE\xFF\x00\x00\x00<\x00\x00\x00h\x00\x00\x00t\x00\x00\x00m\x00\x00\x00l\x00\x00\x00>\x00\x00\x00\xe4'))
self.assertTrue(is_html( # UTF-32-LE
b'\xFF\xFE\x00\x00<\x00\x00\x00h\x00\x00\x00t\x00\x00\x00m\x00\x00\x00l\x00\x00\x00>\x00\x00\x00\xe4\x00\x00\x00'))
def test_render_table(self):
self.assertEqual(
render_table(
['a', 'bcd'],
[[123, 4], [9999, 51]]),
'a bcd\n'
'123 4\n'
'9999 51')
if __name__ == '__main__':
unittest.main()

View File

@ -1,5 +1,6 @@
#!/usr/bin/env python
# coding: utf-8
from __future__ import unicode_literals
# Allow direct execution
import os
@ -31,17 +32,16 @@ params = get_params({
})
TEST_ID = 'gr51aVj-mLg'
ANNOTATIONS_FILE = TEST_ID + '.flv.annotations.xml'
EXPECTED_ANNOTATIONS = ['Speech bubble', 'Note', 'Title', 'Spotlight', 'Label']
class TestAnnotations(unittest.TestCase):
def setUp(self):
# Clear old files
self.tearDown()
def test_info_json(self):
expected = list(EXPECTED_ANNOTATIONS) # Two annotations could have the same text.
ie = youtube_dl.extractor.YoutubeIE()
@ -71,7 +71,6 @@ class TestAnnotations(unittest.TestCase):
# We should have seen (and removed) all the expected annotation texts.
self.assertEqual(len(expected), 0, 'Not all expected annotations were found.')
def tearDown(self):
try_rm(ANNOTATIONS_FILE)

View File

@ -1,75 +0,0 @@
#!/usr/bin/env python
# coding: utf-8
# Allow direct execution
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from test.helper import get_params
import io
import json
import youtube_dl.YoutubeDL
import youtube_dl.extractor
class YoutubeDL(youtube_dl.YoutubeDL):
def __init__(self, *args, **kwargs):
super(YoutubeDL, self).__init__(*args, **kwargs)
self.to_stderr = self.to_screen
params = get_params({
'writeinfojson': True,
'skip_download': True,
'writedescription': True,
})
TEST_ID = 'BaW_jenozKc'
INFO_JSON_FILE = TEST_ID + '.info.json'
DESCRIPTION_FILE = TEST_ID + '.mp4.description'
EXPECTED_DESCRIPTION = u'''test chars: "'/\ä↭𝕐
test URL: https://github.com/rg3/youtube-dl/issues/1892
This is a test video for youtube-dl.
For more information, contact phihag@phihag.de .'''
class TestInfoJSON(unittest.TestCase):
def setUp(self):
# Clear old files
self.tearDown()
def test_info_json(self):
ie = youtube_dl.extractor.YoutubeIE()
ydl = YoutubeDL(params)
ydl.add_info_extractor(ie)
ydl.download([TEST_ID])
self.assertTrue(os.path.exists(INFO_JSON_FILE))
with io.open(INFO_JSON_FILE, 'r', encoding='utf-8') as jsonf:
jd = json.load(jsonf)
self.assertEqual(jd['upload_date'], u'20121002')
self.assertEqual(jd['description'], EXPECTED_DESCRIPTION)
self.assertEqual(jd['id'], TEST_ID)
self.assertEqual(jd['extractor'], 'youtube')
self.assertEqual(jd['title'], u'''youtube-dl test video "'/\ä↭𝕐''')
self.assertEqual(jd['uploader'], 'Philipp Hagemeister')
self.assertTrue(os.path.exists(DESCRIPTION_FILE))
with io.open(DESCRIPTION_FILE, 'r', encoding='utf-8') as descf:
descr = descf.read()
self.assertEqual(descr, EXPECTED_DESCRIPTION)
def tearDown(self):
if os.path.exists(INFO_JSON_FILE):
os.remove(INFO_JSON_FILE)
if os.path.exists(DESCRIPTION_FILE):
os.remove(DESCRIPTION_FILE)
if __name__ == '__main__':
unittest.main()

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
@ -10,13 +11,8 @@ from test.helper import FakeYDL
from youtube_dl.extractor import (
YoutubeUserIE,
YoutubePlaylistIE,
YoutubeIE,
YoutubeChannelIE,
YoutubeShowIE,
YoutubeTopListIE,
YoutubeSearchURLIE,
)
@ -43,28 +39,6 @@ class TestYoutubeLists(unittest.TestCase):
self.assertEqual(len(entries), 25)
self.assertEqual(YoutubeIE().extract_id(entries[-1]['url']), 'rYefUsYuEp0')
def test_youtube_channel(self):
dl = FakeYDL()
ie = YoutubeChannelIE(dl)
#test paginated channel
result = ie.extract('https://www.youtube.com/channel/UCKfVa3S1e4PHvxWcwyMMg8w')
self.assertTrue(len(result['entries']) > 90)
#test autogenerated channel
result = ie.extract('https://www.youtube.com/channel/HCtnHdj3df7iM/videos')
self.assertTrue(len(result['entries']) >= 18)
def test_youtube_user(self):
dl = FakeYDL()
ie = YoutubeUserIE(dl)
result = ie.extract('https://www.youtube.com/user/TheLinuxFoundation')
self.assertTrue(len(result['entries']) >= 320)
def test_youtube_show(self):
dl = FakeYDL()
ie = YoutubeShowIE(dl)
result = ie.extract('http://www.youtube.com/show/airdisasters')
self.assertTrue(len(result) >= 3)
def test_youtube_mix(self):
dl = FakeYDL()
ie = YoutubePlaylistIE(dl)
@ -83,21 +57,5 @@ class TestYoutubeLists(unittest.TestCase):
entries = result['entries']
self.assertEqual(len(entries), 100)
def test_youtube_toplist(self):
dl = FakeYDL()
ie = YoutubeTopListIE(dl)
result = ie.extract('yttoplist:music:Trending')
entries = result['entries']
self.assertTrue(len(entries) >= 5)
def test_youtube_search_url(self):
dl = FakeYDL()
ie = YoutubeSearchURLIE(dl)
result = ie.extract('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video')
entries = result['entries']
self.assertIsPlaylist(result)
self.assertEqual(result['title'], 'youtube-dl test video')
self.assertTrue(len(entries) >= 5)
if __name__ == '__main__':
unittest.main()

View File

@ -14,7 +14,7 @@ import re
import string
from youtube_dl.extractor import YoutubeIE
from youtube_dl.utils import compat_str, compat_urlretrieve
from youtube_dl.compat import compat_str, compat_urlretrieve
_TESTS = [
(
@ -47,18 +47,6 @@ _TESTS = [
'2ACFC7A61CA478CD21425E5A57EBD73DDC78E22A.2094302436B2D377D14A3BBA23022D023B8BC25AA',
'A52CB8B320D22032ABB3A41D773D2B6342034902.A22E87CDD37DBE75A5E52412DC874AC16A7CFCA2',
),
(
'http://s.ytimg.com/yts/swfbin/player-vfl5vIhK2/watch_as3.swf',
'swf',
86,
'O1I3456789abcde0ghijklmnopqrstuvwxyzABCDEFGHfJKLMN2PQRSTUVWXY\\!"#$%&\'()*+,-./:;<=>?'
),
(
'http://s.ytimg.com/yts/swfbin/player-vflmDyk47/watch_as3.swf',
'swf',
'F375F75BF2AFDAAF2666E43868D46816F83F13E81C46.3725A8218E446A0DECD33F79DC282994D6AA92C92C9',
'9C29AA6D499282CD97F33DCED0A644E8128A5273.64C18E31F38361864D86834E6662FAADFA2FB57F'
),
(
'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflBb0OQx.js',
'js',

52
test/testcert.pem Normal file
View File

@ -0,0 +1,52 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDMF0bAzaHAdIyB
HRmnIp4vv40lGqEePmWqicCl0QZ0wsb5dNysSxSa7330M2QeQopGfdaUYF1uTcNp
Qx6ECgBSfg+RrOBI7r/u4F+sKX8MUXVaf/5QoBUrGNGSn/pp7HMGOuQqO6BVg4+h
A1ySSwUG8mZItLRry1ISyErmW8b9xlqfd97uLME/5tX+sMelRFjUbAx8A4CK58Ev
mMguHVTlXzx5RMdYcf1VScYcjlV/qA45uzP8zwI5aigfcmUD+tbGuQRhKxUhmw0J
aobtOR6+JSOAULW5gYa/egE4dWLwbyM6b6eFbdnjlQzEA1EW7ChMPAW/Mo83KyiP
tKMCSQulAgMBAAECggEALCfBDAexPjU5DNoh6bIorUXxIJzxTNzNHCdvgbCGiA54
BBKPh8s6qwazpnjT6WQWDIg/O5zZufqjE4wM9x4+0Zoqfib742ucJO9wY4way6x4
Clt0xzbLPabB+MoZ4H7ip+9n2+dImhe7pGdYyOHoNYeOL57BBi1YFW42Hj6u/8pd
63YCXisto3Rz1YvRQVjwsrS+cRKZlzAFQRviL30jav7Wh1aWEfcXxjj4zhm8pJdk
ITGtq6howz57M0NtX6hZnfe8ywzTnDFIGKIMA2cYHuYJcBh9bc4tCGubTvTKK9UE
8fM+f6UbfGqfpKCq1mcgs0XMoFDSzKS9+mSJn0+5JQKBgQD+OCKaeH3Yzw5zGnlw
XuQfMJGNcgNr+ImjmvzUAC2fAZUJLAcQueE5kzMv5Fmd+EFE2CEX1Vit3tg0SXvA
G+bq609doILHMA03JHnV1npO/YNIhG3AAtJlKYGxQNfWH9mflYj9mEui8ZFxG52o
zWhHYuifOjjZszUR+/eio6NPzwKBgQDNhUBTrT8LIX4SE/EFUiTlYmWIvOMgXYvN
8Cm3IRNQ/yyphZaXEU0eJzfX5uCDfSVOgd6YM/2pRah+t+1Hvey4H8e0GVTu5wMP
gkkqwKPGIR1YOmlw6ippqwvoJD7LuYrm6Q4D6e1PvkjwCq6lEndrOPmPrrXNd0JJ
XO60y3U2SwKBgQDLkyZarryQXxcCI6Q10Tc6pskYDMIit095PUbTeiUOXNT9GE28
Hi32ziLCakk9kCysNasii81MxtQ54tJ/f5iGbNMMddnkKl2a19Hc5LjjAm4cJzg/
98KGEhvyVqvAo5bBDZ06/rcrD+lZOzUglQS5jcIcqCIYa0LHWQ/wJLxFzwKBgFcZ
1SRhdSmDfUmuF+S4ZpistflYjC3IV5rk4NkS9HvMWaJS0nqdw4A3AMzItXgkjq4S
DkOVLTkTI5Do5HAWRv/VwC5M2hkR4NMu1VGAKSisGiKtRsirBWSZMEenLNHshbjN
Jrpz5rZ4H7NT46ZkCCZyFBpX4gb9NyOedjA7Via3AoGARF8RxbYjnEGGFuhnbrJB
FTPR0vaL4faY3lOgRZ8jOG9V2c9Hzi/y8a8TU4C11jnJSDqYCXBTd5XN28npYxtD
pjRsCwy6ze+yvYXPO7C978eMG3YRyj366NXUxnXN59ibwe/lxi2OD9z8J1LEdF6z
VJua1Wn8HKxnXMI61DhTCSo=
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIEEzCCAvugAwIBAgIJAK1haYi6gmSKMA0GCSqGSIb3DQEBCwUAMIGeMQswCQYD
VQQGEwJERTEMMAoGA1UECAwDTlJXMRQwEgYDVQQHDAtEdWVzc2VsZG9yZjEbMBkG
A1UECgwSeW91dHViZS1kbCBwcm9qZWN0MRkwFwYDVQQLDBB5b3V0dWJlLWRsIHRl
c3RzMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEHBoaWhhZ0Bw
aGloYWcuZGUwIBcNMTUwMTMwMDExNTA4WhgPMjExNTAxMDYwMTE1MDhaMIGeMQsw
CQYDVQQGEwJERTEMMAoGA1UECAwDTlJXMRQwEgYDVQQHDAtEdWVzc2VsZG9yZjEb
MBkGA1UECgwSeW91dHViZS1kbCBwcm9qZWN0MRkwFwYDVQQLDBB5b3V0dWJlLWRs
IHRlc3RzMRIwEAYDVQQDDAlsb2NhbGhvc3QxHzAdBgkqhkiG9w0BCQEWEHBoaWhh
Z0BwaGloYWcuZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMF0bA
zaHAdIyBHRmnIp4vv40lGqEePmWqicCl0QZ0wsb5dNysSxSa7330M2QeQopGfdaU
YF1uTcNpQx6ECgBSfg+RrOBI7r/u4F+sKX8MUXVaf/5QoBUrGNGSn/pp7HMGOuQq
O6BVg4+hA1ySSwUG8mZItLRry1ISyErmW8b9xlqfd97uLME/5tX+sMelRFjUbAx8
A4CK58EvmMguHVTlXzx5RMdYcf1VScYcjlV/qA45uzP8zwI5aigfcmUD+tbGuQRh
KxUhmw0JaobtOR6+JSOAULW5gYa/egE4dWLwbyM6b6eFbdnjlQzEA1EW7ChMPAW/
Mo83KyiPtKMCSQulAgMBAAGjUDBOMB0GA1UdDgQWBBTBUZoqhQkzHQ6xNgZfFxOd
ZEVt8TAfBgNVHSMEGDAWgBTBUZoqhQkzHQ6xNgZfFxOdZEVt8TAMBgNVHRMEBTAD
AQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCUOCl3T/J9B08Z+ijfOJAtkbUaEHuVZb4x
5EpZSy2ZbkLvtsftMFieHVNXn9dDswQc5qjYStCC4o60LKw4M6Y63FRsAZ/DNaqb
PY3jyCyuugZ8/sNf50vHYkAcF7SQYqOQFQX4TQsNUk2xMJIt7H0ErQFmkf/u3dg6
cy89zkT462IwxzSG7NNhIlRkL9o5qg+Y1mF9eZA1B0rcL6hO24PPTHOd90HDChBu
SZ6XMi/LzYQSTf0Vg2R+uMIVlzSlkdcZ6sqVnnqeLL8dFyIa4e9sj/D4ZCYP8Mqe
Z73H5/NNhmwCHRqVUTgm307xblQaWGhwAiDkaRvRW2aJQ0qGEdZK
-----END CERTIFICATE-----

View File

@ -7,8 +7,10 @@ import collections
import datetime
import errno
import io
import itertools
import json
import locale
import operator
import os
import platform
import re
@ -22,12 +24,18 @@ import traceback
if os.name == 'nt':
import ctypes
from .utils import (
from .compat import (
compat_basestring,
compat_cookiejar,
compat_expanduser,
compat_http_client,
compat_kwargs,
compat_str,
compat_urllib_error,
compat_urllib_request,
)
from .utils import (
escape_url,
ContentTooShortError,
date_from_str,
DateRange,
@ -43,24 +51,37 @@ from .utils import (
make_HTTPS_handler,
MaxDownloadsReached,
PagedList,
parse_filesize,
PostProcessingError,
platform_name,
preferredencoding,
render_table,
SameFileError,
sanitize_filename,
std_headers,
subtitles_filename,
takewhile_inclusive,
UnavailableVideoError,
url_basename,
version_tuple,
write_json_file,
write_string,
YoutubeDLHandler,
prepend_extension,
args_to_str,
age_restricted,
)
from .cache import Cache
from .extractor import get_info_extractor, gen_extractors
from .downloader import get_suitable_downloader
from .postprocessor import FFmpegMergerPP
from .downloader.rtmp import rtmpdump_version
from .postprocessor import (
FFmpegFixupM4aPP,
FFmpegFixupStretchedPP,
FFmpegMergerPP,
FFmpegPostProcessor,
get_postprocessor,
)
from .version import __version__
@ -106,8 +127,10 @@ class YoutubeDL(object):
forcefilename: Force printing final filename.
forceduration: Force printing duration.
forcejson: Force printing info_dict as JSON.
dump_single_json: Force printing the info_dict of the whole playlist
(or video) as a single JSON line.
simulate: Do not download the video files.
format: Video format code.
format: Video format code. See options.py for more information.
format_limit: Highest quality format to try.
outtmpl: Template for output names.
restrictfilenames: Do not allow "&" and spaces in file names
@ -115,6 +138,8 @@ class YoutubeDL(object):
nooverwrites: Prevent overwriting files.
playliststart: Playlist item to start at.
playlistend: Playlist item to end at.
playlist_items: Specific indices of playlist to download.
playlistreverse: Download playlist items in reverse order.
matchtitle: Download only matching titles.
rejecttitle: Reject downloads for matching titles.
logger: Log messages to a logging.Logger instance.
@ -123,6 +148,7 @@ class YoutubeDL(object):
writeinfojson: Write the video description to a .info.json file
writeannotations: Write the video annotations to a .annotations.xml file
writethumbnail: Write the thumbnail image to a file
write_all_thumbnails: Write all thumbnail formats to files
writesubtitles: Write the video subtitles to a file
writeautomaticsub: Write the automatic subtitles to a file
allsubtitles: Downloads all the subtitles of the video
@ -164,11 +190,51 @@ class YoutubeDL(object):
'auto' for elaborate guessing
encoding: Use this encoding instead of the system-specified.
extract_flat: Do not resolve URLs, return the immediate result.
Pass in 'in_playlist' to only show this behavior for
playlist items.
postprocessors: A list of dictionaries, each with an entry
* key: The name of the postprocessor. See
youtube_dl/postprocessor/__init__.py for a list.
as well as any further keyword arguments for the
postprocessor.
progress_hooks: A list of functions that get called on download
progress, with a dictionary with the entries
* status: One of "downloading" and "finished".
Check this first and ignore unknown values.
If status is one of "downloading" or "finished", the
following properties may also be present:
* filename: The final filename (always present)
* downloaded_bytes: Bytes on disk
* total_bytes: Size of the whole file, None if unknown
* tmpfilename: The filename we're currently writing to
* eta: The estimated time in seconds, None if unknown
* speed: The download speed in bytes/second, None if
unknown
Progress hooks are guaranteed to be called at least once
(with status "finished") if the download is successful.
merge_output_format: Extension to use when merging formats.
fixup: Automatically correct known faults of the file.
One of:
- "never": do nothing
- "warn": only emit a warning
- "detect_or_warn": check whether we can do anything
about it, warn otherwise (default)
source_address: (Experimental) Client-side IP address to bind to.
call_home: Boolean, true iff we are allowed to contact the
youtube-dl servers for debugging.
sleep_interval: Number of seconds to sleep before each download.
external_downloader: Executable of the external downloader to call.
listformats: Print an overview of available video formats and exit.
list_thumbnails: Print a table of all thumbnails and exit.
The following parameters are not used by YoutubeDL itself, they are used by
the FileDownloader:
nopart, updatetime, buffersize, ratelimit, min_filesize, max_filesize, test,
noresizebuffer, retries, continuedl, noprogress, consoletitle
noresizebuffer, retries, continuedl, noprogress, consoletitle,
xattr_set_filesize.
The following options are used by the post processors:
prefer_ffmpeg: If True, use ffmpeg instead of avconv if both are available,
@ -183,7 +249,7 @@ class YoutubeDL(object):
_num_downloads = None
_screen_file = None
def __init__(self, params=None):
def __init__(self, params=None, auto_init=True):
"""Create a FileDownloader object with the given options."""
if params is None:
params = {}
@ -227,11 +293,11 @@ class YoutubeDL(object):
if (sys.version_info >= (3,) and sys.platform != 'win32' and
sys.getfilesystemencoding() in ['ascii', 'ANSI_X3.4-1968']
and not params['restrictfilenames']):
and not params.get('restrictfilenames', False)):
# On Python 3, the Unicode filesystem API will throw errors (#1474)
self.report_warning(
'Assuming --restrict-filenames since file system encoding '
'cannot encode all charactes. '
'cannot encode all characters. '
'Set the LC_ALL environment variable to fix this.')
self.params['restrictfilenames'] = True
@ -240,6 +306,36 @@ class YoutubeDL(object):
self._setup_opener()
if auto_init:
self.print_debug_header()
self.add_default_info_extractors()
for pp_def_raw in self.params.get('postprocessors', []):
pp_class = get_postprocessor(pp_def_raw['key'])
pp_def = dict(pp_def_raw)
del pp_def['key']
pp = pp_class(self, **compat_kwargs(pp_def))
self.add_post_processor(pp)
for ph in self.params.get('progress_hooks', []):
self.add_progress_hook(ph)
def warn_if_short_id(self, argv):
# short YouTube ID starting with dash?
idxs = [
i for i, a in enumerate(argv)
if re.match(r'^-[0-9A-Za-z_-]{10}$', a)]
if idxs:
correct_argv = (
['youtube-dl'] +
[a for i, a in enumerate(argv) if i not in idxs] +
['--'] + [argv[i] for i in idxs]
)
self.report_warning(
'Long argument string detected. '
'Use -- to separate parameters and URLs, like this:\n%s\n' %
args_to_str(correct_argv))
def add_info_extractor(self, ie):
"""Add an InfoExtractor object to the end of the list."""
self._ies.append(ie)
@ -446,8 +542,13 @@ class YoutubeDL(object):
template_dict = collections.defaultdict(lambda: 'NA', template_dict)
outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
tmpl = os.path.expanduser(outtmpl)
tmpl = compat_expanduser(outtmpl)
filename = tmpl % template_dict
# Temporary fix for #4787
# 'Treat' all problem characters by passing filename through preferredencoding
# to workaround encoding issues with subprocess on python2 @ Windows
if sys.version_info < (3, 0) and sys.platform == 'win32':
filename = encodeFilename(filename, True).decode(preferredencoding())
return filename
except ValueError as err:
self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
@ -481,13 +582,8 @@ class YoutubeDL(object):
max_views = self.params.get('max_views')
if max_views is not None and view_count > max_views:
return 'Skipping %s, because it has exceeded the maximum view count (%d/%d)' % (video_title, view_count, max_views)
age_limit = self.params.get('age_limit')
if age_limit is not None:
actual_age_limit = info_dict.get('age_limit')
if actual_age_limit is None:
actual_age_limit = 0
if age_limit < actual_age_limit:
return 'Skipping "' + title + '" because it is age restricted'
if age_restricted(info_dict.get('age_limit'), self.params.get('age_limit')):
return 'Skipping "%s" because it is age restricted' % title
if self.in_download_archive(info_dict):
return '%s has already been recorded in archive' % video_title
return None
@ -567,8 +663,12 @@ class YoutubeDL(object):
result_type = ie_result.get('_type', 'video')
if self.params.get('extract_flat', False):
if result_type in ('url', 'url_transparent'):
extract_flat = self.params.get('extract_flat', False)
if ((extract_flat == 'in_playlist' and 'playlist' in extra_info) or
extract_flat is True):
if self.params.get('forcejson', False):
self.to_stdout(json.dumps(ie_result))
return ie_result
if result_type == 'video':
@ -587,27 +687,19 @@ class YoutubeDL(object):
ie_result['url'], ie_key=ie_result.get('ie_key'),
extra_info=extra_info, download=False, process=False)
def make_result(embedded_info):
new_result = ie_result.copy()
for f in ('_type', 'url', 'ext', 'player_url', 'formats',
'entries', 'ie_key', 'duration',
'subtitles', 'annotations', 'format',
'thumbnail', 'thumbnails'):
if f in new_result:
del new_result[f]
if f in embedded_info:
new_result[f] = embedded_info[f]
return new_result
new_result = make_result(info)
force_properties = dict(
(k, v) for k, v in ie_result.items() if v is not None)
for f in ('_type', 'url'):
if f in force_properties:
del force_properties[f]
new_result = info.copy()
new_result.update(force_properties)
assert new_result.get('_type') != 'url_transparent'
if new_result.get('_type') == 'compat_list':
new_result['entries'] = [
make_result(e) for e in new_result['entries']]
return self.process_ie_result(
new_result, download=download, extra_info=extra_info)
elif result_type == 'playlist':
elif result_type == 'playlist' or result_type == 'multi_video':
# We process each entry in the playlist
playlist = ie_result.get('title', None) or ie_result.get('id', None)
self.to_screen('[download] Downloading playlist: %s' % playlist)
@ -620,27 +712,66 @@ class YoutubeDL(object):
if playlistend == -1:
playlistend = None
if isinstance(ie_result['entries'], list):
n_all_entries = len(ie_result['entries'])
entries = ie_result['entries'][playliststart:playlistend]
playlistitems_str = self.params.get('playlist_items', None)
playlistitems = None
if playlistitems_str is not None:
def iter_playlistitems(format):
for string_segment in format.split(','):
if '-' in string_segment:
start, end = string_segment.split('-')
for item in range(int(start), int(end) + 1):
yield int(item)
else:
yield int(string_segment)
playlistitems = iter_playlistitems(playlistitems_str)
ie_entries = ie_result['entries']
if isinstance(ie_entries, list):
n_all_entries = len(ie_entries)
if playlistitems:
entries = [ie_entries[i - 1] for i in playlistitems]
else:
entries = ie_entries[playliststart:playlistend]
n_entries = len(entries)
self.to_screen(
"[%s] playlist %s: Collected %d video ids (downloading %d of them)" %
(ie_result['extractor'], playlist, n_all_entries, n_entries))
elif isinstance(ie_entries, PagedList):
if playlistitems:
entries = []
for item in playlistitems:
entries.extend(ie_entries.getslice(
item - 1, item
))
else:
assert isinstance(ie_result['entries'], PagedList)
entries = ie_result['entries'].getslice(
entries = ie_entries.getslice(
playliststart, playlistend)
n_entries = len(entries)
self.to_screen(
"[%s] playlist %s: Downloading %d videos" %
(ie_result['extractor'], playlist, n_entries))
else: # iterable
if playlistitems:
entry_list = list(ie_entries)
entries = [entry_list[i - 1] for i in playlistitems]
else:
entries = list(itertools.islice(
ie_entries, playliststart, playlistend))
n_entries = len(entries)
self.to_screen(
"[%s] playlist %s: Downloading %d videos" %
(ie_result['extractor'], playlist, n_entries))
if self.params.get('playlistreverse', False):
entries = entries[::-1]
for i, entry in enumerate(entries, 1):
self.to_screen('[download] Downloading video #%s of %s' % (i, n_entries))
self.to_screen('[download] Downloading video %s of %s' % (i, n_entries))
extra = {
'n_entries': n_entries,
'playlist': playlist,
'playlist_id': ie_result.get('id'),
'playlist_title': ie_result.get('title'),
'playlist_index': i + playliststart,
'extractor': ie_result['extractor'],
'webpage_url': ie_result['webpage_url'],
@ -660,14 +791,20 @@ class YoutubeDL(object):
ie_result['entries'] = playlist_results
return ie_result
elif result_type == 'compat_list':
self.report_warning(
'Extractor %s returned a compat_list result. '
'It needs to be updated.' % ie_result.get('extractor'))
def _fixup(r):
self.add_extra_info(r,
self.add_extra_info(
r,
{
'extractor': ie_result['extractor'],
'webpage_url': ie_result['webpage_url'],
'webpage_url_basename': url_basename(ie_result['webpage_url']),
'extractor_key': ie_result['extractor_key'],
})
}
)
return r
ie_result['entries'] = [
self.process_ie_result(_fixup(r), download, extra_info)
@ -677,7 +814,59 @@ class YoutubeDL(object):
else:
raise Exception('Invalid result type: %s' % result_type)
def _apply_format_filter(self, format_spec, available_formats):
" Returns a tuple of the remaining format_spec and filtered formats "
OPERATORS = {
'<': operator.lt,
'<=': operator.le,
'>': operator.gt,
'>=': operator.ge,
'=': operator.eq,
'!=': operator.ne,
}
operator_rex = re.compile(r'''(?x)\s*\[
(?P<key>width|height|tbr|abr|vbr|filesize|fps)
\s*(?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
(?P<value>[0-9.]+(?:[kKmMgGtTpPeEzZyY]i?[Bb]?)?)
\]$
''' % '|'.join(map(re.escape, OPERATORS.keys())))
m = operator_rex.search(format_spec)
if not m:
raise ValueError('Invalid format specification %r' % format_spec)
try:
comparison_value = int(m.group('value'))
except ValueError:
comparison_value = parse_filesize(m.group('value'))
if comparison_value is None:
comparison_value = parse_filesize(m.group('value') + 'B')
if comparison_value is None:
raise ValueError(
'Invalid value %r in format specification %r' % (
m.group('value'), format_spec))
op = OPERATORS[m.group('op')]
def _filter(f):
actual_value = f.get(m.group('key'))
if actual_value is None:
return m.group('none_inclusive')
return op(actual_value, comparison_value)
new_formats = [f for f in available_formats if _filter(f)]
new_format_spec = format_spec[:-len(m.group(0))]
if not new_format_spec:
new_format_spec = 'best'
return (new_format_spec, new_formats)
def select_format(self, format_spec, available_formats):
while format_spec.endswith(']'):
format_spec, available_formats = self._apply_format_filter(
format_spec, available_formats)
if not available_formats:
return None
if format_spec == 'best' or format_spec is None:
return available_formats[-1]
elif format_spec == 'worst':
@ -707,7 +896,7 @@ class YoutubeDL(object):
if video_formats:
return video_formats[0]
else:
extensions = ['mp4', 'flv', 'webm', '3gp']
extensions = ['mp4', 'flv', 'webm', '3gp', 'm4a', 'mp3', 'ogg', 'aac', 'wav']
if format_spec in extensions:
filter_f = lambda f: f['ext'] == format_spec
else:
@ -717,6 +906,42 @@ class YoutubeDL(object):
return matches[-1]
return None
def _calc_headers(self, info_dict):
res = std_headers.copy()
add_headers = info_dict.get('http_headers')
if add_headers:
res.update(add_headers)
cookies = self._calc_cookies(info_dict)
if cookies:
res['Cookie'] = cookies
return res
def _calc_cookies(self, info_dict):
class _PseudoRequest(object):
def __init__(self, url):
self.url = url
self.headers = {}
self.unverifiable = False
def add_unredirected_header(self, k, v):
self.headers[k] = v
def get_full_url(self):
return self.url
def is_unverifiable(self):
return self.unverifiable
def has_header(self, h):
return h in self.headers
pr = _PseudoRequest(info_dict['url'])
self.cookiejar.add_cookie_header(pr)
return pr.headers.get('Cookie')
def process_video_result(self, info_dict, download=True):
assert info_dict.get('_type', 'video') == 'video'
@ -731,9 +956,14 @@ class YoutubeDL(object):
info_dict['playlist_index'] = None
thumbnails = info_dict.get('thumbnails')
if thumbnails is None:
thumbnail = info_dict.get('thumbnail')
if thumbnail:
info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
if thumbnails:
thumbnails.sort(key=lambda t: (
t.get('width'), t.get('height'), t.get('url')))
t.get('preference'), t.get('width'), t.get('height'),
t.get('id'), t.get('url')))
for t in thumbnails:
if 'width' in t and 'height' in t:
t['resolution'] = '%dx%d' % (t['width'], t['height'])
@ -745,6 +975,10 @@ class YoutubeDL(object):
info_dict['display_id'] = info_dict['id']
if info_dict.get('upload_date') is None and info_dict.get('timestamp') is not None:
# Working around negative timestamps in Windows
# (see http://bugs.python.org/issue1646728)
if info_dict['timestamp'] < 0 and os.name == 'nt':
info_dict['timestamp'] = 0
upload_date = datetime.datetime.utcfromtimestamp(
info_dict['timestamp'])
info_dict['upload_date'] = upload_date.strftime('%Y%m%d')
@ -781,6 +1015,11 @@ class YoutubeDL(object):
# Automatically determine file extension if missing
if 'ext' not in format:
format['ext'] = determine_ext(format['url']).lower()
# Add HTTP headers, so that external programs can use them from the
# json output
full_format_info = info_dict.copy()
full_format_info.update(format)
format['http_headers'] = self._calc_headers(full_format_info)
format_limit = self.params.get('format_limit', None)
if format_limit:
@ -796,9 +1035,12 @@ class YoutubeDL(object):
# element in the 'formats' field in info_dict is info_dict itself,
# wich can't be exported to json
info_dict['formats'] = formats
if self.params.get('listformats', None):
if self.params.get('listformats'):
self.list_formats(info_dict)
return
if self.params.get('list_thumbnails'):
self.list_thumbnails(info_dict)
return
req_format = self.params.get('format')
if req_format is None:
@ -808,9 +1050,10 @@ class YoutubeDL(object):
if req_format in ('-1', 'all'):
formats_to_download = formats
else:
for rfstr in req_format.split(','):
# We can accept formats requested in the format: 34/5/best, we pick
# the first that is available, starting from left
req_formats = req_format.split('/')
req_formats = rfstr.split('/')
for rf in req_formats:
if re.match(r'.+?\+.+?', rf) is not None:
# Two formats have been requested like '137+139'
@ -818,17 +1061,39 @@ class YoutubeDL(object):
formats_info = (self.select_format(format_1, formats),
self.select_format(format_2, formats))
if all(formats_info):
# The first format must contain the video and the
# second the audio
if formats_info[0].get('vcodec') == 'none':
self.report_error('The first format must '
'contain the video, try using '
'"-f %s+%s"' % (format_2, format_1))
return
output_ext = (
formats_info[0]['ext']
if self.params.get('merge_output_format') is None
else self.params['merge_output_format'])
selected_format = {
'requested_formats': formats_info,
'format': rf,
'ext': formats_info[0]['ext'],
'format_id': '%s+%s' % (formats_info[0].get('format_id'),
formats_info[1].get('format_id')),
'width': formats_info[0].get('width'),
'height': formats_info[0].get('height'),
'resolution': formats_info[0].get('resolution'),
'fps': formats_info[0].get('fps'),
'vcodec': formats_info[0].get('vcodec'),
'vbr': formats_info[0].get('vbr'),
'stretched_ratio': formats_info[0].get('stretched_ratio'),
'acodec': formats_info[1].get('acodec'),
'abr': formats_info[1].get('abr'),
'ext': output_ext,
}
else:
selected_format = None
else:
selected_format = self.select_format(rf, formats)
if selected_format is not None:
formats_to_download = [selected_format]
formats_to_download.append(selected_format)
break
if not formats_to_download:
raise ExtractorError('requested format not available',
@ -872,7 +1137,7 @@ class YoutubeDL(object):
self._num_downloads += 1
filename = self.prepare_filename(info_dict)
info_dict['_filename'] = filename = self.prepare_filename(info_dict)
# Forced printings
if self.params.get('forcetitle', False):
@ -880,6 +1145,10 @@ class YoutubeDL(object):
if self.params.get('forceid', False):
self.to_stdout(info_dict['id'])
if self.params.get('forceurl', False):
if info_dict.get('requested_formats') is not None:
for f in info_dict['requested_formats']:
self.to_stdout(f['url'] + f.get('play_path', ''))
else:
# For RTMP URLs, also include the playpath
self.to_stdout(info_dict['url'] + info_dict.get('play_path', ''))
if self.params.get('forcethumbnail', False) and info_dict.get('thumbnail') is not None:
@ -893,7 +1162,6 @@ class YoutubeDL(object):
if self.params.get('forceformat', False):
self.to_stdout(info_dict['format'])
if self.params.get('forcejson', False):
info_dict['_filename'] = filename
self.to_stdout(json.dumps(info_dict))
# Do nothing else if in simulate mode
@ -915,13 +1183,13 @@ class YoutubeDL(object):
descfn = filename + '.description'
if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(descfn)):
self.to_screen('[info] Video description is already present')
elif info_dict.get('description') is None:
self.report_warning('There\'s no description to write.')
else:
try:
self.to_screen('[info] Writing video description to: ' + descfn)
with io.open(encodeFilename(descfn), 'w', encoding='utf-8') as descfile:
descfile.write(info_dict['description'])
except (KeyError, TypeError):
self.report_warning('There\'s no description to write.')
except (OSError, IOError):
self.report_error('Cannot write description file ' + descfn)
return
@ -972,48 +1240,28 @@ class YoutubeDL(object):
else:
self.to_screen('[info] Writing video description metadata as JSON to: ' + infofn)
try:
write_json_file(info_dict, encodeFilename(infofn))
write_json_file(info_dict, infofn)
except (OSError, IOError):
self.report_error('Cannot write metadata to JSON file ' + infofn)
return
if self.params.get('writethumbnail', False):
if info_dict.get('thumbnail') is not None:
thumb_format = determine_ext(info_dict['thumbnail'], 'jpg')
thumb_filename = os.path.splitext(filename)[0] + '.' + thumb_format
if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(thumb_filename)):
self.to_screen('[%s] %s: Thumbnail is already present' %
(info_dict['extractor'], info_dict['id']))
else:
self.to_screen('[%s] %s: Downloading thumbnail ...' %
(info_dict['extractor'], info_dict['id']))
try:
uf = self.urlopen(info_dict['thumbnail'])
with open(thumb_filename, 'wb') as thumbf:
shutil.copyfileobj(uf, thumbf)
self.to_screen('[%s] %s: Writing thumbnail to: %s' %
(info_dict['extractor'], info_dict['id'], thumb_filename))
except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
self.report_warning('Unable to download thumbnail "%s": %s' %
(info_dict['thumbnail'], compat_str(err)))
self._write_thumbnails(info_dict, filename)
if not self.params.get('skip_download', False):
if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(filename)):
success = True
else:
try:
def dl(name, info):
fd = get_suitable_downloader(info)(self, self.params)
fd = get_suitable_downloader(info, self.params)(self, self.params)
for ph in self._progress_hooks:
fd.add_progress_hook(ph)
if self.params.get('verbose'):
self.to_stdout('[debug] Invoking downloader on %r' % info.get('url'))
return fd.download(name, info)
if info_dict.get('requested_formats') is not None:
downloaded = []
success = True
merger = FFmpegMergerPP(self, not self.params.get('keepvideo'))
if not merger._get_executable():
if not merger._executable:
postprocessors = []
self.report_warning('You have requested multiple '
'formats but ffmpeg or avconv are not installed.'
@ -1043,12 +1291,49 @@ class YoutubeDL(object):
return
if success:
# Fixup content
fixup_policy = self.params.get('fixup')
if fixup_policy is None:
fixup_policy = 'detect_or_warn'
stretched_ratio = info_dict.get('stretched_ratio')
if stretched_ratio is not None and stretched_ratio != 1:
if fixup_policy == 'warn':
self.report_warning('%s: Non-uniform pixel ratio (%s)' % (
info_dict['id'], stretched_ratio))
elif fixup_policy == 'detect_or_warn':
stretched_pp = FFmpegFixupStretchedPP(self)
if stretched_pp.available:
info_dict.setdefault('__postprocessors', [])
info_dict['__postprocessors'].append(stretched_pp)
else:
self.report_warning(
'%s: Non-uniform pixel ratio (%s). Install ffmpeg or avconv to fix this automatically.' % (
info_dict['id'], stretched_ratio))
else:
assert fixup_policy in ('ignore', 'never')
if info_dict.get('requested_formats') is None and info_dict.get('container') == 'm4a_dash':
if fixup_policy == 'warn':
self.report_warning('%s: writing DASH m4a. Only some players support this container.' % (
info_dict['id']))
elif fixup_policy == 'detect_or_warn':
fixup_pp = FFmpegFixupM4aPP(self)
if fixup_pp.available:
info_dict.setdefault('__postprocessors', [])
info_dict['__postprocessors'].append(fixup_pp)
else:
self.report_warning(
'%s: writing DASH m4a. Only some players support this container. Install ffmpeg or avconv to fix this automatically.' % (
info_dict['id']))
else:
assert fixup_policy in ('ignore', 'never')
try:
self.post_process(filename, info_dict)
except (PostProcessingError) as err:
self.report_error('postprocessing: %s' % str(err))
return
self.record_download_archive(info_dict)
def download(self, url_list):
@ -1062,12 +1347,15 @@ class YoutubeDL(object):
for url in url_list:
try:
# It also downloads the videos
self.extract_info(url)
res = self.extract_info(url)
except UnavailableVideoError:
self.report_error('unable to download video')
except MaxDownloadsReached:
self.to_screen('[info] Maximum number of downloaded files reached.')
raise
else:
if self.params.get('dump_single_json', False):
self.to_stdout(json.dumps(res))
return self._download_retcode
@ -1089,14 +1377,15 @@ class YoutubeDL(object):
"""Run all the postprocessors on the given file."""
info = dict(ie_info)
info['filepath'] = filename
keep_video = None
pps_chain = []
if ie_info.get('__postprocessors') is not None:
pps_chain.extend(ie_info['__postprocessors'])
pps_chain.extend(self._pps)
for pp in pps_chain:
keep_video = None
old_filename = info['filepath']
try:
keep_video_wish, new_info = pp.run(info)
keep_video_wish, info = pp.run(info)
if keep_video_wish is not None:
if keep_video_wish:
keep_video = keep_video_wish
@ -1107,8 +1396,8 @@ class YoutubeDL(object):
self.report_error(e.msg)
if keep_video is False and not self.params.get('keepvideo', False):
try:
self.to_screen('Deleting original file %s (pass -k to keep)' % filename)
os.remove(encodeFilename(filename))
self.to_screen('Deleting original file %s (pass -k to keep)' % old_filename)
os.remove(encodeFilename(old_filename))
except (IOError, OSError):
self.report_warning('Unable to remove downloaded video file')
@ -1191,6 +1480,8 @@ class YoutubeDL(object):
res += 'video@'
if fdict.get('vbr') is not None:
res += '%4dk' % fdict['vbr']
if fdict.get('fps') is not None:
res += ', %sfps' % fdict['fps']
if fdict.get('acodec') is not None:
if res:
res += ', '
@ -1228,7 +1519,9 @@ class YoutubeDL(object):
formats = info_dict.get('formats', [info_dict])
idlen = max(len('format code'),
max(len(f['format_id']) for f in formats))
formats_s = [line(f, idlen) for f in formats]
formats_s = [
line(f, idlen) for f in formats
if f.get('preference') is None or f['preference'] >= -1000]
if len(formats) > 1:
formats_s[0] += (' ' if self._format_note(formats[0]) else '') + '(worst)'
formats_s[-1] += (' ' if self._format_note(formats[-1]) else '') + '(best)'
@ -1236,11 +1529,49 @@ class YoutubeDL(object):
header_line = line({
'format_id': 'format code', 'ext': 'extension',
'resolution': 'resolution', 'format_note': 'note'}, idlen=idlen)
self.to_screen('[info] Available formats for %s:\n%s\n%s' %
self.to_screen(
'[info] Available formats for %s:\n%s\n%s' %
(info_dict['id'], header_line, '\n'.join(formats_s)))
def list_thumbnails(self, info_dict):
thumbnails = info_dict.get('thumbnails')
if not thumbnails:
tn_url = info_dict.get('thumbnail')
if tn_url:
thumbnails = [{'id': '0', 'url': tn_url}]
else:
self.to_screen(
'[info] No thumbnails present for %s' % info_dict['id'])
return
self.to_screen(
'[info] Thumbnails for %s:' % info_dict['id'])
self.to_screen(render_table(
['ID', 'width', 'height', 'URL'],
[[t['id'], t.get('width', 'unknown'), t.get('height', 'unknown'), t['url']] for t in thumbnails]))
def urlopen(self, req):
""" Start an HTTP download """
# According to RFC 3986, URLs can not contain non-ASCII characters, however this is not
# always respected by websites, some tend to give out URLs with non percent-encoded
# non-ASCII characters (see telemb.py, ard.py [#3412])
# urllib chokes on URLs with non-ASCII characters (see http://bugs.python.org/issue3991)
# To work around aforementioned issue we will replace request's original URL with
# percent-encoded one
req_is_string = isinstance(req, compat_basestring)
url = req if req_is_string else req.get_full_url()
url_escaped = escape_url(url)
# Substitute URL if any change after escaping
if url != url_escaped:
if req_is_string:
req = url_escaped
else:
req = compat_urllib_request.Request(
url_escaped, data=req.data, headers=req.headers,
origin_req_host=req.origin_req_host, unverifiable=req.unverifiable)
return self._opener.open(req, timeout=self._socket_timeout)
def print_debug_header(self):
@ -1252,11 +1583,13 @@ class YoutubeDL(object):
self.report_warning(
'Your Python is broken! Update to a newer and supported version')
stdout_encoding = getattr(
sys.stdout, 'encoding', 'missing (%s)' % type(sys.stdout).__name__)
encoding_str = (
'[debug] Encodings: locale %s, fs %s, out %s, pref %s\n' % (
locale.getpreferredencoding(),
sys.getfilesystemencoding(),
sys.stdout.encoding,
stdout_encoding,
self.get_encoding()))
write_string(encoding_str, encoding=None)
@ -1275,8 +1608,19 @@ class YoutubeDL(object):
sys.exc_clear()
except:
pass
self._write_string('[debug] Python version %s - %s' %
(platform.python_version(), platform_name()) + '\n')
self._write_string('[debug] Python version %s - %s\n' % (
platform.python_version(), platform_name()))
exe_versions = FFmpegPostProcessor.get_versions()
exe_versions['rtmpdump'] = rtmpdump_version()
exe_str = ', '.join(
'%s %s' % (exe, v)
for exe, v in sorted(exe_versions.items())
if v
)
if not exe_str:
exe_str = 'none'
self._write_string('[debug] exe versions: %s\n' % exe_str)
proxy_map = {}
for handler in self._opener.handlers:
@ -1284,6 +1628,17 @@ class YoutubeDL(object):
proxy_map.update(handler.proxies)
self._write_string('[debug] Proxy map: ' + compat_str(proxy_map) + '\n')
if self.params.get('call_home', False):
ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode('utf-8')
self._write_string('[debug] Public IP address: %s\n' % ipaddr)
latest_version = self.urlopen(
'https://yt-dl.org/latest/version').read().decode('utf-8')
if version_tuple(latest_version) > version_tuple(__version__):
self.report_warning(
'You are using an outdated version (newest version: %s)! '
'See https://yt-dl.org/update if you need help updating.' %
latest_version)
def _setup_opener(self):
timeout_val = self.params.get('socket_timeout')
self._socket_timeout = 600 if timeout_val is None else float(timeout_val)
@ -1314,9 +1669,8 @@ class YoutubeDL(object):
proxy_handler = compat_urllib_request.ProxyHandler(proxies)
debuglevel = 1 if self.params.get('debug_printtraffic') else 0
https_handler = make_HTTPS_handler(
self.params.get('nocheckcertificate', False), debuglevel=debuglevel)
ydlh = YoutubeDLHandler(debuglevel=debuglevel)
https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
ydlh = YoutubeDLHandler(self.params, debuglevel=debuglevel)
opener = compat_urllib_request.build_opener(
https_handler, proxy_handler, cookie_processor, ydlh)
# Delete the default user-agent header, which would otherwise apply in
@ -1340,3 +1694,39 @@ class YoutubeDL(object):
if encoding is None:
encoding = preferredencoding()
return encoding
def _write_thumbnails(self, info_dict, filename):
if self.params.get('writethumbnail', False):
thumbnails = info_dict.get('thumbnails')
if thumbnails:
thumbnails = [thumbnails[-1]]
elif self.params.get('write_all_thumbnails', False):
thumbnails = info_dict.get('thumbnails')
else:
return
if not thumbnails:
# No thumbnails present, so return immediately
return
for t in thumbnails:
thumb_ext = determine_ext(t['url'], 'jpg')
suffix = '_%s' % t['id'] if len(thumbnails) > 1 else ''
thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
thumb_filename = os.path.splitext(filename)[0] + suffix + '.' + thumb_ext
if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(thumb_filename)):
self.to_screen('[%s] %s: Thumbnail %sis already present' %
(info_dict['extractor'], info_dict['id'], thumb_display_id))
else:
self.to_screen('[%s] %s: Downloading thumbnail %s...' %
(info_dict['extractor'], info_dict['id'], thumb_display_id))
try:
uf = self.urlopen(t['url'])
with open(thumb_filename, 'wb') as thumbf:
shutil.copyfileobj(uf, thumbf)
self.to_screen('[%s] %s: Writing thumbnail %sto: %s' %
(info_dict['extractor'], info_dict['id'], thumb_display_id, thumb_filename))
except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
self.report_warning('Unable to download thumbnail "%s": %s' %
(t['url'], compat_str(err)))

View File

@ -1,81 +1,7 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
__authors__ = (
'Ricardo Garcia Gonzalez',
'Danny Colligan',
'Benjamin Johnson',
'Vasyl\' Vavrychuk',
'Witold Baryluk',
'Paweł Paprota',
'Gergely Imreh',
'Rogério Brito',
'Philipp Hagemeister',
'Sören Schulze',
'Kevin Ngo',
'Ori Avtalion',
'shizeeg',
'Filippo Valsorda',
'Christian Albrecht',
'Dave Vasilevsky',
'Jaime Marquínez Ferrándiz',
'Jeff Crouse',
'Osama Khalid',
'Michael Walter',
'M. Yasoob Ullah Khalid',
'Julien Fraichard',
'Johny Mo Swag',
'Axel Noack',
'Albert Kim',
'Pierre Rudloff',
'Huarong Huo',
'Ismael Mejía',
'Steffan \'Ruirize\' James',
'Andras Elso',
'Jelle van der Waa',
'Marcin Cieślak',
'Anton Larionov',
'Takuya Tsuchida',
'Sergey M.',
'Michael Orlitzky',
'Chris Gahan',
'Saimadhav Heblikar',
'Mike Col',
'Oleg Prutz',
'pulpe',
'Andreas Schmitz',
'Michael Kaiser',
'Niklas Laxström',
'David Triendl',
'Anthony Weems',
'David Wagner',
'Juan C. Olivares',
'Mattias Harrysson',
'phaer',
'Sainyam Kapoor',
'Nicolas Évrard',
'Jason Normore',
'Hoje Lee',
'Adam Thalhammer',
'Georg Jähnig',
'Ralf Haring',
'Koki Takahashi',
'Ariset Llerena',
'Adam Malcontenti-Wilson',
'Tobias Bell',
'Naglis Jonaitis',
'Charles Chen',
'Hassaan Ali',
'Dobrosław Żybort',
'David Fabijan',
'Sebastian Haas',
'Alexander Kirk',
'Erik Johnson',
'Keith Beckman',
'Ole Ernst',
'Aaron McDaniel (mcd1992)',
'Magnus Kolstad',
)
from __future__ import unicode_literals
__license__ = 'Public Domain'
@ -89,9 +15,13 @@ import sys
from .options import (
parseOpts,
)
from .utils import (
from .compat import (
compat_expanduser,
compat_getpass,
compat_print,
workaround_optparse_bug9161,
)
from .utils import (
DateRange,
DEFAULT_OUTTMPL,
decodeOption,
@ -108,18 +38,8 @@ from .update import update_self
from .downloader import (
FileDownloader,
)
from .extractor import gen_extractors
from .extractor import gen_extractors, list_extractors
from .YoutubeDL import YoutubeDL
from .postprocessor import (
AtomicParsleyPP,
FFmpegAudioFixPP,
FFmpegMetadataPP,
FFmpegVideoConvertor,
FFmpegExtractAudioPP,
FFmpegEmbedSubtitlePP,
XAttrMetadataPP,
ExecAfterDownloadPP,
)
def _real_main(argv=None):
@ -128,7 +48,9 @@ def _real_main(argv=None):
# https://github.com/rg3/youtube-dl/issues/820
codecs.register(lambda name: codecs.lookup('utf-8') if name == 'cp65001' else None)
setproctitle(u'youtube-dl')
workaround_optparse_bug9161()
setproctitle('youtube-dl')
parser, opts, args = parseOpts(argv)
@ -144,10 +66,10 @@ def _real_main(argv=None):
if opts.headers is not None:
for h in opts.headers:
if h.find(':', 1) < 0:
parser.error(u'wrong header formatting, it should be key:value, not "%s"'%h)
parser.error('wrong header formatting, it should be key:value, not "%s"' % h)
key, value = h.split(':', 2)
if opts.verbose:
write_string(u'[debug] Adding header from command line option %s:%s\n'%(key, value))
write_string('[debug] Adding header from command line option %s:%s\n' % (key, value))
std_headers[key] = value
# Dump user agent
@ -165,94 +87,93 @@ def _real_main(argv=None):
batchfd = io.open(opts.batchfile, 'r', encoding='utf-8', errors='ignore')
batch_urls = read_batch_urls(batchfd)
if opts.verbose:
write_string(u'[debug] Batch file urls: ' + repr(batch_urls) + u'\n')
write_string('[debug] Batch file urls: ' + repr(batch_urls) + '\n')
except IOError:
sys.exit(u'ERROR: batch file could not be read')
sys.exit('ERROR: batch file could not be read')
all_urls = batch_urls + args
all_urls = [url.strip() for url in all_urls]
_enc = preferredencoding()
all_urls = [url.decode(_enc, 'ignore') if isinstance(url, bytes) else url for url in all_urls]
extractors = gen_extractors()
if opts.list_extractors:
for ie in sorted(extractors, key=lambda ie: ie.IE_NAME.lower()):
for ie in list_extractors(opts.age_limit):
compat_print(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else ''))
matchedUrls = [url for url in all_urls if ie.suitable(url)]
for mu in matchedUrls:
compat_print(u' ' + mu)
compat_print(' ' + mu)
sys.exit(0)
if opts.list_extractor_descriptions:
for ie in sorted(extractors, key=lambda ie: ie.IE_NAME.lower()):
for ie in list_extractors(opts.age_limit):
if not ie._WORKING:
continue
desc = getattr(ie, 'IE_DESC', ie.IE_NAME)
if desc is False:
continue
if hasattr(ie, 'SEARCH_KEY'):
_SEARCHES = (u'cute kittens', u'slithering pythons', u'falling cat', u'angry poodle', u'purple fish', u'running tortoise', u'sleeping bunny')
_COUNTS = (u'', u'5', u'10', u'all')
desc += u' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
_SEARCHES = ('cute kittens', 'slithering pythons', 'falling cat', 'angry poodle', 'purple fish', 'running tortoise', 'sleeping bunny', 'burping cow')
_COUNTS = ('', '5', '10', 'all')
desc += ' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
compat_print(desc)
sys.exit(0)
# Conflicting, missing and erroneous options
if opts.usenetrc and (opts.username is not None or opts.password is not None):
parser.error(u'using .netrc conflicts with giving username/password')
parser.error('using .netrc conflicts with giving username/password')
if opts.password is not None and opts.username is None:
parser.error(u'account username missing\n')
parser.error('account username missing\n')
if opts.outtmpl is not None and (opts.usetitle or opts.autonumber or opts.useid):
parser.error(u'using output template conflicts with using title, video ID or auto number')
parser.error('using output template conflicts with using title, video ID or auto number')
if opts.usetitle and opts.useid:
parser.error(u'using title conflicts with using video ID')
parser.error('using title conflicts with using video ID')
if opts.username is not None and opts.password is None:
opts.password = compat_getpass(u'Type account password and press [Return]: ')
opts.password = compat_getpass('Type account password and press [Return]: ')
if opts.ratelimit is not None:
numeric_limit = FileDownloader.parse_bytes(opts.ratelimit)
if numeric_limit is None:
parser.error(u'invalid rate limit specified')
parser.error('invalid rate limit specified')
opts.ratelimit = numeric_limit
if opts.min_filesize is not None:
numeric_limit = FileDownloader.parse_bytes(opts.min_filesize)
if numeric_limit is None:
parser.error(u'invalid min_filesize specified')
parser.error('invalid min_filesize specified')
opts.min_filesize = numeric_limit
if opts.max_filesize is not None:
numeric_limit = FileDownloader.parse_bytes(opts.max_filesize)
if numeric_limit is None:
parser.error(u'invalid max_filesize specified')
parser.error('invalid max_filesize specified')
opts.max_filesize = numeric_limit
if opts.retries is not None:
if opts.retries in ('inf', 'infinite'):
opts_retries = float('inf')
else:
try:
opts.retries = int(opts.retries)
opts_retries = int(opts.retries)
except (TypeError, ValueError):
parser.error(u'invalid retry count specified')
parser.error('invalid retry count specified')
if opts.buffersize is not None:
numeric_buffersize = FileDownloader.parse_bytes(opts.buffersize)
if numeric_buffersize is None:
parser.error(u'invalid buffer size specified')
parser.error('invalid buffer size specified')
opts.buffersize = numeric_buffersize
if opts.playliststart <= 0:
raise ValueError(u'Playlist start must be positive')
raise ValueError('Playlist start must be positive')
if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
raise ValueError(u'Playlist end must be greater than playlist start')
raise ValueError('Playlist end must be greater than playlist start')
if opts.extractaudio:
if opts.audioformat not in ['best', 'aac', 'mp3', 'm4a', 'opus', 'vorbis', 'wav']:
parser.error(u'invalid audio format specified')
parser.error('invalid audio format specified')
if opts.audioquality:
opts.audioquality = opts.audioquality.strip('k').strip('K')
if not opts.audioquality.isdigit():
parser.error(u'invalid audio quality specified')
parser.error('invalid audio quality specified')
if opts.recodevideo is not None:
if opts.recodevideo not in ['mp4', 'flv', 'webm', 'ogg', 'mkv']:
parser.error(u'invalid video recode format specified')
parser.error('invalid video recode format specified')
if opts.date is not None:
date = DateRange.day(opts.date)
else:
date = DateRange(opts.dateafter, opts.datebefore)
if opts.default_search not in ('auto', 'auto_warning', 'error', 'fixup_error', None) and ':' not in opts.default_search:
parser.error(u'--default-search invalid; did you forget a colon (:) at the end?')
# Do not download videos when there are audio-only formats
if opts.extractaudio and not opts.keepvideo and opts.format is None:
@ -260,7 +181,7 @@ def _real_main(argv=None):
# --all-sub automatically sets --write-sub if --write-auto-sub is not given
# this was the old behaviour if only --all-sub was given.
if opts.allsubtitles and (opts.writeautomaticsub == False):
if opts.allsubtitles and not opts.writeautomaticsub:
opts.writesubtitles = True
if sys.version_info < (3,):
@ -268,20 +189,64 @@ def _real_main(argv=None):
if opts.outtmpl is not None:
opts.outtmpl = opts.outtmpl.decode(preferredencoding())
outtmpl = ((opts.outtmpl is not None and opts.outtmpl)
or (opts.format == '-1' and opts.usetitle and u'%(title)s-%(id)s-%(format)s.%(ext)s')
or (opts.format == '-1' and u'%(id)s-%(format)s.%(ext)s')
or (opts.usetitle and opts.autonumber and u'%(autonumber)s-%(title)s-%(id)s.%(ext)s')
or (opts.usetitle and u'%(title)s-%(id)s.%(ext)s')
or (opts.useid and u'%(id)s.%(ext)s')
or (opts.autonumber and u'%(autonumber)s-%(id)s.%(ext)s')
or (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s')
or (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s')
or (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s')
or (opts.usetitle and '%(title)s-%(id)s.%(ext)s')
or (opts.useid and '%(id)s.%(ext)s')
or (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s')
or DEFAULT_OUTTMPL)
if not os.path.splitext(outtmpl)[1] and opts.extractaudio:
parser.error(u'Cannot download a video and extract audio into the same'
u' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
u' template'.format(outtmpl))
parser.error('Cannot download a video and extract audio into the same'
' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
' template'.format(outtmpl))
any_printing = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson
download_archive_fn = os.path.expanduser(opts.download_archive) if opts.download_archive is not None else opts.download_archive
any_getting = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
any_printing = opts.print_json
download_archive_fn = compat_expanduser(opts.download_archive) if opts.download_archive is not None else opts.download_archive
# PostProcessors
postprocessors = []
# Add the metadata pp first, the other pps will copy it
if opts.addmetadata:
postprocessors.append({'key': 'FFmpegMetadata'})
if opts.extractaudio:
postprocessors.append({
'key': 'FFmpegExtractAudio',
'preferredcodec': opts.audioformat,
'preferredquality': opts.audioquality,
'nopostoverwrites': opts.nopostoverwrites,
})
if opts.recodevideo:
postprocessors.append({
'key': 'FFmpegVideoConvertor',
'preferedformat': opts.recodevideo,
})
if opts.embedsubtitles:
postprocessors.append({
'key': 'FFmpegEmbedSubtitle',
'subtitlesformat': opts.subtitlesformat,
})
if opts.xattrs:
postprocessors.append({'key': 'XAttrMetadata'})
if opts.embedthumbnail:
if not opts.addmetadata:
postprocessors.append({'key': 'FFmpegAudioFix'})
postprocessors.append({'key': 'AtomicParsley'})
# Please keep ExecAfterDownload towards the bottom as it allows the user to modify the final file in any way.
# So if the user is able to remove the file before your postprocessor runs it might cause a few problems.
if opts.exec_cmd:
postprocessors.append({
'key': 'ExecAfterDownload',
'verboseOutput': opts.verbose,
'exec_cmd': opts.exec_cmd,
})
if opts.xattr_set_filesize:
try:
import xattr
xattr # Confuse flake8
except ImportError:
parser.error('setting filesize xattr requested but python-xattr is not available')
ydl_opts = {
'usenetrc': opts.usenetrc,
@ -289,7 +254,7 @@ def _real_main(argv=None):
'password': opts.password,
'twofactor': opts.twofactor,
'videopassword': opts.videopassword,
'quiet': (opts.quiet or any_printing),
'quiet': (opts.quiet or any_getting or any_printing),
'no_warnings': opts.no_warnings,
'forceurl': opts.geturl,
'forcetitle': opts.gettitle,
@ -299,9 +264,10 @@ def _real_main(argv=None):
'forceduration': opts.getduration,
'forcefilename': opts.getfilename,
'forceformat': opts.getformat,
'forcejson': opts.dumpjson,
'simulate': opts.simulate,
'skip_download': (opts.skip_download or opts.simulate or any_printing),
'forcejson': opts.dumpjson or opts.print_json,
'dump_single_json': opts.dump_single_json,
'simulate': opts.simulate or any_getting,
'skip_download': opts.skip_download,
'format': opts.format,
'format_limit': opts.format_limit,
'listformats': opts.listformats,
@ -311,7 +277,7 @@ def _real_main(argv=None):
'ignoreerrors': opts.ignoreerrors,
'ratelimit': opts.ratelimit,
'nooverwrites': opts.nooverwrites,
'retries': opts.retries,
'retries': opts_retries,
'buffersize': opts.buffersize,
'noresizebuffer': opts.noresizebuffer,
'continuedl': opts.continue_dl,
@ -319,6 +285,7 @@ def _real_main(argv=None):
'progress_with_newline': opts.progress_with_newline,
'playliststart': opts.playliststart,
'playlistend': opts.playlistend,
'playlistreverse': opts.playlist_reverse,
'noplaylist': opts.noplaylist,
'logtostderr': opts.outtmpl == '-',
'consoletitle': opts.consoletitle,
@ -328,6 +295,7 @@ def _real_main(argv=None):
'writeannotations': opts.writeannotations,
'writeinfojson': opts.writeinfojson,
'writethumbnail': opts.writethumbnail,
'write_all_thumbnails': opts.write_all_thumbnails,
'writesubtitles': opts.writesubtitles,
'writeautomaticsub': opts.writeautomaticsub,
'allsubtitles': opts.allsubtitles,
@ -365,36 +333,20 @@ def _real_main(argv=None):
'youtube_include_dash_manifest': opts.youtube_include_dash_manifest,
'encoding': opts.encoding,
'exec_cmd': opts.exec_cmd,
'extract_flat': opts.extract_flat,
'merge_output_format': opts.merge_output_format,
'postprocessors': postprocessors,
'fixup': opts.fixup,
'source_address': opts.source_address,
'call_home': opts.call_home,
'sleep_interval': opts.sleep_interval,
'external_downloader': opts.external_downloader,
'list_thumbnails': opts.list_thumbnails,
'playlist_items': opts.playlist_items,
'xattr_set_filesize': opts.xattr_set_filesize,
}
with YoutubeDL(ydl_opts) as ydl:
ydl.print_debug_header()
ydl.add_default_info_extractors()
# PostProcessors
# Add the metadata pp first, the other pps will copy it
if opts.addmetadata:
ydl.add_post_processor(FFmpegMetadataPP())
if opts.extractaudio:
ydl.add_post_processor(FFmpegExtractAudioPP(preferredcodec=opts.audioformat, preferredquality=opts.audioquality, nopostoverwrites=opts.nopostoverwrites))
if opts.recodevideo:
ydl.add_post_processor(FFmpegVideoConvertor(preferedformat=opts.recodevideo))
if opts.embedsubtitles:
ydl.add_post_processor(FFmpegEmbedSubtitlePP(subtitlesformat=opts.subtitlesformat))
if opts.xattrs:
ydl.add_post_processor(XAttrMetadataPP())
if opts.embedthumbnail:
if not opts.addmetadata:
ydl.add_post_processor(FFmpegAudioFixPP())
ydl.add_post_processor(AtomicParsleyPP())
# Please keep ExecAfterDownload towards the bottom as it allows the user to modify the final file in any way.
# So if the user is able to remove the file before your postprocessor runs it might cause a few problems.
if opts.exec_cmd:
ydl.add_post_processor(ExecAfterDownloadPP(
verboseOutput=opts.verbose, exec_cmd=opts.exec_cmd))
# Update version
if opts.update_self:
update_self(ydl.to_screen, opts.verbose)
@ -405,18 +357,21 @@ def _real_main(argv=None):
# Maybe do nothing
if (len(all_urls) < 1) and (opts.load_info_filename is None):
if not (opts.update_self or opts.rm_cachedir):
parser.error(u'you must provide at least one URL')
else:
if opts.update_self or opts.rm_cachedir:
sys.exit()
ydl.warn_if_short_id(sys.argv[1:] if argv is None else argv)
parser.error(
'You must provide at least one URL.\n'
'Type youtube-dl --help to see a list of all options.')
try:
if opts.load_info_filename is not None:
retcode = ydl.download_with_info_file(opts.load_info_filename)
else:
retcode = ydl.download(all_urls)
except MaxDownloadsReached:
ydl.to_screen(u'--max-download limit reached, aborting.')
ydl.to_screen('--max-download limit reached, aborting.')
retcode = 101
sys.exit(retcode)
@ -428,6 +383,8 @@ def main(argv=None):
except DownloadError:
sys.exit(1)
except SameFileError:
sys.exit(u'ERROR: fixed output name but more than one file to download')
sys.exit('ERROR: fixed output name but more than one file to download')
except KeyboardInterrupt:
sys.exit(u'\nERROR: Interrupted by user')
sys.exit('\nERROR: Interrupted by user')
__all__ = ['main', 'YoutubeDL', 'gen_extractors', 'list_extractors']

View File

@ -1,4 +1,5 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Execute with
# $ python youtube_dl/__main__.py (2.6+)

View File

@ -1,3 +1,5 @@
from __future__ import unicode_literals
__all__ = ['aes_encrypt', 'key_expansion', 'aes_ctr_decrypt', 'aes_cbc_decrypt', 'aes_decrypt_text']
import base64
@ -7,6 +9,7 @@ from .utils import bytes_to_intlist, intlist_to_bytes
BLOCK_SIZE_BYTES = 16
def aes_ctr_decrypt(data, key, counter):
"""
Decrypt with aes in counter mode
@ -32,6 +35,7 @@ def aes_ctr_decrypt(data, key, counter):
return decrypted_data
def aes_cbc_decrypt(data, key, iv):
"""
Decrypt with aes in CBC mode
@ -57,6 +61,7 @@ def aes_cbc_decrypt(data, key, iv):
return decrypted_data
def key_expansion(data):
"""
Generate key schedule
@ -91,6 +96,7 @@ def key_expansion(data):
return data
def aes_encrypt(data, expanded_key):
"""
Encrypt one block with aes
@ -111,6 +117,7 @@ def aes_encrypt(data, expanded_key):
return data
def aes_decrypt(data, expanded_key):
"""
Decrypt one block with aes
@ -131,6 +138,7 @@ def aes_decrypt(data, expanded_key):
return data
def aes_decrypt_text(data, password, key_size_bytes):
"""
Decrypt text
@ -157,6 +165,7 @@ def aes_decrypt_text(data, password, key_size_bytes):
class Counter:
__value = nonce + [0] * (BLOCK_SIZE_BYTES - NONCE_LENGTH_BYTES)
def next_value(self):
temp = self.__value
self.__value = inc(self.__value)
@ -241,15 +250,19 @@ RIJNDAEL_LOG_TABLE = (0x00, 0x00, 0x19, 0x01, 0x32, 0x02, 0x1a, 0xc6, 0x4b, 0xc7
0x44, 0x11, 0x92, 0xd9, 0x23, 0x20, 0x2e, 0x89, 0xb4, 0x7c, 0xb8, 0x26, 0x77, 0x99, 0xe3, 0xa5,
0x67, 0x4a, 0xed, 0xde, 0xc5, 0x31, 0xfe, 0x18, 0x0d, 0x63, 0x8c, 0x80, 0xc0, 0xf7, 0x70, 0x07)
def sub_bytes(data):
return [SBOX[x] for x in data]
def sub_bytes_inv(data):
return [SBOX_INV[x] for x in data]
def rotate(data):
return data[1:] + [data[0]]
def key_schedule_core(data, rcon_iteration):
data = rotate(data)
data = sub_bytes(data)
@ -257,14 +270,17 @@ def key_schedule_core(data, rcon_iteration):
return data
def xor(data1, data2):
return [x ^ y for x, y in zip(data1, data2)]
def rijndael_mul(a, b):
if(a == 0 or b == 0):
return 0
return RIJNDAEL_EXP_TABLE[(RIJNDAEL_LOG_TABLE[a] + RIJNDAEL_LOG_TABLE[b]) % 0xFF]
def mix_column(data, matrix):
data_mixed = []
for row in range(4):
@ -275,6 +291,7 @@ def mix_column(data, matrix):
data_mixed.append(mixed)
return data_mixed
def mix_columns(data, matrix=MIX_COLUMN_MATRIX):
data_mixed = []
for i in range(4):
@ -282,9 +299,11 @@ def mix_columns(data, matrix=MIX_COLUMN_MATRIX):
data_mixed += mix_column(column, matrix)
return data_mixed
def mix_columns_inv(data):
return mix_columns(data, MIX_COLUMN_MATRIX_INV)
def shift_rows(data):
data_shifted = []
for column in range(4):
@ -292,6 +311,7 @@ def shift_rows(data):
data_shifted.append(data[((column + row) & 0b11) * 4 + row])
return data_shifted
def shift_rows_inv(data):
data_shifted = []
for column in range(4):
@ -299,6 +319,7 @@ def shift_rows_inv(data):
data_shifted.append(data[((column - row) & 0b11) * 4 + row])
return data_shifted
def inc(data):
data = data[:] # copy
for i in range(len(data) - 1, -1, -1):

View File

@ -8,9 +8,8 @@ import re
import shutil
import traceback
from .utils import (
write_json_file,
)
from .compat import compat_expanduser, compat_getenv
from .utils import write_json_file
class Cache(object):
@ -20,9 +19,9 @@ class Cache(object):
def _get_root_dir(self):
res = self._ydl.params.get('cachedir')
if res is None:
cache_root = os.environ.get('XDG_CACHE_HOME', '~/.cache')
cache_root = compat_getenv('XDG_CACHE_HOME', '~/.cache')
res = os.path.join(cache_root, 'youtube-dl')
return os.path.expanduser(res)
return compat_expanduser(res)
def _get_cache_fn(self, section, key, dtype):
assert re.match(r'^[a-zA-Z0-9_.-]+$', section), \

398
youtube_dl/compat.py Normal file
View File

@ -0,0 +1,398 @@
from __future__ import unicode_literals
import getpass
import optparse
import os
import re
import socket
import subprocess
import sys
try:
import urllib.request as compat_urllib_request
except ImportError: # Python 2
import urllib2 as compat_urllib_request
try:
import urllib.error as compat_urllib_error
except ImportError: # Python 2
import urllib2 as compat_urllib_error
try:
import urllib.parse as compat_urllib_parse
except ImportError: # Python 2
import urllib as compat_urllib_parse
try:
from urllib.parse import urlparse as compat_urllib_parse_urlparse
except ImportError: # Python 2
from urlparse import urlparse as compat_urllib_parse_urlparse
try:
import urllib.parse as compat_urlparse
except ImportError: # Python 2
import urlparse as compat_urlparse
try:
import http.cookiejar as compat_cookiejar
except ImportError: # Python 2
import cookielib as compat_cookiejar
try:
import html.entities as compat_html_entities
except ImportError: # Python 2
import htmlentitydefs as compat_html_entities
try:
import html.parser as compat_html_parser
except ImportError: # Python 2
import HTMLParser as compat_html_parser
try:
import http.client as compat_http_client
except ImportError: # Python 2
import httplib as compat_http_client
try:
from urllib.error import HTTPError as compat_HTTPError
except ImportError: # Python 2
from urllib2 import HTTPError as compat_HTTPError
try:
from urllib.request import urlretrieve as compat_urlretrieve
except ImportError: # Python 2
from urllib import urlretrieve as compat_urlretrieve
try:
from subprocess import DEVNULL
compat_subprocess_get_DEVNULL = lambda: DEVNULL
except ImportError:
compat_subprocess_get_DEVNULL = lambda: open(os.path.devnull, 'w')
try:
import http.server as compat_http_server
except ImportError:
import BaseHTTPServer as compat_http_server
try:
from urllib.parse import unquote as compat_urllib_parse_unquote
except ImportError:
def compat_urllib_parse_unquote(string, encoding='utf-8', errors='replace'):
if string == '':
return string
res = string.split('%')
if len(res) == 1:
return string
if encoding is None:
encoding = 'utf-8'
if errors is None:
errors = 'replace'
# pct_sequence: contiguous sequence of percent-encoded bytes, decoded
pct_sequence = b''
string = res[0]
for item in res[1:]:
try:
if not item:
raise ValueError
pct_sequence += item[:2].decode('hex')
rest = item[2:]
if not rest:
# This segment was just a single percent-encoded character.
# May be part of a sequence of code units, so delay decoding.
# (Stored in pct_sequence).
continue
except ValueError:
rest = '%' + item
# Encountered non-percent-encoded characters. Flush the current
# pct_sequence.
string += pct_sequence.decode(encoding, errors) + rest
pct_sequence = b''
if pct_sequence:
# Flush the final pct_sequence
string += pct_sequence.decode(encoding, errors)
return string
try:
compat_str = unicode # Python 2
except NameError:
compat_str = str
try:
compat_basestring = basestring # Python 2
except NameError:
compat_basestring = str
try:
compat_chr = unichr # Python 2
except NameError:
compat_chr = chr
try:
from xml.etree.ElementTree import ParseError as compat_xml_parse_error
except ImportError: # Python 2.6
from xml.parsers.expat import ExpatError as compat_xml_parse_error
try:
from urllib.parse import parse_qs as compat_parse_qs
except ImportError: # Python 2
# HACK: The following is the correct parse_qs implementation from cpython 3's stdlib.
# Python 2's version is apparently totally broken
def _parse_qsl(qs, keep_blank_values=False, strict_parsing=False,
encoding='utf-8', errors='replace'):
qs, _coerce_result = qs, compat_str
pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')]
r = []
for name_value in pairs:
if not name_value and not strict_parsing:
continue
nv = name_value.split('=', 1)
if len(nv) != 2:
if strict_parsing:
raise ValueError("bad query field: %r" % (name_value,))
# Handle case of a control-name with no equal sign
if keep_blank_values:
nv.append('')
else:
continue
if len(nv[1]) or keep_blank_values:
name = nv[0].replace('+', ' ')
name = compat_urllib_parse_unquote(
name, encoding=encoding, errors=errors)
name = _coerce_result(name)
value = nv[1].replace('+', ' ')
value = compat_urllib_parse_unquote(
value, encoding=encoding, errors=errors)
value = _coerce_result(value)
r.append((name, value))
return r
def compat_parse_qs(qs, keep_blank_values=False, strict_parsing=False,
encoding='utf-8', errors='replace'):
parsed_result = {}
pairs = _parse_qsl(qs, keep_blank_values, strict_parsing,
encoding=encoding, errors=errors)
for name, value in pairs:
if name in parsed_result:
parsed_result[name].append(value)
else:
parsed_result[name] = [value]
return parsed_result
try:
from shlex import quote as shlex_quote
except ImportError: # Python < 3.3
def shlex_quote(s):
if re.match(r'^[-_\w./]+$', s):
return s
else:
return "'" + s.replace("'", "'\"'\"'") + "'"
def compat_ord(c):
if type(c) is int:
return c
else:
return ord(c)
if sys.version_info >= (3, 0):
compat_getenv = os.getenv
compat_expanduser = os.path.expanduser
else:
# Environment variables should be decoded with filesystem encoding.
# Otherwise it will fail if any non-ASCII characters present (see #3854 #3217 #2918)
def compat_getenv(key, default=None):
from .utils import get_filesystem_encoding
env = os.getenv(key, default)
if env:
env = env.decode(get_filesystem_encoding())
return env
# HACK: The default implementations of os.path.expanduser from cpython do not decode
# environment variables with filesystem encoding. We will work around this by
# providing adjusted implementations.
# The following are os.path.expanduser implementations from cpython 2.7.8 stdlib
# for different platforms with correct environment variables decoding.
if os.name == 'posix':
def compat_expanduser(path):
"""Expand ~ and ~user constructions. If user or $HOME is unknown,
do nothing."""
if not path.startswith('~'):
return path
i = path.find('/', 1)
if i < 0:
i = len(path)
if i == 1:
if 'HOME' not in os.environ:
import pwd
userhome = pwd.getpwuid(os.getuid()).pw_dir
else:
userhome = compat_getenv('HOME')
else:
import pwd
try:
pwent = pwd.getpwnam(path[1:i])
except KeyError:
return path
userhome = pwent.pw_dir
userhome = userhome.rstrip('/')
return (userhome + path[i:]) or '/'
elif os.name == 'nt' or os.name == 'ce':
def compat_expanduser(path):
"""Expand ~ and ~user constructs.
If user or $HOME is unknown, do nothing."""
if path[:1] != '~':
return path
i, n = 1, len(path)
while i < n and path[i] not in '/\\':
i = i + 1
if 'HOME' in os.environ:
userhome = compat_getenv('HOME')
elif 'USERPROFILE' in os.environ:
userhome = compat_getenv('USERPROFILE')
elif 'HOMEPATH' not in os.environ:
return path
else:
try:
drive = compat_getenv('HOMEDRIVE')
except KeyError:
drive = ''
userhome = os.path.join(drive, compat_getenv('HOMEPATH'))
if i != 1: # ~user
userhome = os.path.join(os.path.dirname(userhome), path[1:i])
return userhome + path[i:]
else:
compat_expanduser = os.path.expanduser
if sys.version_info < (3, 0):
def compat_print(s):
from .utils import preferredencoding
print(s.encode(preferredencoding(), 'xmlcharrefreplace'))
else:
def compat_print(s):
assert isinstance(s, compat_str)
print(s)
try:
subprocess_check_output = subprocess.check_output
except AttributeError:
def subprocess_check_output(*args, **kwargs):
assert 'input' not in kwargs
p = subprocess.Popen(*args, stdout=subprocess.PIPE, **kwargs)
output, _ = p.communicate()
ret = p.poll()
if ret:
raise subprocess.CalledProcessError(ret, p.args, output=output)
return output
if sys.version_info < (3, 0) and sys.platform == 'win32':
def compat_getpass(prompt, *args, **kwargs):
if isinstance(prompt, compat_str):
from .utils import preferredencoding
prompt = prompt.encode(preferredencoding())
return getpass.getpass(prompt, *args, **kwargs)
else:
compat_getpass = getpass.getpass
# Old 2.6 and 2.7 releases require kwargs to be bytes
try:
def _testfunc(x):
pass
_testfunc(**{'x': 0})
except TypeError:
def compat_kwargs(kwargs):
return dict((bytes(k), v) for k, v in kwargs.items())
else:
compat_kwargs = lambda kwargs: kwargs
if sys.version_info < (2, 7):
def compat_socket_create_connection(address, timeout, source_address=None):
host, port = address
err = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socket.socket(af, socktype, proto)
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
sock.connect(sa)
return sock
except socket.error as _:
err = _
if sock is not None:
sock.close()
if err is not None:
raise err
else:
raise socket.error("getaddrinfo returns an empty list")
else:
compat_socket_create_connection = socket.create_connection
# Fix https://github.com/rg3/youtube-dl/issues/4223
# See http://bugs.python.org/issue9161 for what is broken
def workaround_optparse_bug9161():
op = optparse.OptionParser()
og = optparse.OptionGroup(op, 'foo')
try:
og.add_option('-t')
except TypeError:
real_add_option = optparse.OptionGroup.add_option
def _compat_add_option(self, *args, **kwargs):
enc = lambda v: (
v.encode('ascii', 'replace') if isinstance(v, compat_str)
else v)
bargs = [enc(a) for a in args]
bkwargs = dict(
(k, enc(v)) for k, v in kwargs.items())
return real_add_option(self, *bargs, **bkwargs)
optparse.OptionGroup.add_option = _compat_add_option
__all__ = [
'compat_HTTPError',
'compat_basestring',
'compat_chr',
'compat_cookiejar',
'compat_expanduser',
'compat_getenv',
'compat_getpass',
'compat_html_entities',
'compat_html_parser',
'compat_http_client',
'compat_http_server',
'compat_kwargs',
'compat_ord',
'compat_parse_qs',
'compat_print',
'compat_socket_create_connection',
'compat_str',
'compat_subprocess_get_DEVNULL',
'compat_urllib_error',
'compat_urllib_parse',
'compat_urllib_parse_unquote',
'compat_urllib_parse_urlparse',
'compat_urllib_request',
'compat_urlparse',
'compat_urlretrieve',
'compat_xml_parse_error',
'shlex_quote',
'subprocess_check_output',
'workaround_optparse_bug9161',
]

View File

@ -1,29 +1,43 @@
from __future__ import unicode_literals
from .common import FileDownloader
from .external import get_external_downloader
from .f4m import F4mFD
from .hls import HlsFD
from .hls import NativeHlsFD
from .http import HttpFD
from .mplayer import MplayerFD
from .rtmp import RtmpFD
from .f4m import F4mFD
from ..utils import (
determine_ext,
determine_protocol,
)
PROTOCOL_MAP = {
'rtmp': RtmpFD,
'm3u8_native': NativeHlsFD,
'm3u8': HlsFD,
'mms': MplayerFD,
'rtsp': MplayerFD,
'f4m': F4mFD,
}
def get_suitable_downloader(info_dict):
def get_suitable_downloader(info_dict, params={}):
"""Get the downloader class that can handle the info dict."""
url = info_dict['url']
protocol = info_dict.get('protocol')
protocol = determine_protocol(info_dict)
info_dict['protocol'] = protocol
if url.startswith('rtmp'):
return RtmpFD
if (protocol == 'm3u8') or (protocol is None and determine_ext(url) == 'm3u8'):
return HlsFD
if url.startswith('mms') or url.startswith('rtsp'):
return MplayerFD
if determine_ext(url) == 'f4m':
return F4mFD
else:
return HttpFD
external_downloader = params.get('external_downloader')
if external_downloader is not None:
ed = get_external_downloader(external_downloader)
if ed.supports(info_dict):
return ed
return PROTOCOL_MAP.get(protocol, HttpFD)
__all__ = [
'get_suitable_downloader',
'FileDownloader',
]

View File

@ -1,10 +1,12 @@
from __future__ import unicode_literals
import os
import re
import sys
import time
from ..compat import compat_str
from ..utils import (
compat_str,
encodeFilename,
format_bytes,
timeconvert,
@ -38,10 +40,13 @@ class FileDownloader(object):
test: Download only first bytes to test the downloader.
min_filesize: Skip files smaller than this size
max_filesize: Skip files larger than this size
xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
(experimenatal)
Subclasses of this one must re-define the real_download method.
"""
_TEST_FILE_SIZE = 10241
params = None
def __init__(self, ydl, params):
@ -77,6 +82,8 @@ class FileDownloader(object):
def calc_eta(start, now, total, current):
if total is None:
return None
if now is None:
now = time.time()
dif = now - start
if current == 0 or dif < 0.001: # One millisecond
return None
@ -143,29 +150,30 @@ class FileDownloader(object):
def report_error(self, *args, **kargs):
self.ydl.report_error(*args, **kargs)
def slow_down(self, start_time, byte_counter):
def slow_down(self, start_time, now, byte_counter):
"""Sleep if the download speed is over the rate limit."""
rate_limit = self.params.get('ratelimit', None)
if rate_limit is None or byte_counter == 0:
return
if now is None:
now = time.time()
elapsed = now - start_time
if elapsed <= 0.0:
return
speed = float(byte_counter) / elapsed
if speed > rate_limit:
time.sleep((byte_counter - rate_limit * (now - start_time)) / rate_limit)
time.sleep(max((byte_counter // rate_limit) - elapsed, 0))
def temp_name(self, filename):
"""Returns a temporary filename for the given filename."""
if self.params.get('nopart', False) or filename == u'-' or \
if self.params.get('nopart', False) or filename == '-' or \
(os.path.exists(encodeFilename(filename)) and not os.path.isfile(encodeFilename(filename))):
return filename
return filename + u'.part'
return filename + '.part'
def undo_temp_name(self, filename):
if filename.endswith(u'.part'):
return filename[:-len(u'.part')]
if filename.endswith('.part'):
return filename[:-len('.part')]
return filename
def try_rename(self, old_filename, new_filename):
@ -174,7 +182,7 @@ class FileDownloader(object):
return
os.rename(encodeFilename(old_filename), encodeFilename(new_filename))
except (IOError, OSError) as err:
self.report_error(u'unable to rename file: %s' % compat_str(err))
self.report_error('unable to rename file: %s' % compat_str(err))
def try_utime(self, filename, last_modified_hdr):
"""Try to set the last-modified time of the given file."""
@ -199,10 +207,10 @@ class FileDownloader(object):
def report_destination(self, filename):
"""Report destination filename."""
self.to_screen(u'[download] Destination: ' + filename)
self.to_screen('[download] Destination: ' + filename)
def _report_progress_status(self, msg, is_last_line=False):
fullmsg = u'[download] ' + msg
fullmsg = '[download] ' + msg
if self.params.get('progress_with_newline', False):
self.to_screen(fullmsg)
else:
@ -210,13 +218,13 @@ class FileDownloader(object):
prev_len = getattr(self, '_report_progress_prev_line_length',
0)
if prev_len > len(fullmsg):
fullmsg += u' ' * (prev_len - len(fullmsg))
fullmsg += ' ' * (prev_len - len(fullmsg))
self._report_progress_prev_line_length = len(fullmsg)
clear_line = u'\r'
clear_line = '\r'
else:
clear_line = (u'\r\x1b[K' if sys.stderr.isatty() else u'\r')
clear_line = ('\r\x1b[K' if sys.stderr.isatty() else '\r')
self.to_screen(clear_line + fullmsg, skip_eol=not is_last_line)
self.to_console_title(u'youtube-dl ' + msg)
self.to_console_title('youtube-dl ' + msg)
def report_progress(self, percent, data_len_str, speed, eta):
"""Report download progress."""
@ -232,7 +240,7 @@ class FileDownloader(object):
percent_str = 'Unknown %'
speed_str = self.format_speed(speed)
msg = (u'%s of %s at %s ETA %s' %
msg = ('%s of %s at %s ETA %s' %
(percent_str, data_len_str, speed_str, eta_str))
self._report_progress_status(msg)
@ -242,44 +250,56 @@ class FileDownloader(object):
downloaded_str = format_bytes(downloaded_data_len)
speed_str = self.format_speed(speed)
elapsed_str = FileDownloader.format_seconds(elapsed)
msg = u'%s at %s (%s)' % (downloaded_str, speed_str, elapsed_str)
msg = '%s at %s (%s)' % (downloaded_str, speed_str, elapsed_str)
self._report_progress_status(msg)
def report_finish(self, data_len_str, tot_time):
"""Report download finished."""
if self.params.get('noprogress', False):
self.to_screen(u'[download] Download completed')
self.to_screen('[download] Download completed')
else:
self._report_progress_status(
(u'100%% of %s in %s' %
('100%% of %s in %s' %
(data_len_str, self.format_seconds(tot_time))),
is_last_line=True)
def report_resuming_byte(self, resume_len):
"""Report attempt to resume at given byte."""
self.to_screen(u'[download] Resuming download at byte %s' % resume_len)
self.to_screen('[download] Resuming download at byte %s' % resume_len)
def report_retry(self, count, retries):
"""Report retry in case of HTTP error 5xx"""
self.to_screen(u'[download] Got server HTTP error. Retrying (attempt %d of %d)...' % (count, retries))
self.to_screen('[download] Got server HTTP error. Retrying (attempt %d of %d)...' % (count, retries))
def report_file_already_downloaded(self, file_name):
"""Report file has already been fully downloaded."""
try:
self.to_screen(u'[download] %s has already been downloaded' % file_name)
self.to_screen('[download] %s has already been downloaded' % file_name)
except UnicodeEncodeError:
self.to_screen(u'[download] The file has already been downloaded')
self.to_screen('[download] The file has already been downloaded')
def report_unable_to_resume(self):
"""Report it was impossible to resume download."""
self.to_screen(u'[download] Unable to resume')
self.to_screen('[download] Unable to resume')
def download(self, filename, info_dict):
"""Download to a filename using the info from info_dict
Return True on success and False otherwise
"""
nooverwrites_and_exists = (
self.params.get('nooverwrites', False)
and os.path.exists(encodeFilename(filename))
)
continuedl_and_exists = (
self.params.get('continuedl', False)
and os.path.isfile(encodeFilename(filename))
and not self.params.get('nopart', False)
)
# Check file already present
if self.params.get('continuedl', False) and os.path.isfile(encodeFilename(filename)) and not self.params.get('nopart', False):
if filename != '-' and nooverwrites_and_exists or continuedl_and_exists:
self.report_file_already_downloaded(filename)
self._hook_progress({
'filename': filename,
@ -288,30 +308,43 @@ class FileDownloader(object):
})
return True
sleep_interval = self.params.get('sleep_interval')
if sleep_interval:
self.to_screen('[download] Sleeping %s seconds...' % sleep_interval)
time.sleep(sleep_interval)
return self.real_download(filename, info_dict)
def real_download(self, filename, info_dict):
"""Real download process. Redefine in subclasses."""
raise NotImplementedError(u'This method must be implemented by subclasses')
raise NotImplementedError('This method must be implemented by subclasses')
def _hook_progress(self, status):
for ph in self._progress_hooks:
ph(status)
def add_progress_hook(self, ph):
""" ph gets called on download progress, with a dictionary with the entries
* filename: The final filename
* status: One of "downloading" and "finished"
It can also have some of the following entries:
* downloaded_bytes: Bytes on disks
* total_bytes: Total bytes, None if unknown
* tmpfilename: The filename we're currently writing to
* eta: The estimated time in seconds, None if unknown
* speed: The download speed in bytes/second, None if unknown
Hooks are guaranteed to be called at least once (with status "finished")
if the download is successful.
"""
# See YoutubeDl.py (search for progress_hooks) for a description of
# this interface
self._progress_hooks.append(ph)
def _debug_cmd(self, args, subprocess_encoding, exe=None):
if not self.params.get('verbose', False):
return
if exe is None:
exe = os.path.basename(args[0])
if subprocess_encoding:
str_args = [
a.decode(subprocess_encoding) if isinstance(a, bytes) else a
for a in args]
else:
str_args = args
try:
import pipes
shell_quote = lambda args: ' '.join(map(pipes.quote, str_args))
except ImportError:
shell_quote = repr
self.to_screen('[debug] %s command line: %s' % (
exe, shell_quote(str_args)))

View File

@ -0,0 +1,126 @@
from __future__ import unicode_literals
import os.path
import subprocess
import sys
from .common import FileDownloader
from ..utils import (
encodeFilename,
)
class ExternalFD(FileDownloader):
def real_download(self, filename, info_dict):
self.report_destination(filename)
tmpfilename = self.temp_name(filename)
retval = self._call_downloader(tmpfilename, info_dict)
if retval == 0:
fsize = os.path.getsize(encodeFilename(tmpfilename))
self.to_screen('\r[%s] Downloaded %s bytes' % (self.get_basename(), fsize))
self.try_rename(tmpfilename, filename)
self._hook_progress({
'downloaded_bytes': fsize,
'total_bytes': fsize,
'filename': filename,
'status': 'finished',
})
return True
else:
self.to_stderr('\n')
self.report_error('%s exited with code %d' % (
self.get_basename(), retval))
return False
@classmethod
def get_basename(cls):
return cls.__name__[:-2].lower()
@property
def exe(self):
return self.params.get('external_downloader')
@classmethod
def supports(cls, info_dict):
return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps')
def _source_address(self, command_option):
source_address = self.params.get('source_address')
if source_address is None:
return []
return [command_option, source_address]
def _call_downloader(self, tmpfilename, info_dict):
""" Either overwrite this or implement _make_cmd """
cmd = self._make_cmd(tmpfilename, info_dict)
if sys.platform == 'win32' and sys.version_info < (3, 0):
# Windows subprocess module does not actually support Unicode
# on Python 2.x
# See http://stackoverflow.com/a/9951851/35070
subprocess_encoding = sys.getfilesystemencoding()
cmd = [a.encode(subprocess_encoding, 'ignore') for a in cmd]
else:
subprocess_encoding = None
self._debug_cmd(cmd, subprocess_encoding)
p = subprocess.Popen(
cmd, stderr=subprocess.PIPE)
_, stderr = p.communicate()
if p.returncode != 0:
self.to_stderr(stderr)
return p.returncode
class CurlFD(ExternalFD):
def _make_cmd(self, tmpfilename, info_dict):
cmd = [self.exe, '-o', tmpfilename]
for key, val in info_dict['http_headers'].items():
cmd += ['--header', '%s: %s' % (key, val)]
cmd += self._source_address('--interface')
cmd += ['--', info_dict['url']]
return cmd
class WgetFD(ExternalFD):
def _make_cmd(self, tmpfilename, info_dict):
cmd = [self.exe, '-O', tmpfilename, '-nv', '--no-cookies']
for key, val in info_dict['http_headers'].items():
cmd += ['--header', '%s: %s' % (key, val)]
cmd += self._source_address('--bind-address')
cmd += ['--', info_dict['url']]
return cmd
class Aria2cFD(ExternalFD):
def _make_cmd(self, tmpfilename, info_dict):
cmd = [
self.exe, '-c',
'--min-split-size', '1M', '--max-connection-per-server', '4']
dn = os.path.dirname(tmpfilename)
if dn:
cmd += ['--dir', dn]
cmd += ['--out', os.path.basename(tmpfilename)]
for key, val in info_dict['http_headers'].items():
cmd += ['--header', '%s: %s' % (key, val)]
cmd += self._source_address('--interface')
cmd += ['--', info_dict['url']]
return cmd
_BY_NAME = dict(
(klass.get_basename(), klass)
for name, klass in globals().items()
if name.endswith('FD') and name != 'ExternalFD'
)
def list_external_downloaders():
return sorted(_BY_NAME.keys())
def get_external_downloader(external_downloader):
""" Given the name of the executable, see whether we support the given
downloader . """
bn = os.path.basename(external_downloader)
return _BY_NAME[bn]

View File

@ -9,13 +9,16 @@ import xml.etree.ElementTree as etree
from .common import FileDownloader
from .http import HttpFD
from ..compat import (
compat_urlparse,
)
from ..utils import (
struct_pack,
struct_unpack,
compat_urlparse,
format_bytes,
encodeFilename,
sanitize_open,
xpath_text,
)
@ -174,34 +177,43 @@ def build_fragments_list(boot_info):
""" Return a list of (segment, fragment) for each fragment in the video """
res = []
segment_run_table = boot_info['segments'][0]
# I've only found videos with one segment
segment_run_entry = segment_run_table['segment_run'][0]
n_frags = segment_run_entry[1]
fragment_run_entry_table = boot_info['fragments'][0]['fragments']
first_frag_number = fragment_run_entry_table[0]['first']
for (i, frag_number) in zip(range(1, n_frags+1), itertools.count(first_frag_number)):
res.append((1, frag_number))
fragments_counter = itertools.count(first_frag_number)
for segment, fragments_count in segment_run_table['segment_run']:
for _ in range(fragments_count):
res.append((segment, next(fragments_counter)))
return res
def write_flv_header(stream, metadata):
"""Writes the FLV header and the metadata to stream"""
def write_unsigned_int(stream, val):
stream.write(struct_pack('!I', val))
def write_unsigned_int_24(stream, val):
stream.write(struct_pack('!I', val)[1:])
def write_flv_header(stream):
"""Writes the FLV header to stream"""
# FLV header
stream.write(b'FLV\x01')
stream.write(b'\x05')
stream.write(b'\x00\x00\x00\x09')
# FLV File body
stream.write(b'\x00\x00\x00\x00')
# FLVTAG
# Script data
stream.write(b'\x12')
# Size of the metadata with 3 bytes
stream.write(struct_pack('!L', len(metadata))[1:])
def write_metadata_tag(stream, metadata):
"""Writes optional metadata tag to stream"""
SCRIPT_TAG = b'\x12'
FLV_TAG_HEADER_LEN = 11
if metadata:
stream.write(SCRIPT_TAG)
write_unsigned_int_24(stream, len(metadata))
stream.write(b'\x00\x00\x00\x00\x00\x00\x00')
stream.write(metadata)
# Magic numbers extracted from the output files produced by AdobeHDS.php
#(https://github.com/K-S-V/Scripts)
stream.write(b'\x00\x00\x01\x73')
write_unsigned_int(stream, FLV_TAG_HEADER_LEN + len(metadata))
def _add_ns(prop):
@ -218,22 +230,43 @@ class F4mFD(FileDownloader):
A downloader for f4m manifests or AdobeHDS.
"""
def _get_unencrypted_media(self, doc):
media = doc.findall(_add_ns('media'))
if not media:
self.report_error('No media found')
for e in (doc.findall(_add_ns('drmAdditionalHeader')) +
doc.findall(_add_ns('drmAdditionalHeaderSet'))):
# If id attribute is missing it's valid for all media nodes
# without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
if 'id' not in e.attrib:
self.report_error('Missing ID in f4m DRM')
media = list(filter(lambda e: 'drmAdditionalHeaderId' not in e.attrib and
'drmAdditionalHeaderSetId' not in e.attrib,
media))
if not media:
self.report_error('Unsupported DRM')
return media
def real_download(self, filename, info_dict):
man_url = info_dict['url']
requested_bitrate = info_dict.get('tbr')
self.to_screen('[download] Downloading f4m manifest')
manifest = self.ydl.urlopen(man_url).read()
self.report_destination(filename)
http_dl = HttpQuietDownloader(self.ydl,
http_dl = HttpQuietDownloader(
self.ydl,
{
'continuedl': True,
'quiet': True,
'noprogress': True,
'ratelimit': self.params.get('ratelimit', None),
'test': self.params.get('test', False),
})
}
)
doc = etree.fromstring(manifest)
formats = [(int(f.attrib.get('bitrate', -1)), f) for f in doc.findall(_add_ns('media'))]
formats = [(int(f.attrib.get('bitrate', -1)), f)
for f in self._get_unencrypted_media(doc)]
if requested_bitrate is None:
# get the best format
formats = sorted(formats, key=lambda f: f[0])
@ -243,18 +276,32 @@ class F4mFD(FileDownloader):
lambda f: int(f[0]) == requested_bitrate, formats))[0]
base_url = compat_urlparse.urljoin(man_url, media.attrib['url'])
bootstrap = base64.b64decode(doc.find(_add_ns('bootstrapInfo')).text)
metadata = base64.b64decode(media.find(_add_ns('metadata')).text)
bootstrap_node = doc.find(_add_ns('bootstrapInfo'))
if bootstrap_node.text is None:
bootstrap_url = compat_urlparse.urljoin(
base_url, bootstrap_node.attrib['url'])
bootstrap = self.ydl.urlopen(bootstrap_url).read()
else:
bootstrap = base64.b64decode(bootstrap_node.text)
metadata_node = media.find(_add_ns('metadata'))
if metadata_node is not None:
metadata = base64.b64decode(metadata_node.text)
else:
metadata = None
boot_info = read_bootstrap_info(bootstrap)
fragments_list = build_fragments_list(boot_info)
if self.params.get('test', False):
# We only download the first fragment
fragments_list = fragments_list[:1]
total_frags = len(fragments_list)
# For some akamai manifests we'll need to add a query to the fragment url
akamai_pv = xpath_text(doc, _add_ns('pv-2.0'))
tmpfilename = self.temp_name(filename)
(dest_stream, tmpfilename) = sanitize_open(tmpfilename, 'wb')
write_flv_header(dest_stream, metadata)
write_flv_header(dest_stream)
write_metadata_tag(dest_stream, metadata)
# This dict stores the download progress, it's updated by the progress
# hook
@ -290,6 +337,8 @@ class F4mFD(FileDownloader):
for (seg_i, frag_i) in fragments_list:
name = 'Seg%d-Frag%d' % (seg_i, frag_i)
url = base_url + name
if akamai_pv:
url += '?' + akamai_pv.strip(';')
frag_filename = '%s-%s' % (tmpfilename, name)
success = http_dl.download(frag_filename, {'url': url})
if not success:

View File

@ -1,9 +1,17 @@
from __future__ import unicode_literals
import os
import re
import subprocess
from ..postprocessor.ffmpeg import FFmpegPostProcessor
from .common import FileDownloader
from ..compat import (
compat_urlparse,
compat_urllib_request,
)
from ..utils import (
check_executable,
encodeArgument,
encodeFilename,
)
@ -14,23 +22,22 @@ class HlsFD(FileDownloader):
self.report_destination(filename)
tmpfilename = self.temp_name(filename)
args = [
'-y', '-i', url, '-f', 'mp4', '-c', 'copy',
'-bsf:a', 'aac_adtstoasc',
encodeFilename(tmpfilename, for_subprocess=True)]
for program in ['avconv', 'ffmpeg']:
if check_executable(program, ['-version']):
break
else:
self.report_error(u'm3u8 download detected but ffmpeg or avconv could not be found. Please install one.')
ffpp = FFmpegPostProcessor(downloader=self)
program = ffpp._executable
if program is None:
self.report_error('m3u8 download detected but ffmpeg or avconv could not be found. Please install one.')
return False
cmd = [program] + args
ffpp.check_version()
retval = subprocess.call(cmd)
args = [
encodeArgument(opt)
for opt in (program, '-y', '-i', url, '-f', 'mp4', '-c', 'copy', '-bsf:a', 'aac_adtstoasc')]
args.append(encodeFilename(tmpfilename, True))
retval = subprocess.call(args)
if retval == 0:
fsize = os.path.getsize(encodeFilename(tmpfilename))
self.to_screen(u'\r[%s] %s bytes' % (cmd[0], fsize))
self.to_screen('\r[%s] %s bytes' % (args[0], fsize))
self.try_rename(tmpfilename, filename)
self._hook_progress({
'downloaded_bytes': fsize,
@ -40,6 +47,59 @@ class HlsFD(FileDownloader):
})
return True
else:
self.to_stderr(u"\n")
self.report_error(u'%s exited with code %d' % (program, retval))
self.to_stderr('\n')
self.report_error('%s exited with code %d' % (program, retval))
return False
class NativeHlsFD(FileDownloader):
""" A more limited implementation that does not require ffmpeg """
def real_download(self, filename, info_dict):
url = info_dict['url']
self.report_destination(filename)
tmpfilename = self.temp_name(filename)
self.to_screen(
'[hlsnative] %s: Downloading m3u8 manifest' % info_dict['id'])
data = self.ydl.urlopen(url).read()
s = data.decode('utf-8', 'ignore')
segment_urls = []
for line in s.splitlines():
line = line.strip()
if line and not line.startswith('#'):
segment_url = (
line
if re.match(r'^https?://', line)
else compat_urlparse.urljoin(url, line))
segment_urls.append(segment_url)
is_test = self.params.get('test', False)
remaining_bytes = self._TEST_FILE_SIZE if is_test else None
byte_counter = 0
with open(tmpfilename, 'wb') as outf:
for i, segurl in enumerate(segment_urls):
self.to_screen(
'[hlsnative] %s: Downloading segment %d / %d' %
(info_dict['id'], i + 1, len(segment_urls)))
seg_req = compat_urllib_request.Request(segurl)
if remaining_bytes is not None:
seg_req.add_header('Range', 'bytes=0-%d' % (remaining_bytes - 1))
segment = self.ydl.urlopen(seg_req).read()
if remaining_bytes is not None:
segment = segment[:remaining_bytes]
remaining_bytes -= len(segment)
outf.write(segment)
byte_counter += len(segment)
if remaining_bytes is not None and remaining_bytes <= 0:
break
self._hook_progress({
'downloaded_bytes': byte_counter,
'total_bytes': byte_counter,
'filename': filename,
'status': 'finished',
})
self.try_rename(tmpfilename, filename)
return True

View File

@ -1,12 +1,18 @@
from __future__ import unicode_literals
import os
import time
from socket import error as SocketError
import errno
from .common import FileDownloader
from ..utils import (
from ..compat import (
compat_urllib_request,
compat_urllib_error,
)
from ..utils import (
ContentTooShortError,
encodeFilename,
sanitize_open,
format_bytes,
@ -14,8 +20,6 @@ from ..utils import (
class HttpFD(FileDownloader):
_TEST_FILE_SIZE = 10241
def real_download(self, filename, info_dict):
url = info_dict['url']
tmpfilename = self.temp_name(filename)
@ -23,10 +27,6 @@ class HttpFD(FileDownloader):
# Do not include the Accept-Encoding header
headers = {'Youtubedl-no-compression': 'True'}
if 'user_agent' in info_dict:
headers['Youtubedl-user-agent'] = info_dict['user_agent']
if 'http_referer' in info_dict:
headers['Referer'] = info_dict['http_referer']
add_headers = info_dict.get('http_headers')
if add_headers:
headers.update(add_headers)
@ -102,13 +102,18 @@ class HttpFD(FileDownloader):
resume_len = 0
open_mode = 'wb'
break
except SocketError as e:
if e.errno != errno.ECONNRESET:
# Connection reset is no problem, just retry
raise
# Retry
count += 1
if count <= retries:
self.report_retry(count, retries)
if count > retries:
self.report_error(u'giving up after %s retries' % retries)
self.report_error('giving up after %s retries' % retries)
return False
data_len = data.info().get('Content-length', None)
@ -126,26 +131,31 @@ class HttpFD(FileDownloader):
min_data_len = self.params.get("min_filesize", None)
max_data_len = self.params.get("max_filesize", None)
if min_data_len is not None and data_len < min_data_len:
self.to_screen(u'\r[download] File is smaller than min-filesize (%s bytes < %s bytes). Aborting.' % (data_len, min_data_len))
self.to_screen('\r[download] File is smaller than min-filesize (%s bytes < %s bytes). Aborting.' % (data_len, min_data_len))
return False
if max_data_len is not None and data_len > max_data_len:
self.to_screen(u'\r[download] File is larger than max-filesize (%s bytes > %s bytes). Aborting.' % (data_len, max_data_len))
self.to_screen('\r[download] File is larger than max-filesize (%s bytes > %s bytes). Aborting.' % (data_len, max_data_len))
return False
data_len_str = format_bytes(data_len)
byte_counter = 0 + resume_len
block_size = self.params.get('buffersize', 1024)
start = time.time()
# measure time over whole while-loop, so slow_down() and best_block_size() work together properly
now = None # needed for slow_down() in the first loop run
before = start # start measuring
while True:
# Download and write
before = time.time()
data_block = data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
after = time.time()
if len(data_block) == 0:
break
byte_counter += len(data_block)
# Open file just in time
# exit loop when download is finished
if len(data_block) == 0:
break
# Open destination file just in time
if stream is None:
try:
(stream, tmpfilename) = sanitize_open(tmpfilename, open_mode)
@ -153,19 +163,38 @@ class HttpFD(FileDownloader):
filename = self.undo_temp_name(tmpfilename)
self.report_destination(filename)
except (OSError, IOError) as err:
self.report_error(u'unable to open for writing: %s' % str(err))
self.report_error('unable to open for writing: %s' % str(err))
return False
if self.params.get('xattr_set_filesize', False) and data_len is not None:
try:
import xattr
xattr.setxattr(tmpfilename, 'user.ytdl.filesize', str(data_len))
except(OSError, IOError, ImportError) as err:
self.report_error('unable to set filesize xattr: %s' % str(err))
try:
stream.write(data_block)
except (IOError, OSError) as err:
self.to_stderr(u"\n")
self.report_error(u'unable to write data: %s' % str(err))
self.to_stderr('\n')
self.report_error('unable to write data: %s' % str(err))
return False
# Apply rate limit
self.slow_down(start, now, byte_counter - resume_len)
# end measuring of one loop run
now = time.time()
after = now
# Adjust block size
if not self.params.get('noresizebuffer', False):
block_size = self.best_block_size(after - before, len(data_block))
before = after
# Progress message
speed = self.calc_speed(start, time.time(), byte_counter - resume_len)
speed = self.calc_speed(start, now, byte_counter - resume_len)
if data_len is None:
eta = percent = None
else:
@ -186,14 +215,11 @@ class HttpFD(FileDownloader):
if is_test and byte_counter == data_len:
break
# Apply rate limit
self.slow_down(start, byte_counter - resume_len)
if stream is None:
self.to_stderr(u"\n")
self.report_error(u'Did not get any data blocks')
self.to_stderr('\n')
self.report_error('Did not get any data blocks')
return False
if tmpfilename != u'-':
if tmpfilename != '-':
stream.close()
self.report_finish(data_len_str, (time.time() - start))
if data_len is not None and byte_counter != data_len:

View File

@ -1,8 +1,11 @@
from __future__ import unicode_literals
import os
import subprocess
from .common import FileDownloader
from ..utils import (
check_executable,
encodeFilename,
)
@ -13,19 +16,19 @@ class MplayerFD(FileDownloader):
self.report_destination(filename)
tmpfilename = self.temp_name(filename)
args = ['mplayer', '-really-quiet', '-vo', 'null', '-vc', 'dummy', '-dumpstream', '-dumpfile', tmpfilename, url]
args = [
'mplayer', '-really-quiet', '-vo', 'null', '-vc', 'dummy',
'-dumpstream', '-dumpfile', tmpfilename, url]
# Check for mplayer first
try:
subprocess.call(['mplayer', '-h'], stdout=(open(os.path.devnull, 'w')), stderr=subprocess.STDOUT)
except (OSError, IOError):
self.report_error(u'MMS or RTSP download detected but "%s" could not be run' % args[0])
if not check_executable('mplayer', ['-h']):
self.report_error('MMS or RTSP download detected but "%s" could not be run' % args[0])
return False
# Download using mplayer.
retval = subprocess.call(args)
if retval == 0:
fsize = os.path.getsize(encodeFilename(tmpfilename))
self.to_screen(u'\r[%s] %s bytes' % (args[0], fsize))
self.to_screen('\r[%s] %s bytes' % (args[0], fsize))
self.try_rename(tmpfilename, filename)
self._hook_progress({
'downloaded_bytes': fsize,
@ -35,6 +38,6 @@ class MplayerFD(FileDownloader):
})
return True
else:
self.to_stderr(u"\n")
self.report_error(u'mplayer exited with code %d' % retval)
self.to_stderr('\n')
self.report_error('mplayer exited with code %d' % retval)
return False

View File

@ -7,14 +7,20 @@ import sys
import time
from .common import FileDownloader
from ..compat import compat_str
from ..utils import (
check_executable,
compat_str,
encodeFilename,
format_bytes,
get_exe_version,
)
def rtmpdump_version():
return get_exe_version(
'rtmpdump', ['--help'], r'(?i)RTMPDump\s*v?([0-9a-zA-Z._-]+)')
class RtmpFD(FileDownloader):
def real_download(self, filename, info_dict):
def run_rtmpdump(args):
@ -98,6 +104,9 @@ class RtmpFD(FileDownloader):
live = info_dict.get('rtmp_live', False)
conn = info_dict.get('rtmp_conn', None)
protocol = info_dict.get('rtmp_protocol', None)
real_time = info_dict.get('rtmp_real_time', False)
no_resume = info_dict.get('no_resume', False)
continue_dl = info_dict.get('continuedl', False)
self.report_destination(filename)
tmpfilename = self.temp_name(filename)
@ -135,7 +144,14 @@ class RtmpFD(FileDownloader):
basic_args += ['--conn', conn]
if protocol is not None:
basic_args += ['--protocol', protocol]
args = basic_args + [[], ['--resume', '--skip', '1']][not live and self.params.get('continuedl', False)]
if real_time:
basic_args += ['--realtime']
args = basic_args
if not no_resume and continue_dl and not live:
args += ['--resume']
if not live and continue_dl:
args += ['--skip', '1']
if sys.platform == 'win32' and sys.version_info < (3, 0):
# Windows subprocess module does not actually support Unicode
@ -146,19 +162,7 @@ class RtmpFD(FileDownloader):
else:
subprocess_encoding = None
if self.params.get('verbose', False):
if subprocess_encoding:
str_args = [
a.decode(subprocess_encoding) if isinstance(a, bytes) else a
for a in args]
else:
str_args = args
try:
import pipes
shell_quote = lambda args: ' '.join(map(pipes.quote, str_args))
except ImportError:
shell_quote = repr
self.to_screen('[debug] rtmpdump command line: ' + shell_quote(str_args))
self._debug_cmd(args, subprocess_encoding, exe='rtmpdump')
RD_SUCCESS = 0
RD_FAILED = 1

View File

@ -1,8 +1,14 @@
from __future__ import unicode_literals
from .abc import ABCIE
from .abc7news import Abc7NewsIE
from .academicearth import AcademicEarthCourseIE
from .addanime import AddAnimeIE
from .adobetv import AdobeTVIE
from .adultswim import AdultSwimIE
from .aftonbladet import AftonbladetIE
from .aljazeera import AlJazeeraIE
from .alphaporno import AlphaPornoIE
from .anitube import AnitubeIE
from .anysex import AnySexIE
from .aol import AolIE
@ -20,18 +26,26 @@ from .arte import (
ArteTVDDCIE,
ArteTVEmbedIE,
)
from .auengine import AUEngineIE
from .atresplayer import AtresPlayerIE
from .atttechchannel import ATTTechChannelIE
from .audiomack import AudiomackIE, AudiomackAlbumIE
from .azubu import AzubuIE
from .bambuser import BambuserIE, BambuserChannelIE
from .bandcamp import BandcampIE, BandcampAlbumIE
from .bbccouk import BBCCoUkIE
from .beeg import BeegIE
from .behindkink import BehindKinkIE
from .bet import BetIE
from .bild import BildIE
from .bilibili import BiliBiliIE
from .blinkx import BlinkxIE
from .bliptv import BlipTVIE, BlipTVUserIE
from .bloomberg import BloombergIE
from .bpb import BpbIE
from .br import BRIE
from .breakcom import BreakIE
from .brightcove import BrightcoveIE
from .buzzfeed import BuzzFeedIE
from .byutv import BYUtvIE
from .c56 import C56IE
from .canal13cl import Canal13clIE
@ -42,7 +56,7 @@ from .cbsnews import CBSNewsIE
from .ceskatelevize import CeskaTelevizeIE
from .channel9 import Channel9IE
from .chilloutzone import ChilloutzoneIE
from .cinemassacre import CinemassacreIE
from .cinchcast import CinchcastIE
from .clipfish import ClipfishIE
from .cliphunter import CliphunterIE
from .clipsyndicate import ClipsyndicateIE
@ -53,15 +67,22 @@ from .cnet import CNETIE
from .cnn import (
CNNIE,
CNNBlogsIE,
CNNArticleIE,
)
from .collegehumor import CollegeHumorIE
from .collegerama import CollegeRamaIE
from .comedycentral import ComedyCentralIE, ComedyCentralShowsIE
from .comcarcoff import ComCarCoffIE
from .commonmistakes import CommonMistakesIE
from .condenast import CondeNastIE
from .cracked import CrackedIE
from .criterion import CriterionIE
from .crunchyroll import CrunchyrollIE
from .crunchyroll import (
CrunchyrollIE,
CrunchyrollShowPlaylistIE
)
from .cspan import CSpanIE
from .d8 import D8IE
from .ctsnews import CtsNewsIE
from .dailymotion import (
DailymotionIE,
DailymotionPlaylistIE,
@ -69,20 +90,25 @@ from .dailymotion import (
)
from .daum import DaumIE
from .dbtv import DBTVIE
from .dctp import DctpTvIE
from .deezer import DeezerPlaylistIE
from .dfb import DFBIE
from .dotsub import DotsubIE
from .dreisat import DreiSatIE
from .drbonanza import DRBonanzaIE
from .drtuber import DrTuberIE
from .drtv import DRTVIE
from .dvtv import DVTVIE
from .dump import DumpIE
from .defense import DefenseGouvFrIE
from .discovery import DiscoveryIE
from .divxstage import DivxStageIE
from .dropbox import DropboxIE
from .ebaumsworld import EbaumsWorldIE
from .echomsk import EchoMskIE
from .ehow import EHowIE
from .eighttracks import EightTracksIE
from .einthusan import EinthusanIE
from .eitb import EitbIE
from .ellentv import (
EllenTVIE,
@ -92,6 +118,7 @@ from .elpais import ElPaisIE
from .empflix import EMPFlixIE
from .engadget import EngadgetIE
from .eporner import EpornerIE
from .eroprofile import EroProfileIE
from .escapist import EscapistIE
from .everyonesmixtape import EveryonesMixtapeIE
from .exfm import ExfmIE
@ -109,7 +136,10 @@ from .fktv import (
FKTVPosteckeIE,
)
from .flickr import FlickrIE
from .folketinget import FolketingetIE
from .fourtube import FourTubeIE
from .foxgay import FoxgayIE
from .foxnews import FoxNewsIE
from .franceculture import FranceCultureIE
from .franceinter import FranceInterIE
from .francetv import (
@ -121,6 +151,7 @@ from .francetv import (
)
from .freesound import FreesoundIE
from .freespeech import FreespeechIE
from .freevideo import FreeVideoIE
from .funnyordie import FunnyOrDieIE
from .gamekings import GamekingsIE
from .gameone import (
@ -132,15 +163,27 @@ from .gamestar import GameStarIE
from .gametrailers import GametrailersIE
from .gdcvault import GDCVaultIE
from .generic import GenericIE
from .giantbomb import GiantBombIE
from .giga import GigaIE
from .glide import GlideIE
from .globo import GloboIE
from .godtube import GodTubeIE
from .goldenmoustache import GoldenMoustacheIE
from .golem import GolemIE
from .googleplus import GooglePlusIE
from .googlesearch import GoogleSearchIE
from .gorillavid import GorillaVidIE
from .goshgay import GoshgayIE
from .grooveshark import GroovesharkIE
from .groupon import GrouponIE
from .hark import HarkIE
from .hearthisat import HearThisAtIE
from .heise import HeiseIE
from .hellporno import HellPornoIE
from .helsinki import HelsinkiIE
from .hentaistigma import HentaiStigmaIE
from .historicfilms import HistoricFilmsIE
from .hitbox import HitboxIE, HitboxLiveIE
from .hornbunny import HornBunnyIE
from .hostingbulk import HostingBulkIE
from .hotnewhiphop import HotNewHipHopIE
@ -168,9 +211,9 @@ from .jadorecettepub import JadoreCettePubIE
from .jeuxvideo import JeuxVideoIE
from .jove import JoveIE
from .jukebox import JukeboxIE
from .justintv import JustinTVIE
from .jpopsukitv import JpopsukiIE
from .kankan import KankanIE
from .karaoketv import KaraoketvIE
from .keezmovies import KeezMoviesIE
from .khanacademy import KhanAcademyIE
from .kickstarter import KickStarterIE
@ -179,6 +222,7 @@ from .kontrtube import KontrTubeIE
from .krasview import KrasViewIE
from .ku6 import Ku6IE
from .la7 import LA7IE
from .laola1tv import Laola1TvIE
from .lifenews import LifeNewsIE
from .liveleak import LiveLeakIE
from .livestream import (
@ -186,6 +230,8 @@ from .livestream import (
LivestreamOriginalIE,
LivestreamShortenerIE,
)
from .lnkgo import LnkGoIE
from .lrt import LRTIE
from .lynda import (
LyndaIE,
LyndaCourseIE
@ -197,6 +243,8 @@ from .malemotion import MalemotionIE
from .mdr import MDRIE
from .metacafe import MetacafeIE
from .metacritic import MetacriticIE
from .mgoon import MgoonIE
from .minhateca import MinhatecaIE
from .ministrygrid import MinistryGridIE
from .mit import TechTVMITIE, MITIE, OCWMITIE
from .mitele import MiTeleIE
@ -206,6 +254,7 @@ from .mpora import MporaIE
from .moevideo import MoeVideoIE
from .mofosex import MofosexIE
from .mojvideo import MojvideoIE
from .moniker import MonikerIE
from .mooshare import MooshareIE
from .morningstar import MorningstarIE
from .motherless import MotherlessIE
@ -218,12 +267,14 @@ from .mtv import (
MTVServicesEmbeddedIE,
MTVIggyIE,
)
from .muenchentv import MuenchenTVIE
from .musicplayon import MusicPlayOnIE
from .musicvault import MusicVaultIE
from .muzu import MuzuTVIE
from .myspace import MySpaceIE
from .myspace import MySpaceIE, MySpaceAlbumIE
from .myspass import MySpassIE
from .myvideo import MyVideoIE
from .myvidster import MyVidsterIE
from .naver import NaverIE
from .nba import NBAIE
from .nbc import (
@ -232,11 +283,24 @@ from .nbc import (
)
from .ndr import NDRIE
from .ndtv import NDTVIE
from .netzkino import NetzkinoIE
from .nerdcubed import NerdCubedFeedIE
from .newgrounds import NewgroundsIE
from .newstube import NewstubeIE
from .nextmedia import (
NextMediaIE,
NextMediaActionNewsIE,
AppleDailyRealtimeNewsIE,
AppleDailyAnimationNewsIE
)
from .nfb import NFBIE
from .nhl import NHLIE, NHLVideocenterIE
from .niconico import NiconicoIE
from .nfl import NFLIE
from .nhl import (
NHLIE,
NHLNewsIE,
NHLVideocenterIE,
)
from .niconico import NiconicoIE, NiconicoPlaylistIE
from .ninegag import NineGagIE
from .noco import NocoIE
from .normalboots import NormalbootsIE
@ -244,15 +308,22 @@ from .nosvideo import NosVideoIE
from .novamov import NovaMovIE
from .nowness import NownessIE
from .nowvideo import NowVideoIE
from .npo import NPOIE
from .npo import (
NPOIE,
NPOLiveIE,
TegenlichtVproIE,
)
from .nrk import (
NRKIE,
NRKTVIE,
)
from .ntv import NTVIE
from .ntvde import NTVDeIE
from .ntvru import NTVRuIE
from .nytimes import NYTimesIE
from .nuvid import NuvidIE
from .oktoberfesttv import OktoberfestTVIE
from .ooyala import OoyalaIE
from .openfilm import OpenFilmIE
from .orf import (
ORFTVthekIE,
ORFOE1IE,
@ -261,7 +332,10 @@ from .orf import (
from .parliamentliveuk import ParliamentLiveUKIE
from .patreon import PatreonIE
from .pbs import PBSIE
from .phoenix import PhoenixIE
from .photobucket import PhotobucketIE
from .planetaplay import PlanetaPlayIE
from .played import PlayedIE
from .playfm import PlayFMIE
from .playvid import PlayvidIE
from .podomatic import PodomaticIE
@ -272,24 +346,32 @@ from .pornoxo import PornoXOIE
from .promptfile import PromptFileIE
from .prosiebensat1 import ProSiebenSat1IE
from .pyvideo import PyvideoIE
from .quickvid import QuickVidIE
from .radiode import RadioDeIE
from .radiobremen import RadioBremenIE
from .radiofrance import RadioFranceIE
from .rai import RaiIE
from .rbmaradio import RBMARadioIE
from .redtube import RedTubeIE
from .restudy import RestudyIE
from .reverbnation import ReverbNationIE
from .ringtv import RingTVIE
from .ro220 import Ro220IE
from .rottentomatoes import RottenTomatoesIE
from .roxwel import RoxwelIE
from .rtbf import RTBFIE
from .rte import RteIE
from .rtlnl import RtlXlIE
from .rtlnow import RTLnowIE
from .rtl2 import RTL2IE
from .rtp import RTPIE
from .rts import RTSIE
from .rtve import RTVEALaCartaIE, RTVELiveIE
from .ruhd import RUHDIE
from .rutube import (
RutubeIE,
RutubeChannelIE,
RutubeEmbedIE,
RutubeMovieIE,
RutubePersonIE,
)
@ -299,7 +381,11 @@ from .savefrom import SaveFromIE
from .sbs import SBSIE
from .scivee import SciVeeIE
from .screencast import ScreencastIE
from .screencastomatic import ScreencastOMaticIE
from .screenwavemedia import CinemassacreIE, ScreenwaveMediaIE, TeamFourIE
from .servingsys import ServingSysIE
from .sexu import SexuIE
from .sexykarma import SexyKarmaIE
from .shared import SharedIE
from .sharesix import ShareSixIE
from .sina import SinaIE
@ -330,16 +416,22 @@ from .spankwire import SpankwireIE
from .spiegel import SpiegelIE, SpiegelArticleIE
from .spiegeltv import SpiegeltvIE
from .spike import SpikeIE
from .sport5 import Sport5IE
from .sportbox import SportBoxIE
from .sportdeutschland import SportDeutschlandIE
from .srmediathek import SRMediathekIE
from .stanfordoc import StanfordOpenClassroomIE
from .steam import SteamIE
from .streamcloud import StreamcloudIE
from .streamcz import StreamCZIE
from .streetvoice import StreetVoiceIE
from .sunporno import SunPornoIE
from .swrmediathek import SWRMediathekIE
from .syfy import SyfyIE
from .sztvhu import SztvHuIE
from .tagesschau import TagesschauIE
from .tapely import TapelyIE
from .tass import TassIE
from .teachertube import (
TeacherTubeIE,
TeacherTubeUserIE,
@ -348,15 +440,26 @@ from .teachingchannel import TeachingChannelIE
from .teamcoco import TeamcocoIE
from .techtalks import TechTalksIE
from .ted import TEDIE
from .telebruxelles import TeleBruxellesIE
from .telecinco import TelecincoIE
from .telemb import TeleMBIE
from .teletask import TeleTaskIE
from .tenplay import TenPlayIE
from .testurl import TestURLIE
from .testtube import TestTubeIE
from .tf1 import TF1IE
from .theonion import TheOnionIE
from .theplatform import ThePlatformIE
from .thesixtyone import TheSixtyOneIE
from .thisav import ThisAVIE
from .tinypic import TinyPicIE
from .tlc import TlcIE, TlcDeIE
from .tmz import TMZIE
from .tnaflix import TNAFlixIE
from .thvideo import (
THVideoIE,
THVideoPlaylistIE
)
from .toutv import TouTvIE
from .toypics import ToypicsUserIE, ToypicsIE
from .traileraddict import TrailerAddictIE
@ -365,10 +468,22 @@ from .trutube import TruTubeIE
from .tube8 import Tube8IE
from .tudou import TudouIE
from .tumblr import TumblrIE
from .tunein import TuneInIE
from .turbo import TurboIE
from .tutv import TutvIE
from .tvigle import TvigleIE
from .tvp import TvpIE
from .tvp import TvpIE, TvpSeriesIE
from .tvplay import TVPlayIE
from .twentyfourvideo import TwentyFourVideoIE
from .twitch import (
TwitchVideoIE,
TwitchChapterIE,
TwitchVodIE,
TwitchProfileIE,
TwitchPastBroadcastsIE,
TwitchBookmarksIE,
TwitchStreamIE,
)
from .ubu import UbuIE
from .udemy import (
UdemyIE,
@ -384,22 +499,27 @@ from .vesti import VestiIE
from .vevo import VevoIE
from .vgtv import VGTVIE
from .vh1 import VH1IE
from .vice import ViceIE
from .viddler import ViddlerIE
from .videobam import VideoBamIE
from .videodetective import VideoDetectiveIE
from .videolecturesnet import VideoLecturesNetIE
from .videofyme import VideofyMeIE
from .videomega import VideoMegaIE
from .videopremium import VideoPremiumIE
from .videott import VideoTtIE
from .videoweed import VideoWeedIE
from .vidme import VidmeIE
from .vidzi import VidziIE
from .vier import VierIE, VierVideosIE
from .vimeo import (
VimeoIE,
VimeoChannelIE,
VimeoUserIE,
VimeoAlbumIE,
VimeoChannelIE,
VimeoGroupsIE,
VimeoLikesIE,
VimeoReviewIE,
VimeoUserIE,
VimeoWatchLaterIE,
)
from .vimple import VimpleIE
@ -408,12 +528,17 @@ from .vine import (
VineUserIE,
)
from .viki import VikiIE
from .vk import VKIE
from .vk import (
VKIE,
VKUserVideosIE,
)
from .vodlocker import VodlockerIE
from .vporn import VpornIE
from .vrt import VRTIE
from .vube import VubeIE
from .vuclip import VuClipIE
from .vulture import VultureIE
from .walla import WallaIE
from .washingtonpost import WashingtonPostIE
from .wat import WatIE
from .wayofthemaster import WayOfTheMasterIE
@ -422,6 +547,7 @@ from .wdr import (
WDRMobileIE,
WDRMausIE,
)
from .webofstories import WebOfStoriesIE
from .weibo import WeiboIE
from .wimp import WimpIE
from .wistia import WistiaIE
@ -430,17 +556,22 @@ from .wrzuta import WrzutaIE
from .xbef import XBefIE
from .xboxclips import XboxClipsIE
from .xhamster import XHamsterIE
from .xminus import XMinusIE
from .xnxx import XNXXIE
from .xvideos import XVideosIE
from .xtube import XTubeUserIE, XTubeIE
from .xuite import XuiteIE
from .xxxymovies import XXXYMoviesIE
from .yahoo import (
YahooIE,
YahooNewsIE,
YahooSearchIE,
)
from .yesjapan import YesJapanIE
from .ynet import YnetIE
from .youjizz import YouJizzIE
from .youku import YoukuIE
from .youporn import YouPornIE
from .yourupload import YourUploadIE
from .youtube import (
YoutubeIE,
YoutubeChannelIE,
@ -453,14 +584,16 @@ from .youtube import (
YoutubeSearchURLIE,
YoutubeShowIE,
YoutubeSubscriptionsIE,
YoutubeTopListIE,
YoutubeTruncatedIDIE,
YoutubeTruncatedURLIE,
YoutubeUserIE,
YoutubeWatchLaterIE,
)
from .zdf import ZDFIE
from .zdf import ZDFIE, ZDFChannelIE
from .zingmp3 import (
ZingMp3SongIE,
ZingMp3AlbumIE,
)
_ALL_CLASSES = [
klass
@ -477,6 +610,17 @@ def gen_extractors():
return [klass() for klass in _ALL_CLASSES]
def list_extractors(age_limit):
"""
Return a list of extractors that are suitable for the given age,
sorted by extractor ID.
"""
return sorted(
filter(lambda ie: ie.is_suitable(age_limit), gen_extractors()),
key=lambda ie: ie.IE_NAME.lower())
def get_info_extractor(ie_name):
"""Returns the info extractor class with the given ie_name"""
return globals()[ie_name + 'IE']

View File

@ -11,19 +11,18 @@ class ABCIE(InfoExtractor):
_VALID_URL = r'http://www\.abc\.net\.au/news/[^/]+/[^/]+/(?P<id>\d+)'
_TEST = {
'url': 'http://www.abc.net.au/news/2014-07-25/bringing-asylum-seekers-to-australia-would-give/5624716',
'md5': 'dad6f8ad011a70d9ddf887ce6d5d0742',
'url': 'http://www.abc.net.au/news/2014-11-05/australia-to-staff-ebola-treatment-centre-in-sierra-leone/5868334',
'md5': 'cb3dd03b18455a661071ee1e28344d9f',
'info_dict': {
'id': '5624716',
'id': '5868334',
'ext': 'mp4',
'title': 'Bringing asylum seekers to Australia would give them right to asylum claims: professor',
'description': 'md5:ba36fa5e27e5c9251fd929d339aea4af',
'title': 'Australia to help staff Ebola treatment centre in Sierra Leone',
'description': 'md5:809ad29c67a05f54eb41f2a105693a67',
},
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
urls_info_json = self._search_regex(

View File

@ -0,0 +1,68 @@
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import parse_iso8601
class Abc7NewsIE(InfoExtractor):
_VALID_URL = r'https?://abc7news\.com(?:/[^/]+/(?P<display_id>[^/]+))?/(?P<id>\d+)'
_TESTS = [
{
'url': 'http://abc7news.com/entertainment/east-bay-museum-celebrates-vintage-synthesizers/472581/',
'info_dict': {
'id': '472581',
'display_id': 'east-bay-museum-celebrates-vintage-synthesizers',
'ext': 'mp4',
'title': 'East Bay museum celebrates history of synthesized music',
'description': 'md5:a4f10fb2f2a02565c1749d4adbab4b10',
'thumbnail': 're:^https?://.*\.jpg$',
'timestamp': 1421123075,
'upload_date': '20150113',
'uploader': 'Jonathan Bloom',
},
'params': {
# m3u8 download
'skip_download': True,
},
},
{
'url': 'http://abc7news.com/472581',
'only_matching': True,
},
]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
display_id = mobj.group('display_id') or video_id
webpage = self._download_webpage(url, display_id)
m3u8 = self._html_search_meta(
'contentURL', webpage, 'm3u8 url', fatal=True)
formats = self._extract_m3u8_formats(m3u8, display_id, 'mp4')
self._sort_formats(formats)
title = self._og_search_title(webpage).strip()
description = self._og_search_description(webpage).strip()
thumbnail = self._og_search_thumbnail(webpage)
timestamp = parse_iso8601(self._search_regex(
r'<div class="meta">\s*<time class="timeago" datetime="([^"]+)">',
webpage, 'upload date', fatal=False))
uploader = self._search_regex(
r'rel="author">([^<]+)</a>',
webpage, 'uploader', default=None)
return {
'id': video_id,
'display_id': display_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'timestamp': timestamp,
'uploader': uploader,
'formats': formats,
}

View File

@ -1,4 +1,5 @@
from __future__ import unicode_literals
import re
from .common import InfoExtractor
@ -18,15 +19,14 @@ class AcademicEarthCourseIE(InfoExtractor):
}
def _real_extract(self, url):
m = re.match(self._VALID_URL, url)
playlist_id = m.group('id')
playlist_id = self._match_id(url)
webpage = self._download_webpage(url, playlist_id)
title = self._html_search_regex(
r'<h1 class="playlist-name"[^>]*?>(.*?)</h1>', webpage, u'title')
r'<h1 class="playlist-name"[^>]*?>(.*?)</h1>', webpage, 'title')
description = self._html_search_regex(
r'<p class="excerpt"[^>]*?>(.*?)</p>',
webpage, u'description', fatal=False)
webpage, 'description', fatal=False)
urls = re.findall(
r'<li class="lecture-preview">\s*?<a target="_blank" href="([^"]+)">',
webpage)

View File

@ -3,19 +3,19 @@ from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
from ..compat import (
compat_HTTPError,
compat_str,
compat_urllib_parse,
compat_urllib_parse_urlparse,
)
from ..utils import (
ExtractorError,
)
class AddAnimeIE(InfoExtractor):
_VALID_URL = r'^http://(?:\w+\.)?add-anime\.net/watch_video\.php\?(?:.*?)v=(?P<video_id>[\w_]+)(?:.*)'
_VALID_URL = r'^http://(?:\w+\.)?add-anime\.net/watch_video\.php\?(?:.*?)v=(?P<id>[\w_]+)(?:.*)'
_TEST = {
'url': 'http://www.add-anime.net/watch_video.php?v=24MR3YO5SAS9',
'md5': '72954ea10bc979ab5e2eb288b21425a0',
@ -28,9 +28,9 @@ class AddAnimeIE(InfoExtractor):
}
def _real_extract(self, url):
video_id = self._match_id(url)
try:
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('video_id')
webpage = self._download_webpage(url, video_id)
except ExtractorError as ee:
if not isinstance(ee.cause, compat_HTTPError) or \
@ -48,7 +48,7 @@ class AddAnimeIE(InfoExtractor):
r'a\.value = ([0-9]+)[+]([0-9]+)[*]([0-9]+);',
redir_webpage)
if av is None:
raise ExtractorError(u'Cannot find redirect math task')
raise ExtractorError('Cannot find redirect math task')
av_res = int(av.group(1)) + int(av.group(2)) * int(av.group(3))
parsed_url = compat_urllib_parse_urlparse(url)

View File

@ -0,0 +1,70 @@
from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import (
parse_duration,
unified_strdate,
str_to_int,
)
class AdobeTVIE(InfoExtractor):
_VALID_URL = r'https?://tv\.adobe\.com/watch/[^/]+/(?P<id>[^/]+)'
_TEST = {
'url': 'http://tv.adobe.com/watch/the-complete-picture-with-julieanne-kost/quick-tip-how-to-draw-a-circle-around-an-object-in-photoshop/',
'md5': '9bc5727bcdd55251f35ad311ca74fa1e',
'info_dict': {
'id': 'quick-tip-how-to-draw-a-circle-around-an-object-in-photoshop',
'ext': 'mp4',
'title': 'Quick Tip - How to Draw a Circle Around an Object in Photoshop',
'description': 'md5:99ec318dc909d7ba2a1f2b038f7d2311',
'thumbnail': 're:https?://.*\.jpg$',
'upload_date': '20110914',
'duration': 60,
'view_count': int,
},
}
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
player = self._parse_json(
self._search_regex(r'html5player:\s*({.+?})\s*\n', webpage, 'player'),
video_id)
title = player.get('title') or self._search_regex(
r'data-title="([^"]+)"', webpage, 'title')
description = self._og_search_description(webpage)
thumbnail = self._og_search_thumbnail(webpage)
upload_date = unified_strdate(
self._html_search_meta('datepublished', webpage, 'upload date'))
duration = parse_duration(
self._html_search_meta('duration', webpage, 'duration')
or self._search_regex(r'Runtime:\s*(\d{2}:\d{2}:\d{2})', webpage, 'duration'))
view_count = str_to_int(self._search_regex(
r'<div class="views">\s*Views?:\s*([\d,.]+)\s*</div>',
webpage, 'view count'))
formats = [{
'url': source['src'],
'format_id': source.get('quality') or source['src'].split('-')[-1].split('.')[0] or None,
'tbr': source.get('bitrate'),
} for source in player['sources']]
self._sort_formats(formats)
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'upload_date': upload_date,
'duration': duration,
'view_count': view_count,
'formats': formats,
}

View File

@ -2,122 +2,150 @@
from __future__ import unicode_literals
import re
import json
from .common import InfoExtractor
from ..utils import (
ExtractorError,
xpath_text,
float_or_none,
)
class AdultSwimIE(InfoExtractor):
_VALID_URL = r'https?://video\.adultswim\.com/(?P<path>.+?)(?:\.html)?(?:\?.*)?(?:#.*)?$'
_TEST = {
'url': 'http://video.adultswim.com/rick-and-morty/close-rick-counters-of-the-rick-kind.html?x=y#title',
_VALID_URL = r'https?://(?:www\.)?adultswim\.com/videos/(?P<is_playlist>playlists/)?(?P<show_path>[^/]+)/(?P<episode_path>[^/?#]+)/?'
_TESTS = [{
'url': 'http://adultswim.com/videos/rick-and-morty/pilot',
'playlist': [
{
'md5': '4da359ec73b58df4575cd01a610ba5dc',
'md5': '247572debc75c7652f253c8daa51a14d',
'info_dict': {
'id': '8a250ba1450996e901453d7f02ca02f5',
'id': 'rQxZvXQ4ROaSOqq-or2Mow-0',
'ext': 'flv',
'title': 'Rick and Morty Close Rick-Counters of the Rick Kind part 1',
'description': 'Rick has a run in with some old associates, resulting in a fallout with Morty. You got any chips, broh?',
'uploader': 'Rick and Morty',
'thumbnail': 'http://i.cdn.turner.com/asfix/repository/8a250ba13f865824013fc9db8b6b0400/thumbnail_267549017116827057.jpg'
}
'title': 'Rick and Morty - Pilot Part 1',
'description': "Rick moves in with his daughter's family and establishes himself as a bad influence on his grandson, Morty. "
},
},
{
'md5': 'ffbdf55af9331c509d95350bd0cc1819',
'md5': '77b0e037a4b20ec6b98671c4c379f48d',
'info_dict': {
'id': '8a250ba1450996e901453d7f4bd102f6',
'id': 'rQxZvXQ4ROaSOqq-or2Mow-3',
'ext': 'flv',
'title': 'Rick and Morty Close Rick-Counters of the Rick Kind part 2',
'description': 'Rick has a run in with some old associates, resulting in a fallout with Morty. You got any chips, broh?',
'uploader': 'Rick and Morty',
'thumbnail': 'http://i.cdn.turner.com/asfix/repository/8a250ba13f865824013fc9db8b6b0400/thumbnail_267549017116827057.jpg'
}
'title': 'Rick and Morty - Pilot Part 4',
'description': "Rick moves in with his daughter's family and establishes himself as a bad influence on his grandson, Morty. "
},
{
'md5': 'b92409635540304280b4b6c36bd14a0a',
'info_dict': {
'id': '8a250ba1450996e901453d7fa73c02f7',
'ext': 'flv',
'title': 'Rick and Morty Close Rick-Counters of the Rick Kind part 3',
'description': 'Rick has a run in with some old associates, resulting in a fallout with Morty. You got any chips, broh?',
'uploader': 'Rick and Morty',
'thumbnail': 'http://i.cdn.turner.com/asfix/repository/8a250ba13f865824013fc9db8b6b0400/thumbnail_267549017116827057.jpg'
}
},
{
'md5': 'e8818891d60e47b29cd89d7b0278156d',
],
'info_dict': {
'id': '8a250ba1450996e901453d7fc8ba02f8',
'title': 'Rick and Morty - Pilot',
'description': "Rick moves in with his daughter's family and establishes himself as a bad influence on his grandson, Morty. "
}
}, {
'url': 'http://www.adultswim.com/videos/playlists/american-parenting/putting-francine-out-of-business/',
'playlist': [
{
'md5': '2eb5c06d0f9a1539da3718d897f13ec5',
'info_dict': {
'id': '-t8CamQlQ2aYZ49ItZCFog-0',
'ext': 'flv',
'title': 'Rick and Morty Close Rick-Counters of the Rick Kind part 4',
'description': 'Rick has a run in with some old associates, resulting in a fallout with Morty. You got any chips, broh?',
'uploader': 'Rick and Morty',
'thumbnail': 'http://i.cdn.turner.com/asfix/repository/8a250ba13f865824013fc9db8b6b0400/thumbnail_267549017116827057.jpg'
}
}
]
'title': 'American Dad - Putting Francine Out of Business',
'description': 'Stan hatches a plan to get Francine out of the real estate business.Watch more American Dad on [adult swim].'
},
}
],
'info_dict': {
'title': 'American Dad - Putting Francine Out of Business',
'description': 'Stan hatches a plan to get Francine out of the real estate business.Watch more American Dad on [adult swim].'
},
}]
_video_extensions = {
'3500': 'flv',
'640': 'mp4',
'150': 'mp4',
'ipad': 'm3u8',
'iphone': 'm3u8'
}
_video_dimensions = {
'3500': (1280, 720),
'640': (480, 270),
'150': (320, 180)
}
@staticmethod
def find_video_info(collection, slug):
for video in collection.get('videos'):
if video.get('slug') == slug:
return video
@staticmethod
def find_collection_by_linkURL(collections, linkURL):
for collection in collections:
if collection.get('linkURL') == linkURL:
return collection
@staticmethod
def find_collection_containing_video(collections, slug):
for collection in collections:
for video in collection.get('videos'):
if video.get('slug') == slug:
return collection, video
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_path = mobj.group('path')
show_path = mobj.group('show_path')
episode_path = mobj.group('episode_path')
is_playlist = True if mobj.group('is_playlist') else False
webpage = self._download_webpage(url, video_path)
episode_id = self._html_search_regex(
r'<link rel="video_src" href="http://i\.adultswim\.com/adultswim/adultswimtv/tools/swf/viralplayer.swf\?id=([0-9a-f]+?)"\s*/?\s*>',
webpage, 'episode_id')
title = self._og_search_title(webpage)
webpage = self._download_webpage(url, episode_path)
index_url = 'http://asfix.adultswim.com/asfix-svc/episodeSearch/getEpisodesByIDs?networkName=AS&ids=%s' % episode_id
idoc = self._download_xml(index_url, title, 'Downloading episode index', 'Unable to download episode index')
# Extract the value of `bootstrappedData` from the Javascript in the page.
bootstrappedDataJS = self._search_regex(r'var bootstrappedData = ({.*});', webpage, episode_path)
episode_el = idoc.find('.//episode')
show_title = episode_el.attrib.get('collectionTitle')
episode_title = episode_el.attrib.get('title')
thumbnail = episode_el.attrib.get('thumbnailUrl')
description = episode_el.find('./description').text.strip()
try:
bootstrappedData = json.loads(bootstrappedDataJS)
except ValueError as ve:
errmsg = '%s: Failed to parse JSON ' % episode_path
raise ExtractorError(errmsg, cause=ve)
# Downloading videos from a /videos/playlist/ URL needs to be handled differently.
# NOTE: We are only downloading one video (the current one) not the playlist
if is_playlist:
collections = bootstrappedData['playlists']['collections']
collection = self.find_collection_by_linkURL(collections, show_path)
video_info = self.find_video_info(collection, episode_path)
show_title = video_info['showTitle']
segment_ids = [video_info['videoPlaybackID']]
else:
collections = bootstrappedData['show']['collections']
collection, video_info = self.find_collection_containing_video(collections, episode_path)
show = bootstrappedData['show']
show_title = show['title']
segment_ids = [clip['videoPlaybackID'] for clip in video_info['clips']]
episode_id = video_info['id']
episode_title = video_info['title']
episode_description = video_info['description']
episode_duration = video_info.get('duration')
entries = []
segment_els = episode_el.findall('./segments/segment')
for part_num, segment_id in enumerate(segment_ids):
segment_url = 'http://www.adultswim.com/videos/api/v0/assets?id=%s&platform=mobile' % segment_id
for part_num, segment_el in enumerate(segment_els):
segment_id = segment_el.attrib.get('id')
segment_title = '%s %s part %d' % (show_title, episode_title, part_num + 1)
thumbnail = segment_el.attrib.get('thumbnailUrl')
duration = segment_el.attrib.get('duration')
segment_title = '%s - %s' % (show_title, episode_title)
if len(segment_ids) > 1:
segment_title += ' Part %d' % (part_num + 1)
segment_url = 'http://asfix.adultswim.com/asfix-svc/episodeservices/getCvpPlaylist?networkName=AS&id=%s' % segment_id
idoc = self._download_xml(
segment_url, segment_title,
'Downloading segment information', 'Unable to download segment information')
segment_duration = float_or_none(
xpath_text(idoc, './/trt', 'segment duration').strip())
formats = []
file_els = idoc.findall('.//files/file')
for file_el in file_els:
bitrate = file_el.attrib.get('bitrate')
type = file_el.attrib.get('type')
width, height = self._video_dimensions.get(bitrate, (None, None))
ftype = file_el.attrib.get('type')
formats.append({
'format_id': '%s-%s' % (bitrate, type),
'url': file_el.text,
'ext': self._video_extensions.get(bitrate, 'mp4'),
'format_id': '%s_%s' % (bitrate, ftype),
'url': file_el.text.strip(),
# The bitrate may not be a number (for example: 'iphone')
'tbr': int(bitrate) if bitrate.isdigit() else None,
'height': height,
'width': width
'quality': 1 if ftype == 'hd' else -1
})
self._sort_formats(formats)
@ -126,18 +154,16 @@ class AdultSwimIE(InfoExtractor):
'id': segment_id,
'title': segment_title,
'formats': formats,
'uploader': show_title,
'thumbnail': thumbnail,
'duration': duration,
'description': description
'duration': segment_duration,
'description': episode_description
})
return {
'_type': 'playlist',
'id': episode_id,
'display_id': video_path,
'display_id': episode_path,
'entries': entries,
'title': '%s %s' % (show_title, episode_title),
'description': description,
'thumbnail': thumbnail
'title': '%s - %s' % (show_title, episode_title),
'description': episode_description,
'duration': episode_duration
}

View File

@ -0,0 +1,35 @@
from __future__ import unicode_literals
from .common import InfoExtractor
class AlJazeeraIE(InfoExtractor):
_VALID_URL = r'http://www\.aljazeera\.com/programmes/.*?/(?P<id>[^/]+)\.html'
_TEST = {
'url': 'http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html',
'info_dict': {
'id': '3792260579001',
'ext': 'mp4',
'title': 'The Slum - Episode 1: Deliverance',
'description': 'As a birth attendant advocating for family planning, Remy is on the frontline of Tondo\'s battle with overcrowding.',
'uploader': 'Al Jazeera English',
},
'add_ie': ['Brightcove'],
}
def _real_extract(self, url):
program_name = self._match_id(url)
webpage = self._download_webpage(url, program_name)
brightcove_id = self._search_regex(
r'RenderPagesVideo\(\'(.+?)\'', webpage, 'brightcove id')
return {
'_type': 'url',
'url': (
'brightcove:'
'playerKey=AQ~~%2CAAAAmtVJIFk~%2CTVGOQ5ZTwJbeMWnq5d_H4MOM57xfzApc'
'&%40videoPlayer={0}'.format(brightcove_id)
),
'ie_key': 'Brightcove',
}

View File

@ -5,15 +5,14 @@ import re
import json
from .common import InfoExtractor
from ..compat import compat_str
from ..utils import (
compat_str,
qualities,
determine_ext,
)
class AllocineIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?allocine\.fr/(?P<typ>article|video|film)/(fichearticle_gen_carticle=|player_gen_cmedia=|fichefilm_gen_cfilm=)(?P<id>[0-9]+)(?:\.html)?'
_VALID_URL = r'https?://(?:www\.)?allocine\.fr/(?P<typ>article|video|film)/(fichearticle_gen_carticle=|player_gen_cmedia=|fichefilm_gen_cfilm=|video-)(?P<id>[0-9]+)(?:\.html)?'
_TESTS = [{
'url': 'http://www.allocine.fr/article/fichearticle_gen_carticle=18635087.html',
@ -22,7 +21,7 @@ class AllocineIE(InfoExtractor):
'id': '19546517',
'ext': 'mp4',
'title': 'Astérix - Le Domaine des Dieux Teaser VF',
'description': 'md5:4a754271d9c6f16c72629a8a993ee884',
'description': 'md5:abcd09ce503c6560512c14ebfdb720d2',
'thumbnail': 're:http://.*\.jpg',
},
}, {
@ -45,6 +44,9 @@ class AllocineIE(InfoExtractor):
'description': 'md5:71742e3a74b0d692c7fce0dd2017a4ac',
'thumbnail': 're:http://.*\.jpg',
},
}, {
'url': 'http://www.allocine.fr/video/video-19550147/',
'only_matching': True,
}]
def _real_extract(self, url):
@ -75,9 +77,7 @@ class AllocineIE(InfoExtractor):
'format_id': format_id,
'quality': quality(format_id),
'url': v,
'ext': determine_ext(v),
})
self._sort_formats(formats)
return {

View File

@ -0,0 +1,77 @@
from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import (
parse_iso8601,
parse_duration,
parse_filesize,
int_or_none,
)
class AlphaPornoIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?alphaporno\.com/videos/(?P<id>[^/]+)'
_TEST = {
'url': 'http://www.alphaporno.com/videos/sensual-striptease-porn-with-samantha-alexandra/',
'md5': 'feb6d3bba8848cd54467a87ad34bd38e',
'info_dict': {
'id': '258807',
'display_id': 'sensual-striptease-porn-with-samantha-alexandra',
'ext': 'mp4',
'title': 'Sensual striptease porn with Samantha Alexandra',
'thumbnail': 're:https?://.*\.jpg$',
'timestamp': 1418694611,
'upload_date': '20141216',
'duration': 387,
'filesize_approx': 54120000,
'tbr': 1145,
'categories': list,
'age_limit': 18,
}
}
def _real_extract(self, url):
display_id = self._match_id(url)
webpage = self._download_webpage(url, display_id)
video_id = self._search_regex(
r"video_id\s*:\s*'([^']+)'", webpage, 'video id', default=None)
video_url = self._search_regex(
r"video_url\s*:\s*'([^']+)'", webpage, 'video url')
ext = self._html_search_meta(
'encodingFormat', webpage, 'ext', default='.mp4')[1:]
title = self._search_regex(
[r'<meta content="([^"]+)" itemprop="description">',
r'class="title" itemprop="name">([^<]+)<'],
webpage, 'title')
thumbnail = self._html_search_meta('thumbnail', webpage, 'thumbnail')
timestamp = parse_iso8601(self._html_search_meta(
'uploadDate', webpage, 'upload date'))
duration = parse_duration(self._html_search_meta(
'duration', webpage, 'duration'))
filesize_approx = parse_filesize(self._html_search_meta(
'contentSize', webpage, 'file size'))
bitrate = int_or_none(self._html_search_meta(
'bitrate', webpage, 'bitrate'))
categories = self._html_search_meta(
'keywords', webpage, 'categories', default='').split(',')
age_limit = self._rta_search(webpage)
return {
'id': video_id,
'display_id': display_id,
'url': video_url,
'ext': ext,
'title': title,
'thumbnail': thumbnail,
'timestamp': timestamp,
'duration': duration,
'filesize_approx': filesize_approx,
'tbr': bitrate,
'categories': categories,
'age_limit': age_limit,
}

View File

@ -35,7 +35,7 @@ class AnySexIE(InfoExtractor):
title = self._html_search_regex(r'<title>(.*?)</title>', webpage, 'title')
description = self._html_search_regex(
r'<div class="description">([^<]+)</div>', webpage, 'description', fatal=False)
r'<div class="description"[^>]*>([^<]+)</div>', webpage, 'description', fatal=False)
thumbnail = self._html_search_regex(
r'preview_url\s*:\s*\'(.*?)\'', webpage, 'thumbnail', fatal=False)
@ -43,7 +43,7 @@ class AnySexIE(InfoExtractor):
r'<a href="http://anysex\.com/categories/[^"]+" title="[^"]*">([^<]+)</a>', webpage)
duration = parse_duration(self._search_regex(
r'<b>Duration:</b> (\d+:\d+)', webpage, 'duration', fatal=False))
r'<b>Duration:</b> (?:<q itemprop="duration">)?(\d+:\d+)', webpage, 'duration', fatal=False))
view_count = int_or_none(self._html_search_regex(
r'<b>Views:</b> (\d+)', webpage, 'view count', fatal=False))

View File

@ -3,7 +3,6 @@ from __future__ import unicode_literals
import re
from .common import InfoExtractor
from .fivemin import FiveMinIE
class AolIE(InfoExtractor):
@ -42,9 +41,10 @@ class AolIE(InfoExtractor):
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
playlist_id = mobj.group('playlist_id')
if playlist_id and not self._downloader.params.get('noplaylist'):
if not playlist_id or self._downloader.params.get('noplaylist'):
return self.url_result('5min:%s' % video_id)
self.to_screen('Downloading playlist %s - add --no-playlist to just download video %s' % (playlist_id, video_id))
webpage = self._download_webpage(url, playlist_id)
@ -68,5 +68,3 @@ class AolIE(InfoExtractor):
'title': title,
'entries': entries,
}
return FiveMinIE._build_result(video_id)

View File

@ -1,5 +1,4 @@
# coding: utf-8
from __future__ import unicode_literals
import re
@ -26,8 +25,7 @@ class AparatIE(InfoExtractor):
}
def _real_extract(self, url):
m = re.match(self._VALID_URL, url)
video_id = m.group('id')
video_id = self._match_id(url)
# Note: There is an easier-to-parse configuration at
# http://www.aparat.com/video/video/config/videohash/%video_id
@ -40,15 +38,15 @@ class AparatIE(InfoExtractor):
for i, video_url in enumerate(video_urls):
req = HEADRequest(video_url)
res = self._request_webpage(
req, video_id, note=u'Testing video URL %d' % i, errnote=False)
req, video_id, note='Testing video URL %d' % i, errnote=False)
if res:
break
else:
raise ExtractorError(u'No working video URLs found')
raise ExtractorError('No working video URLs found')
title = self._search_regex(r'\s+title:\s*"([^"]+)"', webpage, u'title')
title = self._search_regex(r'\s+title:\s*"([^"]+)"', webpage, 'title')
thumbnail = self._search_regex(
r'\s+image:\s*"([^"]+)"', webpage, u'thumbnail', fatal=False)
r'\s+image:\s*"([^"]+)"', webpage, 'thumbnail', fatal=False)
return {
'id': video_id,

View File

@ -4,8 +4,8 @@ import re
import json
from .common import InfoExtractor
from ..compat import compat_urlparse
from ..utils import (
compat_urlparse,
int_or_none,
)
@ -70,15 +70,17 @@ class AppleTrailersIE(InfoExtractor):
uploader_id = mobj.group('company')
playlist_url = compat_urlparse.urljoin(url, 'includes/playlists/itunes.inc')
def fix_html(s):
s = re.sub(r'(?s)<script[^<]*?>.*?</script>', '', s)
s = re.sub(r'<img ([^<]*?)>', r'<img \1/>', s)
# The ' in the onClick attributes are not escaped, it couldn't be parsed
# like: http://trailers.apple.com/trailers/wb/gravity/
def _clean_json(m):
return 'iTunes.playURL(%s);' % m.group(1).replace('\'', '&#39;')
s = re.sub(self._JSON_RE, _clean_json, s)
s = '<html>' + s + u'</html>'
s = '<html>%s</html>' % s
return s
doc = self._download_xml(playlist_url, movie, transform_source=fix_html)
@ -120,14 +122,15 @@ class AppleTrailersIE(InfoExtractor):
playlist.append({
'_type': 'video',
'id': video_id,
'title': title,
'formats': formats,
'title': title,
'duration': duration,
'thumbnail': thumbnail,
'upload_date': upload_date,
'uploader_id': uploader_id,
'user_agent': 'QuickTime compatible (youtube-dl)',
'http_headers': {
'User-Agent': 'QuickTime compatible (youtube-dl)',
},
})
return {

View File

@ -1,42 +1,48 @@
from __future__ import unicode_literals
import json
import re
from .common import InfoExtractor
from ..utils import (
unified_strdate,
)
from ..utils import unified_strdate
class ArchiveOrgIE(InfoExtractor):
IE_NAME = 'archive.org'
IE_DESC = 'archive.org videos'
_VALID_URL = r'(?:https?://)?(?:www\.)?archive\.org/details/(?P<id>[^?/]+)(?:[?].*)?$'
_TEST = {
"url": "http://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect",
'file': 'XD300-23_68HighlightsAResearchCntAugHumanIntellect.ogv',
_VALID_URL = r'https?://(?:www\.)?archive\.org/details/(?P<id>[^?/]+)(?:[?].*)?$'
_TESTS = [{
'url': 'http://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
'md5': '8af1d4cf447933ed3c7f4871162602db',
'info_dict': {
"title": "1968 Demo - FJCC Conference Presentation Reel #1",
"description": "Reel 1 of 3: Also known as the \"Mother of All Demos\", Doug Engelbart's presentation at the Fall Joint Computer Conference in San Francisco, December 9, 1968 titled \"A Research Center for Augmenting Human Intellect.\" For this presentation, Doug and his team astonished the audience by not only relating their research, but demonstrating it live. This was the debut of the mouse, interactive computing, hypermedia, computer supported software engineering, video teleconferencing, etc. See also <a href=\"http://dougengelbart.org/firsts/dougs-1968-demo.html\" rel=\"nofollow\">Doug's 1968 Demo page</a> for more background, highlights, links, and the detailed paper published in this conference proceedings. Filmed on 3 reels: Reel 1 | <a href=\"http://www.archive.org/details/XD300-24_68HighlightsAResearchCntAugHumanIntellect\" rel=\"nofollow\">Reel 2</a> | <a href=\"http://www.archive.org/details/XD300-25_68HighlightsAResearchCntAugHumanIntellect\" rel=\"nofollow\">Reel 3</a>",
"upload_date": "19681210",
"uploader": "SRI International"
'id': 'XD300-23_68HighlightsAResearchCntAugHumanIntellect',
'ext': 'ogv',
'title': '1968 Demo - FJCC Conference Presentation Reel #1',
'description': 'md5:1780b464abaca9991d8968c877bb53ed',
'upload_date': '19681210',
'uploader': 'SRI International'
}
}, {
'url': 'https://archive.org/details/Cops1922',
'md5': '18f2a19e6d89af8425671da1cf3d4e04',
'info_dict': {
'id': 'Cops1922',
'ext': 'ogv',
'title': 'Buster Keaton\'s "Cops" (1922)',
'description': 'md5:70f72ee70882f713d4578725461ffcc3',
}
}]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
video_id = self._match_id(url)
json_url = url + ('?' if '?' in url else '&') + 'output=json'
json_data = self._download_webpage(json_url, video_id)
data = json.loads(json_data)
data = self._download_json(json_url, video_id)
title = data['metadata']['title'][0]
description = data['metadata']['description'][0]
uploader = data['metadata']['creator'][0]
upload_date = unified_strdate(data['metadata']['date'][0])
def get_optional(data_dict, field):
return data_dict['metadata'].get(field, [None])[0]
title = get_optional(data, 'title')
description = get_optional(data, 'description')
uploader = get_optional(data, 'creator')
upload_date = unified_strdate(get_optional(data, 'date'))
formats = [
{

View File

@ -4,16 +4,16 @@ from __future__ import unicode_literals
import re
from .common import InfoExtractor
from .generic import GenericIE
from ..utils import (
determine_ext,
ExtractorError,
qualities,
compat_urllib_parse_urlparse,
compat_urllib_parse,
int_or_none,
parse_duration,
unified_strdate,
xpath_text,
parse_xml,
)
@ -23,13 +23,7 @@ class ARDMediathekIE(InfoExtractor):
_TESTS = [{
'url': 'http://mediathek.daserste.de/sendungen_a-z/328454_anne-will/22429276_vertrauen-ist-gut-spionieren-ist-besser-geht',
'file': '22429276.mp4',
'md5': '469751912f1de0816a9fc9df8336476c',
'info_dict': {
'title': 'Vertrauen ist gut, Spionieren ist besser - Geht so deutsch-amerikanische Freundschaft?',
'description': 'Das Erste Mediathek [ARD]: Vertrauen ist gut, Spionieren ist besser - Geht so deutsch-amerikanische Freundschaft?, Anne Will, Über die Spionage-Affäre diskutieren Clemens Binninger, Katrin Göring-Eckardt, Georg Mascolo, Andrew B. Denison und Constanze Kurz.. Das Video zur Sendung Anne Will am Mittwoch, 16.07.2014',
},
'skip': 'Blocked outside of Germany',
'only_matching': True,
}, {
'url': 'http://www.ardmediathek.de/tv/Tatort/Das-Wunder-von-Wolbeck-Video-tgl-ab-20/Das-Erste/Video?documentId=22490580&bcastId=602916',
'info_dict': {
@ -51,14 +45,16 @@ class ARDMediathekIE(InfoExtractor):
else:
video_id = m.group('video_id')
urlp = compat_urllib_parse_urlparse(url)
url = urlp._replace(path=compat_urllib_parse.quote(urlp.path.encode('utf-8'))).geturl()
webpage = self._download_webpage(url, video_id)
if '>Der gewünschte Beitrag ist nicht mehr verfügbar.<' in webpage:
raise ExtractorError('Video %s is no longer available' % video_id, expected=True)
if re.search(r'[\?&]rss($|[=&])', url):
doc = parse_xml(webpage)
if doc.tag == 'rss':
return GenericIE()._extract_rss(url, video_id, doc)
title = self._html_search_regex(
[r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
r'<meta name="dcterms.title" content="(.*?)"/>',
@ -190,4 +186,3 @@ class ARDIE(InfoExtractor):
'upload_date': upload_date,
'thumbnail': thumbnail,
}

View File

@ -5,13 +5,12 @@ import re
from .common import InfoExtractor
from ..utils import (
ExtractorError,
find_xpath_attr,
unified_strdate,
determine_ext,
get_element_by_id,
compat_str,
get_element_by_attribute,
int_or_none,
qualities,
)
# There are different sources of video in arte.tv, the extraction process
@ -38,7 +37,7 @@ class ArteTvIE(InfoExtractor):
config_xml_url, video_id, note='Downloading configuration')
formats = [{
'forma_id': q.attrib['quality'],
'format_id': q.attrib['quality'],
# The playpath starts at 'mp4:', if we don't manually
# split the url, rtmpdump will incorrectly parse them
'url': q.text.split('mp4:', 1)[0],
@ -86,92 +85,70 @@ class ArteTVPlus7IE(InfoExtractor):
info = self._download_json(json_url, video_id)
player_info = info['videoJsonPlayer']
upload_date_str = player_info.get('shootingDate')
if not upload_date_str:
upload_date_str = player_info.get('VDA', '').split(' ')[0]
title = player_info['VTI'].strip()
subtitle = player_info.get('VSU', '').strip()
if subtitle:
title += ' - %s' % subtitle
info_dict = {
'id': player_info['VID'],
'title': player_info['VTI'],
'title': title,
'description': player_info.get('VDE'),
'upload_date': unified_strdate(player_info.get('VDA', '').split(' ')[0]),
'upload_date': unified_strdate(upload_date_str),
'thumbnail': player_info.get('programImage') or player_info.get('VTU', {}).get('IUR'),
}
qfunc = qualities(['HQ', 'MQ', 'EQ', 'SQ'])
all_formats = player_info['VSR'].values()
# Some formats use the m3u8 protocol
all_formats = list(filter(lambda f: f.get('videoFormat') != 'M3U8', all_formats))
def _match_lang(f):
if f.get('versionCode') is None:
return True
# Return true if that format is in the language of the url
if lang == 'fr':
l = 'F'
elif lang == 'de':
l = 'A'
else:
l = lang
regexes = [r'VO?%s' % l, r'VO?.-ST%s' % l]
return any(re.match(r, f['versionCode']) for r in regexes)
# Some formats may not be in the same language as the url
# TODO: Might want not to drop videos that does not match requested language
# but to process those formats with lower precedence
formats = filter(_match_lang, all_formats)
formats = list(formats) # in python3 filter returns an iterator
if not formats:
# Some videos are only available in the 'Originalversion'
# they aren't tagged as being in French or German
# Sometimes there are neither videos of requested lang code
# nor original version videos available
# For such cases we just take all_formats as is
formats = all_formats
if not formats:
raise ExtractorError('The formats list is empty')
if re.match(r'[A-Z]Q', formats[0]['quality']) is not None:
def sort_key(f):
return ['HQ', 'MQ', 'EQ', 'SQ'].index(f['quality'])
else:
def sort_key(f):
formats = []
for format_id, format_dict in player_info['VSR'].items():
f = dict(format_dict)
versionCode = f.get('versionCode')
if versionCode is None:
versionCode = ''
return (
# Sort first by quality
int(f.get('height', -1)),
int(f.get('bitrate', -1)),
# The original version with subtitles has lower relevance
re.match(r'VO-ST(F|A)', versionCode) is None,
# The version with sourds/mal subtitles has also lower relevance
re.match(r'VO?(F|A)-STM\1', versionCode) is None,
# Prefer http downloads over m3u8
0 if f['url'].endswith('m3u8') else 1,
)
formats = sorted(formats, key=sort_key)
def _format(format_info):
quality = ''
height = format_info.get('height')
if height is not None:
quality = compat_str(height)
bitrate = format_info.get('bitrate')
if bitrate is not None:
quality += '-%d' % bitrate
if format_info.get('versionCode') is not None:
format_id = '%s-%s' % (quality, format_info['versionCode'])
else:
format_id = quality
info = {
'format_id': format_id,
'format_note': format_info.get('versionLibelle'),
'width': format_info.get('width'),
'height': height,
}
if format_info['mediaType'] == 'rtmp':
info['url'] = format_info['streamer']
info['play_path'] = 'mp4:' + format_info['url']
info['ext'] = 'flv'
else:
info['url'] = format_info['url']
info['ext'] = determine_ext(info['url'])
return info
info_dict['formats'] = [_format(f) for f in formats]
langcode = {
'fr': 'F',
'de': 'A',
}.get(lang, lang)
lang_rexs = [r'VO?%s' % langcode, r'VO?.-ST%s' % langcode]
lang_pref = (
None if versionCode is None else (
10 if any(re.match(r, versionCode) for r in lang_rexs)
else -10))
source_pref = 0
if versionCode is not None:
# The original version with subtitles has lower relevance
if re.match(r'VO-ST(F|A)', versionCode):
source_pref -= 10
# The version with sourds/mal subtitles has also lower relevance
elif re.match(r'VO?(F|A)-STM\1', versionCode):
source_pref -= 9
format = {
'format_id': format_id,
'preference': -10 if f.get('videoFormat') == 'M3U8' else None,
'language_preference': lang_pref,
'format_note': '%s, %s' % (f.get('versionCode'), f.get('versionLibelle')),
'width': int_or_none(f.get('width')),
'height': int_or_none(f.get('height')),
'tbr': int_or_none(f.get('bitrate')),
'quality': qfunc(f.get('quality')),
'source_preference': source_pref,
}
if f.get('mediaType') == 'rtmp':
format['url'] = f['streamer']
format['play_path'] = 'mp4:' + f['url']
format['ext'] = 'flv'
else:
format['url'] = f['url']
formats.append(format)
self._sort_formats(formats)
info_dict['formats'] = formats
return info_dict

View File

@ -0,0 +1,163 @@
from __future__ import unicode_literals
import time
import hmac
from .subtitles import SubtitlesInfoExtractor
from ..compat import (
compat_str,
compat_urllib_parse,
compat_urllib_request,
)
from ..utils import (
int_or_none,
float_or_none,
xpath_text,
ExtractorError,
)
class AtresPlayerIE(SubtitlesInfoExtractor):
_VALID_URL = r'https?://(?:www\.)?atresplayer\.com/television/[^/]+/[^/]+/[^/]+/(?P<id>.+?)_\d+\.html'
_TESTS = [
{
'url': 'http://www.atresplayer.com/television/programas/el-club-de-la-comedia/temporada-4/capitulo-10-especial-solidario-nochebuena_2014122100174.html',
'md5': 'efd56753cda1bb64df52a3074f62e38a',
'info_dict': {
'id': 'capitulo-10-especial-solidario-nochebuena',
'ext': 'mp4',
'title': 'Especial Solidario de Nochebuena',
'description': 'md5:e2d52ff12214fa937107d21064075bf1',
'duration': 5527.6,
'thumbnail': 're:^https?://.*\.jpg$',
},
},
{
'url': 'http://www.atresplayer.com/television/series/el-secreto-de-puente-viejo/el-chico-de-los-tres-lunares/capitulo-977-29-12-14_2014122400174.html',
'only_matching': True,
},
]
_USER_AGENT = 'Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J'
_MAGIC = 'QWtMLXs414Yo+c#_+Q#K@NN)'
_TIMESTAMP_SHIFT = 30000
_TIME_API_URL = 'http://servicios.atresplayer.com/api/admin/time.json'
_URL_VIDEO_TEMPLATE = 'https://servicios.atresplayer.com/api/urlVideo/{1}/{0}/{1}|{2}|{3}.json'
_PLAYER_URL_TEMPLATE = 'https://servicios.atresplayer.com/episode/getplayer.json?episodePk=%s'
_EPISODE_URL_TEMPLATE = 'http://www.atresplayer.com/episodexml/%s'
_LOGIN_URL = 'https://servicios.atresplayer.com/j_spring_security_check'
def _real_initialize(self):
self._login()
def _login(self):
(username, password) = self._get_login_info()
if username is None:
return
login_form = {
'j_username': username,
'j_password': password,
}
request = compat_urllib_request.Request(
self._LOGIN_URL, compat_urllib_parse.urlencode(login_form).encode('utf-8'))
request.add_header('Content-Type', 'application/x-www-form-urlencoded')
response = self._download_webpage(
request, None, 'Logging in as %s' % username)
error = self._html_search_regex(
r'(?s)<ul class="list_error">(.+?)</ul>', response, 'error', default=None)
if error:
raise ExtractorError(
'Unable to login: %s' % error, expected=True)
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
episode_id = self._search_regex(
r'episode="([^"]+)"', webpage, 'episode id')
timestamp = int_or_none(self._download_webpage(
self._TIME_API_URL,
video_id, 'Downloading timestamp', fatal=False), 1000, time.time())
timestamp_shifted = compat_str(timestamp + self._TIMESTAMP_SHIFT)
token = hmac.new(
self._MAGIC.encode('ascii'),
(episode_id + timestamp_shifted).encode('utf-8')
).hexdigest()
formats = []
for fmt in ['windows', 'android_tablet']:
request = compat_urllib_request.Request(
self._URL_VIDEO_TEMPLATE.format(fmt, episode_id, timestamp_shifted, token))
request.add_header('User-Agent', self._USER_AGENT)
fmt_json = self._download_json(
request, video_id, 'Downloading %s video JSON' % fmt)
result = fmt_json.get('resultDes')
if result.lower() != 'ok':
raise ExtractorError(
'%s returned error: %s' % (self.IE_NAME, result), expected=True)
for format_id, video_url in fmt_json['resultObject'].items():
if format_id == 'token' or not video_url.startswith('http'):
continue
if video_url.endswith('/Manifest'):
if 'geodeswowsmpra3player' in video_url:
f4m_path = video_url.split('smil:', 1)[-1].split('free_', 1)[0]
f4m_url = 'http://drg.antena3.com/{0}hds/es/sd.f4m'.format(f4m_path)
# this videos are protected by DRM, the f4m downloader doesn't support them
continue
else:
f4m_url = video_url[:-9] + '/manifest.f4m'
formats.extend(self._extract_f4m_formats(f4m_url, video_id))
else:
formats.append({
'url': video_url,
'format_id': 'android-%s' % format_id,
'preference': 1,
})
self._sort_formats(formats)
player = self._download_json(
self._PLAYER_URL_TEMPLATE % episode_id,
episode_id)
path_data = player.get('pathData')
episode = self._download_xml(
self._EPISODE_URL_TEMPLATE % path_data,
video_id, 'Downloading episode XML')
duration = float_or_none(xpath_text(
episode, './media/asset/info/technical/contentDuration', 'duration'))
art = episode.find('./media/asset/info/art')
title = xpath_text(art, './name', 'title')
description = xpath_text(art, './description', 'description')
thumbnail = xpath_text(episode, './media/asset/files/background', 'thumbnail')
subtitles = {}
subtitle = xpath_text(episode, './media/asset/files/subtitle', 'subtitle')
if subtitle:
subtitles['es'] = subtitle
if self._downloader.params.get('listsubtitles', False):
self._list_available_subtitles(video_id, subtitles)
return
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'duration': duration,
'formats': formats,
'subtitles': self.extract_subtitles(video_id, subtitles),
}

View File

@ -0,0 +1,55 @@
from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import unified_strdate
class ATTTechChannelIE(InfoExtractor):
_VALID_URL = r'https?://techchannel\.att\.com/play-video\.cfm/([^/]+/)*(?P<id>.+)'
_TEST = {
'url': 'http://techchannel.att.com/play-video.cfm/2014/1/27/ATT-Archives-The-UNIX-System-Making-Computers-Easier-to-Use',
'info_dict': {
'id': '11316',
'display_id': 'ATT-Archives-The-UNIX-System-Making-Computers-Easier-to-Use',
'ext': 'flv',
'title': 'AT&T Archives : The UNIX System: Making Computers Easier to Use',
'description': 'A 1982 film about UNIX is the foundation for software in use around Bell Labs and AT&T.',
'thumbnail': 're:^https?://.*\.jpg$',
'upload_date': '20140127',
},
'params': {
# rtmp download
'skip_download': True,
},
}
def _real_extract(self, url):
display_id = self._match_id(url)
webpage = self._download_webpage(url, display_id)
video_url = self._search_regex(
r"url\s*:\s*'(rtmp://[^']+)'",
webpage, 'video URL')
video_id = self._search_regex(
r'mediaid\s*=\s*(\d+)',
webpage, 'video id', fatal=False)
title = self._og_search_title(webpage)
description = self._og_search_description(webpage)
thumbnail = self._og_search_thumbnail(webpage)
upload_date = unified_strdate(self._search_regex(
r'[Rr]elease\s+date:\s*(\d{1,2}/\d{1,2}/\d{4})',
webpage, 'upload date', fatal=False), False)
return {
'id': video_id,
'display_id': display_id,
'url': video_url,
'ext': 'flv',
'title': title,
'description': description,
'thumbnail': thumbnail,
'upload_date': upload_date,
}

View File

@ -0,0 +1,144 @@
# coding: utf-8
from __future__ import unicode_literals
import itertools
import time
from .common import InfoExtractor
from .soundcloud import SoundcloudIE
from ..utils import (
ExtractorError,
url_basename,
)
class AudiomackIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?audiomack\.com/song/(?P<id>[\w/-]+)'
IE_NAME = 'audiomack'
_TESTS = [
# hosted on audiomack
{
'url': 'http://www.audiomack.com/song/roosh-williams/extraordinary',
'info_dict':
{
'id': '310086',
'ext': 'mp3',
'uploader': 'Roosh Williams',
'title': 'Extraordinary'
}
},
# audiomack wrapper around soundcloud song
{
'add_ie': ['Soundcloud'],
'url': 'http://www.audiomack.com/song/xclusiveszone/take-kare',
'info_dict': {
'id': '172419696',
'ext': 'mp3',
'description': 'md5:1fc3272ed7a635cce5be1568c2822997',
'title': 'Young Thug ft Lil Wayne - Take Kare',
'uploader': 'Young Thug World',
'upload_date': '20141016',
}
},
]
def _real_extract(self, url):
# URLs end with [uploader name]/[uploader title]
# this title is whatever the user types in, and is rarely
# the proper song title. Real metadata is in the api response
album_url_tag = self._match_id(url)
# Request the extended version of the api for extra fields like artist and title
api_response = self._download_json(
'http://www.audiomack.com/api/music/url/song/%s?extended=1&_=%d' % (
album_url_tag, time.time()),
album_url_tag)
# API is inconsistent with errors
if 'url' not in api_response or not api_response['url'] or 'error' in api_response:
raise ExtractorError('Invalid url %s', url)
# Audiomack wraps a lot of soundcloud tracks in their branded wrapper
# if so, pass the work off to the soundcloud extractor
if SoundcloudIE.suitable(api_response['url']):
return {'_type': 'url', 'url': api_response['url'], 'ie_key': 'Soundcloud'}
return {
'id': api_response.get('id', album_url_tag),
'uploader': api_response.get('artist'),
'title': api_response.get('title'),
'url': api_response['url'],
}
class AudiomackAlbumIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?audiomack\.com/album/(?P<id>[\w/-]+)'
IE_NAME = 'audiomack:album'
_TESTS = [
# Standard album playlist
{
'url': 'http://www.audiomack.com/album/flytunezcom/tha-tour-part-2-mixtape',
'playlist_count': 15,
'info_dict':
{
'id': '812251',
'title': 'Tha Tour: Part 2 (Official Mixtape)'
}
},
# Album playlist ripped from fakeshoredrive with no metadata
{
'url': 'http://www.audiomack.com/album/fakeshoredrive/ppp-pistol-p-project',
'info_dict': {
'title': 'PPP (Pistol P Project)',
'id': '837572',
},
'playlist': [{
'info_dict': {
'title': 'PPP (Pistol P Project) - 9. Heaven or Hell (CHIMACA) ft Zuse (prod by DJ FU)',
'id': '837577',
'ext': 'mp3',
'uploader': 'Lil Herb a.k.a. G Herbo',
}
}],
'params': {
'playliststart': 9,
'playlistend': 9,
}
}
]
def _real_extract(self, url):
# URLs end with [uploader name]/[uploader title]
# this title is whatever the user types in, and is rarely
# the proper song title. Real metadata is in the api response
album_url_tag = self._match_id(url)
result = {'_type': 'playlist', 'entries': []}
# There is no one endpoint for album metadata - instead it is included/repeated in each song's metadata
# Therefore we don't know how many songs the album has and must infi-loop until failure
for track_no in itertools.count():
# Get song's metadata
api_response = self._download_json(
'http://www.audiomack.com/api/music/url/album/%s/%d?extended=1&_=%d'
% (album_url_tag, track_no, time.time()), album_url_tag,
note='Querying song information (%d)' % (track_no + 1))
# Total failure, only occurs when url is totally wrong
# Won't happen in middle of valid playlist (next case)
if 'url' not in api_response or 'error' in api_response:
raise ExtractorError('Invalid url for track %d of album url %s' % (track_no, url))
# URL is good but song id doesn't exist - usually means end of playlist
elif not api_response['url']:
break
else:
# Pull out the album metadata and add to result (if it exists)
for resultkey, apikey in [('id', 'album_id'), ('title', 'album_title')]:
if apikey in api_response and resultkey not in result:
result[resultkey] = api_response[apikey]
song_id = url_basename(api_response['url']).rpartition('.')[0]
result['entries'].append({
'id': api_response.get('id', song_id),
'uploader': api_response.get('artist'),
'title': api_response.get('title', song_id),
'url': api_response['url'],
})
return result

View File

@ -1,55 +0,0 @@
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
compat_urllib_parse,
determine_ext,
ExtractorError,
)
class AUEngineIE(InfoExtractor):
_VALID_URL = r'http://(?:www\.)?auengine\.com/embed\.php\?.*?file=(?P<id>[^&]+).*?'
_TEST = {
'url': 'http://auengine.com/embed.php?file=lfvlytY6&w=650&h=370',
'md5': '48972bdbcf1a3a2f5533e62425b41d4f',
'info_dict': {
'id': 'lfvlytY6',
'ext': 'mp4',
'title': '[Commie]The Legend of the Legendary Heroes - 03 - Replication Eye (Alpha Stigma)[F9410F5A]'
}
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
webpage = self._download_webpage(url, video_id)
title = self._html_search_regex(r'<title>(?P<title>.+?)</title>', webpage, 'title')
title = title.strip()
links = re.findall(r'\s(?:file|url):\s*["\']([^\'"]+)["\']', webpage)
links = map(compat_urllib_parse.unquote, links)
thumbnail = None
video_url = None
for link in links:
if link.endswith('.png'):
thumbnail = link
elif '/videos/' in link:
video_url = link
if not video_url:
raise ExtractorError('Could not find video URL')
ext = '.' + determine_ext(video_url)
if ext == title[-len(ext):]:
title = title[:-len(ext)]
return {
'id': video_id,
'url': video_url,
'title': title,
'thumbnail': thumbnail,
'http_referer': 'http://www.auengine.com/flowplayer/flowplayer.commercial-3.2.14.swf',
}

View File

@ -0,0 +1,93 @@
from __future__ import unicode_literals
import json
from .common import InfoExtractor
from ..utils import float_or_none
class AzubuIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?azubu\.tv/[^/]+#!/play/(?P<id>\d+)'
_TESTS = [
{
'url': 'http://www.azubu.tv/GSL#!/play/15575/2014-hot6-cup-last-big-match-ro8-day-1',
'md5': 'a88b42fcf844f29ad6035054bd9ecaf4',
'info_dict': {
'id': '15575',
'ext': 'mp4',
'title': '2014 HOT6 CUP LAST BIG MATCH Ro8 Day 1',
'description': 'md5:d06bdea27b8cc4388a90ad35b5c66c01',
'thumbnail': 're:^https?://.*\.jpe?g',
'timestamp': 1417523507.334,
'upload_date': '20141202',
'duration': 9988.7,
'uploader': 'GSL',
'uploader_id': 414310,
'view_count': int,
},
},
{
'url': 'http://www.azubu.tv/FnaticTV#!/play/9344/-fnatic-at-worlds-2014:-toyz---%22i-love-rekkles,-he-has-amazing-mechanics%22-',
'md5': 'b72a871fe1d9f70bd7673769cdb3b925',
'info_dict': {
'id': '9344',
'ext': 'mp4',
'title': 'Fnatic at Worlds 2014: Toyz - "I love Rekkles, he has amazing mechanics"',
'description': 'md5:4a649737b5f6c8b5c5be543e88dc62af',
'thumbnail': 're:^https?://.*\.jpe?g',
'timestamp': 1410530893.320,
'upload_date': '20140912',
'duration': 172.385,
'uploader': 'FnaticTV',
'uploader_id': 272749,
'view_count': int,
},
},
]
def _real_extract(self, url):
video_id = self._match_id(url)
data = self._download_json(
'http://www.azubu.tv/api/video/%s' % video_id, video_id)['data']
title = data['title'].strip()
description = data['description']
thumbnail = data['thumbnail']
view_count = data['view_count']
uploader = data['user']['username']
uploader_id = data['user']['id']
stream_params = json.loads(data['stream_params'])
timestamp = float_or_none(stream_params['creationDate'], 1000)
duration = float_or_none(stream_params['length'], 1000)
renditions = stream_params.get('renditions') or []
video = stream_params.get('FLVFullLength') or stream_params.get('videoFullLength')
if video:
renditions.append(video)
formats = [{
'url': fmt['url'],
'width': fmt['frameWidth'],
'height': fmt['frameHeight'],
'vbr': float_or_none(fmt['encodingRate'], 1000),
'filesize': fmt['size'],
'vcodec': fmt['videoCodec'],
'container': fmt['videoContainer'],
} for fmt in renditions if fmt['url']]
self._sort_formats(formats)
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'timestamp': timestamp,
'duration': duration,
'uploader': uploader,
'uploader_id': uploader_id,
'view_count': view_count,
'formats': formats,
}

View File

@ -5,7 +5,7 @@ import json
import itertools
from .common import InfoExtractor
from ..utils import (
from ..compat import (
compat_urllib_request,
)
@ -18,7 +18,7 @@ class BambuserIE(InfoExtractor):
_TEST = {
'url': 'http://bambuser.com/v/4050584',
# MD5 seems to be flaky, see https://travis-ci.org/rg3/youtube-dl/jobs/14051016#L388
#u'md5': 'fba8f7693e48fd4e8641b3fd5539a641',
# 'md5': 'fba8f7693e48fd4e8641b3fd5539a641',
'info_dict': {
'id': '4050584',
'ext': 'flv',
@ -73,7 +73,8 @@ class BambuserChannelIE(InfoExtractor):
urls = []
last_id = ''
for i in itertools.count(1):
req_url = ('http://bambuser.com/xhr-api/index.php?username={user}'
req_url = (
'http://bambuser.com/xhr-api/index.php?username={user}'
'&sort=created&access_mode=0%2C1%2C2&limit={count}'
'&method=broadcast&format=json&vid_older_than={last}'
).format(user=user, count=self._STEP, last=last_id)

View File

@ -4,9 +4,11 @@ import json
import re
from .common import InfoExtractor
from ..utils import (
from ..compat import (
compat_str,
compat_urlparse,
)
from ..utils import (
ExtractorError,
)
@ -15,13 +17,23 @@ class BandcampIE(InfoExtractor):
_VALID_URL = r'https?://.*?\.bandcamp\.com/track/(?P<title>.*)'
_TESTS = [{
'url': 'http://youtube-dl.bandcamp.com/track/youtube-dl-test-song',
'file': '1812978515.mp3',
'md5': 'c557841d5e50261777a6585648adf439',
'info_dict': {
"title": "youtube-dl \"'/\\\u00e4\u21ad - youtube-dl test song \"'/\\\u00e4\u21ad",
"duration": 9.8485,
'id': '1812978515',
'ext': 'mp3',
'title': "youtube-dl \"'/\\\u00e4\u21ad - youtube-dl test song \"'/\\\u00e4\u21ad",
'duration': 9.8485,
},
'_skip': 'There is a limit of 200 free downloads / month for the test song'
}, {
'url': 'http://benprunty.bandcamp.com/track/lanius-battle',
'md5': '2b68e5851514c20efdff2afc5603b8b4',
'info_dict': {
'id': '2650410135',
'ext': 'mp3',
'title': 'Lanius (Battle)',
'uploader': 'Ben Prunty Music',
},
}]
def _real_extract(self, url):
@ -59,9 +71,9 @@ class BandcampIE(InfoExtractor):
raise ExtractorError('No free songs found')
download_link = m_download.group(1)
video_id = re.search(
r'var TralbumData = {(.*?)id: (?P<id>\d*?)$',
webpage, re.MULTILINE | re.DOTALL).group('id')
video_id = self._search_regex(
r'var TralbumData = {.*?id: (?P<id>\d+),?$',
webpage, 'video id', flags=re.MULTILINE | re.DOTALL)
download_webpage = self._download_webpage(download_link, video_id, 'Downloading free downloads page')
# We get the dictionary of the track from some javascript code
@ -94,26 +106,31 @@ class BandcampIE(InfoExtractor):
class BandcampAlbumIE(InfoExtractor):
IE_NAME = 'Bandcamp:album'
_VALID_URL = r'https?://(?:(?P<subdomain>[^.]+)\.)?bandcamp\.com(?:/album/(?P<title>[^?#]+))'
_VALID_URL = r'https?://(?:(?P<subdomain>[^.]+)\.)?bandcamp\.com(?:/album/(?P<title>[^?#]+)|/?(?:$|[?#]))'
_TESTS = [{
'url': 'http://blazo.bandcamp.com/album/jazz-format-mixtape-vol-1',
'playlist': [
{
'file': '1353101989.mp3',
'md5': '39bc1eded3476e927c724321ddf116cf',
'info_dict': {
'id': '1353101989',
'ext': 'mp3',
'title': 'Intro',
}
},
{
'file': '38097443.mp3',
'md5': '1a2c32e2691474643e912cc6cd4bffaa',
'info_dict': {
'id': '38097443',
'ext': 'mp3',
'title': 'Kero One - Keep It Alive (Blazo remix)',
}
},
],
'info_dict': {
'title': 'Jazz Format Mixtape vol.1',
},
'params': {
'playlistend': 2
},
@ -124,6 +141,12 @@ class BandcampAlbumIE(InfoExtractor):
'title': 'Hierophany of the Open Grave',
},
'playlist_mincount': 9,
}, {
'url': 'http://dotscale.bandcamp.com',
'info_dict': {
'title': 'Loom',
},
'playlist_mincount': 7,
}]
def _real_extract(self, url):
@ -138,7 +161,8 @@ class BandcampAlbumIE(InfoExtractor):
entries = [
self.url_result(compat_urlparse.urljoin(url, t_path), ie=BandcampIE.ie_key())
for t_path in tracks_paths]
title = self._search_regex(r'album_title : "(.*?)"', webpage, 'title')
title = self._search_regex(
r'album_title\s*:\s*"(.*?)"', webpage, 'title', fatal=False)
return {
'_type': 'playlist',
'id': playlist_id,

View File

@ -1,15 +1,16 @@
from __future__ import unicode_literals
import re
import xml.etree.ElementTree
from .subtitles import SubtitlesInfoExtractor
from ..utils import ExtractorError
from ..compat import compat_HTTPError
class BBCCoUkIE(SubtitlesInfoExtractor):
IE_NAME = 'bbc.co.uk'
IE_DESC = 'BBC iPlayer'
_VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/(?:programmes|iplayer/episode)/(?P<id>[\da-z]{8})'
_VALID_URL = r'https?://(?:www\.)?bbc\.co\.uk/(?:(?:(?:programmes|iplayer(?:/[^/]+)?/(?:episode|playlist))/)|music/clips[/#])(?P<id>[\da-z]{8})'
_TESTS = [
{
@ -17,8 +18,8 @@ class BBCCoUkIE(SubtitlesInfoExtractor):
'info_dict': {
'id': 'b039d07m',
'ext': 'flv',
'title': 'Kaleidoscope: Leonard Cohen',
'description': 'md5:db4755d7a665ae72343779f7dacb402c',
'title': 'Kaleidoscope, Leonard Cohen',
'description': 'The Canadian poet and songwriter reflects on his musical career.',
'duration': 1740,
},
'params': {
@ -55,6 +56,71 @@ class BBCCoUkIE(SubtitlesInfoExtractor):
'skip_download': True,
},
'skip': 'Currently BBC iPlayer TV programmes are available to play in the UK only',
},
{
'url': 'http://www.bbc.co.uk/iplayer/episode/p026c7jt/tomorrows-worlds-the-unearthly-history-of-science-fiction-2-invasion',
'info_dict': {
'id': 'b03k3pb7',
'ext': 'flv',
'title': "Tomorrow's Worlds: The Unearthly History of Science Fiction",
'description': '2. Invasion',
'duration': 3600,
},
'params': {
# rtmp download
'skip_download': True,
},
'skip': 'Currently BBC iPlayer TV programmes are available to play in the UK only',
}, {
'url': 'http://www.bbc.co.uk/programmes/b04v20dw',
'info_dict': {
'id': 'b04v209v',
'ext': 'flv',
'title': 'Pete Tong, The Essential New Tune Special',
'description': "Pete has a very special mix - all of 2014's Essential New Tunes!",
'duration': 10800,
},
'params': {
# rtmp download
'skip_download': True,
}
}, {
'url': 'http://www.bbc.co.uk/music/clips/p02frcc3',
'note': 'Audio',
'info_dict': {
'id': 'p02frcch',
'ext': 'flv',
'title': 'Pete Tong, Past, Present and Future Special, Madeon - After Hours mix',
'description': 'French house superstar Madeon takes us out of the club and onto the after party.',
'duration': 3507,
},
'params': {
# rtmp download
'skip_download': True,
}
}, {
'url': 'http://www.bbc.co.uk/music/clips/p025c0zz',
'note': 'Video',
'info_dict': {
'id': 'p025c103',
'ext': 'flv',
'title': 'Reading and Leeds Festival, 2014, Rae Morris - Closer (Live on BBC Three)',
'description': 'Rae Morris performs Closer for BBC Three at Reading 2014',
'duration': 226,
},
'params': {
# rtmp download
'skip_download': True,
}
}, {
'url': 'http://www.bbc.co.uk/iplayer/playlist/p01dvks4',
'only_matching': True,
}, {
'url': 'http://www.bbc.co.uk/music/clips#p02frcc3',
'only_matching': True,
}, {
'url': 'http://www.bbc.co.uk/iplayer/cbeebies/episode/b0480276/bing-14-atchoo',
'only_matching': True,
}
]
@ -102,6 +168,10 @@ class BBCCoUkIE(SubtitlesInfoExtractor):
return playlist.findall('./{http://bbc.co.uk/2008/emp/playlist}item')
def _extract_medias(self, media_selection):
error = media_selection.find('./{http://bbc.co.uk/2008/mp/mediaselection}error')
if error is not None:
raise ExtractorError(
'%s returned error: %s' % (self.IE_NAME, error.get('id')), expected=True)
return media_selection.findall('./{http://bbc.co.uk/2008/mp/mediaselection}media')
def _extract_connections(self, media):
@ -158,45 +228,19 @@ class BBCCoUkIE(SubtitlesInfoExtractor):
subtitles[lang] = srt
return subtitles
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
group_id = mobj.group('id')
webpage = self._download_webpage(url, group_id, 'Downloading video page')
if re.search(r'id="emp-error" class="notinuk">', webpage):
raise ExtractorError('Currently BBC iPlayer TV programmes are available to play in the UK only',
expected=True)
playlist = self._download_xml('http://www.bbc.co.uk/iplayer/playlist/%s' % group_id, group_id,
'Downloading playlist XML')
no_items = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}noItems')
if no_items is not None:
reason = no_items.get('reason')
if reason == 'preAvailability':
msg = 'Episode %s is not yet available' % group_id
elif reason == 'postAvailability':
msg = 'Episode %s is no longer available' % group_id
else:
msg = 'Episode %s is not available: %s' % (group_id, reason)
raise ExtractorError(msg, expected=True)
formats = []
subtitles = None
for item in self._extract_items(playlist):
kind = item.get('kind')
if kind != 'programme' and kind != 'radioProgramme':
continue
title = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}title').text
description = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}summary').text
programme_id = item.get('identifier')
duration = int(item.get('duration'))
def _download_media_selector(self, programme_id):
try:
media_selection = self._download_xml(
'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/%s' % programme_id,
programme_id, 'Downloading media selection XML')
except ExtractorError as ee:
if isinstance(ee.cause, compat_HTTPError) and ee.cause.code == 403:
media_selection = xml.etree.ElementTree.fromstring(ee.cause.read().encode('utf-8'))
else:
raise
formats = []
subtitles = None
for media in self._extract_medias(media_selection):
kind = media.get('kind')
@ -207,6 +251,79 @@ class BBCCoUkIE(SubtitlesInfoExtractor):
elif kind == 'captions':
subtitles = self._extract_captions(media, programme_id)
return formats, subtitles
def _download_playlist(self, playlist_id):
try:
playlist = self._download_json(
'http://www.bbc.co.uk/programmes/%s/playlist.json' % playlist_id,
playlist_id, 'Downloading playlist JSON')
version = playlist.get('defaultAvailableVersion')
if version:
smp_config = version['smpConfig']
title = smp_config['title']
description = smp_config['summary']
for item in smp_config['items']:
kind = item['kind']
if kind != 'programme' and kind != 'radioProgramme':
continue
programme_id = item.get('vpid')
duration = int(item.get('duration'))
formats, subtitles = self._download_media_selector(programme_id)
return programme_id, title, description, duration, formats, subtitles
except ExtractorError as ee:
if not isinstance(ee.cause, compat_HTTPError) and ee.cause.code == 404:
raise
# fallback to legacy playlist
playlist = self._download_xml(
'http://www.bbc.co.uk/iplayer/playlist/%s' % playlist_id,
playlist_id, 'Downloading legacy playlist XML')
no_items = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}noItems')
if no_items is not None:
reason = no_items.get('reason')
if reason == 'preAvailability':
msg = 'Episode %s is not yet available' % playlist_id
elif reason == 'postAvailability':
msg = 'Episode %s is no longer available' % playlist_id
elif reason == 'noMedia':
msg = 'Episode %s is not currently available' % playlist_id
else:
msg = 'Episode %s is not available: %s' % (playlist_id, reason)
raise ExtractorError(msg, expected=True)
for item in self._extract_items(playlist):
kind = item.get('kind')
if kind != 'programme' and kind != 'radioProgramme':
continue
title = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}title').text
description = playlist.find('./{http://bbc.co.uk/2008/emp/playlist}summary').text
programme_id = item.get('identifier')
duration = int(item.get('duration'))
formats, subtitles = self._download_media_selector(programme_id)
return programme_id, title, description, duration, formats, subtitles
def _real_extract(self, url):
group_id = self._match_id(url)
webpage = self._download_webpage(url, group_id, 'Downloading video page')
programme_id = self._search_regex(
r'"vpid"\s*:\s*"([\da-z]{8})"', webpage, 'vpid', fatal=False, default=None)
if programme_id:
player = self._download_json(
'http://www.bbc.co.uk/iplayer/episode/%s.json' % group_id,
group_id)['jsConf']['player']
title = player['title']
description = player['subtitle']
duration = player['duration']
formats, subtitles = self._download_media_selector(programme_id)
else:
programme_id, title, description, duration, formats, subtitles = self._download_playlist(group_id)
if self._downloader.params.get('listsubtitles', False):
self._list_available_subtitles(programme_id, subtitles)
return

View File

@ -0,0 +1,46 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import url_basename
class BehindKinkIE(InfoExtractor):
_VALID_URL = r'http://(?:www\.)?behindkink\.com/(?P<year>[0-9]{4})/(?P<month>[0-9]{2})/(?P<day>[0-9]{2})/(?P<id>[^/#?_]+)'
_TEST = {
'url': 'http://www.behindkink.com/2014/12/05/what-are-you-passionate-about-marley-blaze/',
'md5': '507b57d8fdcd75a41a9a7bdb7989c762',
'info_dict': {
'id': '37127',
'ext': 'mp4',
'title': 'What are you passionate about Marley Blaze',
'description': 'md5:aee8e9611b4ff70186f752975d9b94b4',
'upload_date': '20141205',
'thumbnail': 'http://www.behindkink.com/wp-content/uploads/2014/12/blaze-1.jpg',
'age_limit': 18,
}
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
display_id = mobj.group('id')
webpage = self._download_webpage(url, display_id)
video_url = self._search_regex(
r'<source src="([^"]+)"', webpage, 'video URL')
video_id = url_basename(video_url).split('_')[0]
upload_date = mobj.group('year') + mobj.group('month') + mobj.group('day')
return {
'id': video_id,
'display_id': display_id,
'url': video_url,
'title': self._og_search_title(webpage),
'thumbnail': self._og_search_thumbnail(webpage),
'description': self._og_search_description(webpage),
'upload_date': upload_date,
'age_limit': 18,
}

Some files were not shown because too many files have changed in this diff Show More