Mercurial > channeldownloader

annotate channeldownloader.py @ 14:03c8fd4069fb default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
*: big refactor, switch to GPLv2, and add README Okay: now, we use a modular approach for downloaders. Each downloader is provided through a single function (which does the fetching). Additionally, the internetarchive library is optional now if the user does not want to install it. yt-dlp is still necessary though for it's sanitize_filename function. If and when I get to adding vanity features (such as finding the best possible source by comparing resolution and bitrate), I'll probably separate out all of the downloaders into different files. I also moved this project to a separate repository from 'codedump', keeping all of the relevant commit history :)
author Paper <paper@tflc.us>
date Sat, 30 Aug 2025 17:09:56 -0400
parents 2e7a3725ad21
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5
d4740dc7470c [channeldownloader.py] Python 2.7 compatibility
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 4
diff changeset
1 #!/usr/bin/env python3
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
2 # -*- coding: utf-8 -*-
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
3 # channeldownloader.py - scrapes youtube videos from a channel from
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
4 # a variety of sources
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
5
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
6 # Copyright (c) 2021-2025 Paper <paper@tflc.us>
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
7 # This program is free software: you can redistribute it and/or modify
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
8 # it under the terms of the GNU General Public License as published by
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
9 # the Free Software Foundation, either version 2 of the License, or
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
10 # (at your option) any later version.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
11 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
12 # This program is distributed in the hope that it will be useful,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
13 # but WITHOUT ANY WARRANTY; without even the implied warranty of
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
14 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
15 # GNU General Public License for more details.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
16 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
17 # You should have received a copy of the GNU General Public License
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
18 # along with this program. If not, see <http://www.gnu.org/licenses/>.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
19
9
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
20 """
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
21 Usage:
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
22 channeldownloader.py <url>... (--database <file>)
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
23 [--output <folder>]
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
24 channeldownloader.py -h | --help
5
d4740dc7470c [channeldownloader.py] Python 2.7 compatibility
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 4
diff changeset
25
9
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
26 Arguments:
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
27 <url> YouTube channel URL to download from
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
28
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
29 Options:
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
30 -h --help Show this screen
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
31 -o --output <folder> Output folder, relative to the current directory
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
32 [default: .]
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
33 -d --database <file> yt-dlp style database of videos. Should contain
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
34 an array of yt-dlp .info.json data. For example,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
35 FinnOtaku's YTPMV metadata archive.
9
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
36 """
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
37
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
38 # Built-in python stuff (no possible missing dependencies)
5
d4740dc7470c [channeldownloader.py] Python 2.7 compatibility
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 4
diff changeset
39 from __future__ import print_function
9
2e9ed463c0be Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 8
diff changeset
40 import docopt
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
41 import os
2
c65d14f01453 Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 1
diff changeset
42 import re
6
5d93490e60e2 [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 5
diff changeset
43 import time
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
44 import urllib.request
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
45 import os
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
46 import ssl
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
47 from urllib.error import HTTPError
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
48 from pathlib import Path
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
49
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
50 # We can utilize special simdjson features if it is available
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
51 simdjson = False
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
52
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
53 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
54 import simdjson as json
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
55 simdjson = True
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
56 print("INFO: using simdjson")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
57 except ImportError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
58 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
59 import ujson as json
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
60 print("INFO: using ujson")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
61 except ImportError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
62 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
63 import orjson as json
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
64 print("INFO: using orjson")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
65 except ImportError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
66 import json
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
67 print("INFO: using built-in json (slow!)")
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
68
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
69 ytdlp_works = False
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
70
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
71 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
72 import yt_dlp as youtube_dl
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
73 from yt_dlp.utils import sanitize_filename, DownloadError
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
74 ytdlp_works = True
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
75 except ImportError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
76 print("failed to import yt-dlp!")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
77 print("downloading from YouTube directly will not work.")
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
78
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
79 ia_works = False
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
80
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
81 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
82 import internetarchive
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
83 from requests.exceptions import ConnectTimeout
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
84 ia_works = True
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
85 except ImportError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
86 print("failed to import the Internet Archive's python library!")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
87 print("downloading from IA will not work.")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
88
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
89 ##############################################################################
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
90 ## DOWNLOADERS
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
91
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
92 # All downloaders should be a function under this signature:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
93 # dl(video: dict, basename: str, output: str) -> int
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
94 # where:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
95 # 'video': the .info.json scraped from the YTPMV metadata archive.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
96 # 'basename': the basename output to write as.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
97 # 'output': the output directory.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
98 # yes, it's weird, but I don't care ;)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
99 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
100 # Magic return values:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
101 # 0 -- all good, video is downloaded
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
102 # 1 -- error downloading video; it may still be available if we try again
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
103 # 2 -- video is proved totally unavailable here. give up
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
104
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
105
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
106 # Basic downloader template.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
107 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
108 # This does a brute-force of all extensions within vexts and iexts
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
109 # in an attempt to find a working video link.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
110 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
111 # linktemplate is a template to be created using the video ID and
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
112 # extension. For example:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
113 # https://cdn.ytarchiver.com/%s.%s
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
114 def basic_dl_template(video: dict, basename: str, output: str,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
115 linktemplate: str, vexts: list, iexts: list) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
116 # actual downloader
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
117 def basic_dl_impl(vid: str, ext: str) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
118 url = (linktemplate % (vid, ext))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
119 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
120 with urllib.request.urlopen(url) as headers:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
121 with open("%s.%s" % (basename, ext), "wb") as f:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
122 f.write(headers.read())
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
123 print(" downloaded %s.%s" % (basename, ext))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
124 return 0
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
125 except TimeoutError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
126 return 1
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
127 except HTTPError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
128 return 2
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
129 except Exception as e:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
130 print(" unknown error downloading video!")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
131 print(e)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
132 return 1
4
aa652a6f97af Update channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 3
diff changeset
133
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
134 for exts in [vexts, iexts]:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
135 for ext in exts:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
136 r = basic_dl_impl(video["id"], ext)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
137 if r == 0:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
138 break # done!
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
139 elif r == 1:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
140 # timeout; try again later?
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
141 return 1
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
142 elif r == 2:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
143 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
144 else:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
145 # we did not break out of the loop
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
146 # which means all extensions were unavailable
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
147 return 2
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
148
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
149 # video was downloaded successfully
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
150 return 0
6
5d93490e60e2 [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 5
diff changeset
151
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
152
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
153 # GhostArchive, basic...
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
154 def ghostarchive_dl(video: dict, basename: str, output: str) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
155 return basic_dl_template(video, basename, output,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
156 "https://ghostvideo.b-cdn.net/chimurai/%s.%s",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
157 ["mp4", "webm", "mkv"],
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
158 [] # none
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
159 )
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
160
0
d098a293a02d Add channeldownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
161
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
162 # media.desirintoplaisir.net
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
163 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
164 # holds PRIMARILY popular videos (i.e. no niche internet microcelebrities)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
165 # or weeb shit, however it seems to be growing to other stuff.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
166 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
167 # there isn't really a proper API; I've based the scraping off of the HTML
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
168 # and the public source code.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
169 def desirintoplaisir_dl(video: dict, basename: str, output: str) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
170 return basic_dl_template(video, basename, output,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
171 "https://media.desirintoplaisir.net/content/%s.%s",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
172 ["mp4", "webm", "mkv"],
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
173 ["webp"]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
174 )
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
175
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
176
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
177 # Internet Archive's Wayback Machine
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
178 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
179 # Internally, IA's javascript routines forward to the magic
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
180 # URL used here.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
181 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
182 # TODO: Download thumbnails through the CDX API:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
183 # https://github.com/TheTechRobo/youtubevideofinder/blob/master/lostmediafinder/finder.py
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
184 # the CDX API is pretty slow though, so it should be used as a last resort.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
185 def wayback_dl(video: dict, basename: str, output: str) -> int:
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
186 try:
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
187 url = ("https://web.archive.org/web/2oe_/http://wayback-fakeurl.archiv"
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
188 "e.org/yt/%s" % video["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
189 with urllib.request.urlopen(url) as headers:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
190 contenttype = headers.getheader("Content-Type")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
191 if contenttype == "video/webm" or contenttype == "video/mp4":
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
192 ext = contenttype.split("/")[-1]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
193 else:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
194 raise HTTPError(url=None, code=None, msg=None,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
195 hdrs=None, fp=None)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
196 with open("%s.%s" % (basename, ext), "wb") as f:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
197 f.write(headers.read())
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
198 print(" downloaded %s.%s" % (basename, ext))
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
199 return 0
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
200 except TimeoutError:
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
201 return 1
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
202 except HTTPError:
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
203 # dont keep trying
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
204 return 2
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
205 except Exception as e:
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
206 print(" unknown error downloading video!")
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
207 print(e)
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
208 return 1
11
1ac85f6f40c4 channeldownloader: insane memory optimizations
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 10
diff changeset
209
1ac85f6f40c4 channeldownloader: insane memory optimizations
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 10
diff changeset
210
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
211 # Internet Archive (tubeup)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
212 def ia_dl(video: dict, basename: str, output: str) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
213 def ia_file_legit(file: internetarchive.File, vidid: str) -> bool:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
214 # FIXME:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
215 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
216 # There are some items on IA that combine the old tubeup behavior
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
217 # (i.e., including the sanitized video name before the ID)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
218 # and the new tubeup behavior (filename only contains the video ID)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
219 # hence we will download the entire video twice.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
220 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
221 # This isn't much of a problem anymore (and hasn't been for like 3
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
222 # years), since I contributed code to not upload something if there
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
223 # is already something there. However we should handle this case
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
224 # anyway.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
225 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
226 # Additionally, there are some items that have duplicate video files
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
227 # (from when the owners changed the title). We should ideally only
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
228 # download unique files. IA seems to provide SHA1 hashes...
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
229 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
230 # We should also check if whether the copy on IA is higher quality
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
231 # than a local copy... :)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
232 if not re.search(r"((?:.+?-)?" + vidid + r"\.(?:mp4|jpg|webp|mkv|w"
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
233 r"ebm|info\.json|description|annotations.xml))",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
234 f.name):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
235 return False
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
236
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
237 # now, check the metadata
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
238 print(f)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
239 return True
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
240
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
241
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
242 if not internetarchive.get_item("youtube-%s" % video["id"]).exists:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
243 return 2
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
244
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
245 flist = [
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
246 f.name
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
247 for f in internetarchive.get_files("youtube-%s" % video["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
248 if ia_file_legit(f.name, video["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
249 ]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
250 while True:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
251 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
252 internetarchive.download("youtube-%s" % video["id"], files=flist,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
253 verbose=True, destdir=output,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
254 no_directory=True, ignore_existing=True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
255 retries=9999)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
256 break
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
257 except ConnectTimeout:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
258 time.sleep(1)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
259 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
260 except Exception as e:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
261 print(e)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
262 return 1
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
263
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
264 # Newer versions of tubeup save only the video ID.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
265 # Account for this by replacing it.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
266 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
267 # paper/2025-08-30: fixed a bug where video IDs with hyphens
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
268 # would incorrectly truncate
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
269 for fname in flist:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
270 # ignore any files whose names are not simply the ID
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
271 if os.path.splitext(fname)[0] != video["id"]:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
272 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
273
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
274 if os.path.exists("%s/%s" % (output, fname)):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
275 os.replace("%s/%s" % (output, fname),
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
276 "%s.%s" % (basename, os.path.splitext(fname))[1])
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
277 return 0
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
278
11
1ac85f6f40c4 channeldownloader: insane memory optimizations
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 10
diff changeset
279
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
280 def ytdlp_dl(video: dict, basename: str, output: str) -> int:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
281 # intentionally ignores all messages besides errors
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
282 class MyLogger(object):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
283 def debug(self, msg):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
284 pass
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
285
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
286 def warning(self, msg):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
287 pass
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
288
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
289 def error(self, msg):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
290 print(" " + msg)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
291 pass
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
292
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
293
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
294 def ytdl_hook(d) -> None:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
295 if d["status"] == "finished":
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
296 print(" downloaded %s: 100%% " % (os.path.basename(d["filename"])))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
297 if d["status"] == "downloading":
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
298 print(" downloading %s: %s\r" % (os.path.basename(d["filename"]),
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
299 d["_percent_str"]), end="")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
300 if d["status"] == "error":
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
301 print("\n an error occurred downloading %s!"
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
302 % (os.path.basename(d["filename"])))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
303
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
304 ytdl_opts = {
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
305 "retries": 100,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
306 "nooverwrites": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
307 "call_home": False,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
308 "quiet": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
309 "writeinfojson": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
310 "writedescription": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
311 "writethumbnail": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
312 "writeannotations": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
313 "writesubtitles": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
314 "allsubtitles": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
315 "addmetadata": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
316 "continuedl": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
317 "embedthumbnail": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
318 "format": "bestvideo+bestaudio/best",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
319 "restrictfilenames": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
320 "no_warnings": True,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
321 "progress_hooks": [ytdl_hook],
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
322 "logger": MyLogger(),
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
323 "ignoreerrors": False,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
324
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
325 #mm, output template
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
326 "outtmpl": output + "/%(title)s-%(id)s.%(ext)s",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
327 }
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
328
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
329 with youtube_dl.YoutubeDL(ytdl_opts) as ytdl:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
330 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
331 ytdl.extract_info("https://youtube.com/watch?v=%s" % video["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
332 return 0
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
333 except DownloadError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
334 return 2
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
335 except Exception as e:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
336 print(" unknown error downloading video!\n")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
337 print(e)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
338
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
339 return 1
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
340
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
341
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
342 # TODO: There are multiple other youtube archival websites available.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
343 # Most notable is https://findyoutubevideo.thetechrobo.ca .
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
344 # This combines a lot of sparse youtube archival services, and has
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
345 # a convenient API we can use. Nice!
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
346 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
347 # There is also the "Distributed YouTube Archive" which is totally
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
348 # useless because there's way to automate it...
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
349
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
350 ##############################################################################
11
1ac85f6f40c4 channeldownloader: insane memory optimizations
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 10
diff changeset
351
1ac85f6f40c4 channeldownloader: insane memory optimizations
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 10
diff changeset
352
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
353 def main():
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
354 # generator; creates a list of files, and returns the parsed form of
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
355 # each. note that the parser is not necessarily
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
356 def load_split_files(path: str):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
357 list_files = []
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
358
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
359 # build the path list
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
360 if not os.path.isdir(path):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
361 list_files.append(path)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
362 else:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
363 for fi in os.listdir(path):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
364 if re.search(r"vids[0-9\-]+?\.json", fi):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
365 list_files.append(path + "/" + fi)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
366
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
367 # now open each as a json
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
368 for fi in list_files:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
369 print(fi)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
370 with open(fi, "r", encoding="utf-8") as infile:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
371 if simdjson:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
372 # Using this is a lot faster in SIMDJSON, since instead
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
373 # of converting all of the JSON key/value pairs into
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
374 # native Python objects, they stay in an internal state.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
375 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
376 # This means we only get the stuff we absolutely need,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
377 # which is the uploader ID, and copy everything else
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
378 # if the ID is one we are looking for.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
379 parser = json.Parser()
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
380 yield parser.parse(infile.read())
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
381 del parser
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
382 else:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
383 yield json.load(infile)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
384
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
385
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
386 def write_metadata(i: dict, basename: str) -> None:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
387 # ehhh
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
388 if not os.path.exists(basename + ".info.json"):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
389 with open(basename + ".info.json", "w", encoding="utf-8") as jsonfile:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
390 try:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
391 # orjson outputs bytes
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
392 jsonfile.write(json.dumps(i).decode("utf-8"))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
393 except AttributeError:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
394 # everything else outputs a string
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
395 jsonfile.write(json.dumps(i))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
396 print(" saved %s" % os.path.basename(jsonfile.name))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
397 if not os.path.exists(basename + ".description"):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
398 with open(basename + ".description", "w",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
399 encoding="utf-8") as descfile:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
400 descfile.write(i["description"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
401 print(" saved %s" % os.path.basename(descfile.name))
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
402
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
403 args = docopt.docopt(__doc__)
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
404
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
405 if not os.path.exists(args["--output"]):
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
406 os.mkdir(args["--output"])
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
407
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
408 channels = dict()
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
409
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
410 for url in args["<url>"]:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
411 chn = url.split("/")[-1]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
412 channels[chn] = {"output": "%s/%s" % (args["--output"], chn)}
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
413
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
414 for channel in channels.values():
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
415 if not os.path.exists(channel["output"]):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
416 os.mkdir(channel["output"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
417
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
418 # find videos in the database.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
419 #
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
420 # despite how it may seem, this is actually really fast, and fairly
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
421 # memory efficient too (but really only if we're using simdjson...)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
422 videos = [
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
423 i if not simdjson else i.as_dict()
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
424 for f in load_split_files(args["--database"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
425 for i in (f if not "videos" in f else f["videos"]) # logic is reversed kinda, python is weird
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
426 if "uploader_id" in i and i["uploader_id"] in channels
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
427 ]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
428
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
429 while True:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
430 if len(videos) == 0:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
431 break
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
432
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
433 videos_copy = videos
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
434
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
435 for i in videos_copy:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
436 channel = channels[i["uploader_id"]]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
437
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
438 # precalculated for speed
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
439 output = channel["output"]
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
440
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
441 print("%s:" % i["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
442 basename = "%s/%s-%s" % (output, sanitize_filename(i["title"],
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
443 restricted=True), i["id"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
444 files = [y
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
445 for p in ["mkv", "mp4", "webm"]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
446 for y in Path(output).glob(("*-%s." + p) % i["id"])]
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
447 if files:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
448 print(" video already downloaded!")
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
449 videos.remove(i)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
450 write_metadata(i, basename)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
451 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
452
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
453 # high level "download" function.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
454 def dl(video: dict, basename: str, output: str):
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
455 dls = []
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
456
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
457 if ytdlp_works:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
458 dls.append({
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
459 "func": ytdlp_dl,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
460 "name": "using yt-dlp",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
461 })
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
462
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
463 if ia_works:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
464 dls.append({
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
465 "func": ia_dl,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
466 "name": "from the Internet Archive",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
467 })
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
468
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
469 dls.append({
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
470 "func": desirintoplaisir_dl,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
471 "name": "from LMIJLM/DJ Plaisir's archive",
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
472 })
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
473 dls.append({
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
474 "func": ghostarchive_dl,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
475 "name": "from GhostArchive"
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
476 })
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
477 dls.append({
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
478 "func": wayback_dl,
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
479 "name": "from the Wayback Machine"
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
480 })
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
481
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
482 for dl in dls:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
483 print(" attempting to download %s" % dl["name"])
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
484 r = dl["func"](i, basename, output)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
485 if r == 0:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
486 # all good, video's downloaded
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
487 return 0
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
488 elif r == 2:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
489 # video is unavailable here
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
490 print(" oops, video is not available there...")
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
491 continue
14
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
492 elif r == 1:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
493 # error while downloading; likely temporary.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
494 # TODO we should save which downloader the video
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
495 # was on, so we can continue back at it later.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
496 return 1
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
497 # video is unavailable everywhere
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
498 return 2
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
499
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
500 r = dl(i, basename, output)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
501 if r == 1:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
502 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
503
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
504 # video is downloaded, or it's totally unavailable, so
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
505 # remove it from being checked again.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
506 videos.remove(i)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
507 # ... and then dump the metadata, if there isn't any on disk.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
508 write_metadata(i, basename)
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
509
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
510 if r == 0:
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
511 # video is downloaded
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
512 continue
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
513
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
514 # video is unavailable; write out the metadata.
03c8fd4069fb *: big refactor, switch to GPLv2, and add README
Paper <paper@tflc.us>
parents: 13
diff changeset
515 print(" video is unavailable everywhere; dumping out metadata only")
6
5d93490e60e2 [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 5
diff changeset
516
10
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
517
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
518 if __name__ == "__main__":
8969930a9fa4 *: major cleanup
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 9
diff changeset
519 main()