codedump: channeldownloader.py annotate

annotate channeldownloader.py @ 70:eafe13de3f76

Update channeldownloader.py committer: GitHub <noreply@github.com>

author	Paper <37962225+mrpapersonic@users.noreply.github.com>
date	Wed, 01 Jun 2022 07:19:02 -0400
parents	63e6bc911606
children	80bd4a99ea00

rev	line source
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	1 #!/usr/bin/env python3
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	2 #
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	3 # download deleted vids from old yt channels
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	4 # script by paper
69 63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	5 # it's pretty old and could definitely use some refining
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	6
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	7 from __future__ import print_function
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	8 import argparse
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	9 import internetarchive
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	10 try:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	11 import orjson as json
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	12 except ImportError:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	13 import json
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	14 import os
59 a3927b2ec6e6 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 58 diff changeset	15 import re
68 a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	16 import time
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	17 try:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	18 import urllib.request as compat_urllib
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	19 from urllib.error import HTTPError
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	20 except ImportError: # Python 2
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	21 import urllib as compat_urllib
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	22 from urllib2 import HTTPError
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	23 try:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	24 import yt_dlp as youtube_dl
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	25 from yt_dlp.utils import sanitize_filename
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	26 except ImportError:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	27 try:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	28 import youtube_dl
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	29 from youtube_dl.utils import sanitize_filename
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	30 except ImportError:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	31 print("ERROR: youtube-dl/yt-dlp not installed!")
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	32 exit(1)
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	33 from io import open # for Python 2 compatibility, in Python 3 this
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	34 # just maps to the built-in function
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	35
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	36 class MyLogger(object):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	37 def debug(self, msg):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	38 pass
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	39
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	40 def warning(self, msg):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	41 pass
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	42
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	43 def error(self, msg):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	44 pass
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	45
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	46 def ytdl_hook(d):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	47 if d["status"] == "finished":
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	48 print(" downloaded %s: 100% " % (os.path.basename(d["filename"])))
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	49 if d["status"] == "downloading":
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	50 print(" downloading %s: %s\r" % (os.path.basename(d["filename"]), d["_percent_str"]), end="")
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	51 if d["status"] == "error":
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	52 print(" an error occurred downloading {0}!")
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	53
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	54 def load_split_files(path):
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	55 if os.path.isdir(path):
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	56 result = {"videos": []}
69 63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	57 for fi in os.listdir(path):
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	58 for f in re.findall(r"vids.+?\.json", fi):
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	59 with open(path + "/" + f, "r", encoding="utf-8") as infile:
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	60 for i in json.loads(infile.read())["videos"]:
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	61 result["videos"].append(i)
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	62 return result
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	63 else:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	64 return json.loads(open(path, "r", encoding="utf-8").read())
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	65
68 a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	66 def reporthook(count, block_size, total_size):
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	67 global start_time
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	68 if count == 0:
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	69 start_time = time.time()
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	70 return
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	71 duration = time.time() - start_time
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	72 percent = int(count * block_size * 100 / total_size)
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	73 print(" downloading %d%% \r" % (percent), end="")
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	74
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	75
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	76 parser = argparse.ArgumentParser(description="Downloads (deleted) videos from YTPMV creators")
70 eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	77 parser.add_argument("-c", "--channel", help="channel URL", metavar="<url>", required=True)
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	78 parser.add_argument("-d", "--database", help="json database (https://finnrepo.a2hosted.com/YTPMV_Database)", metavar="<path>", required=True)
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	79 parser.add_argument("-o", "--output", help="output directory, defaults to the channel ID", metavar="<output>")
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	80 args = parser.parse_args()
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	81
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	82 if args.channel[:8] == "https://" or args.channel[:7] == "http://":
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	83 channel = args.channel.split("/")[-1]
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	84 else:
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	85 channel = args.channel
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	86
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	87 if args.output:
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	88 output = args.output
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	89 else:
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	90 output = channel
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	91
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	92 if not os.path.exists(output):
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	93 os.mkdir(output)
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	94
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	95 ytdl_opts = {
68 a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	96 "outtmpl": output + "/%(title)s-%(id)s.%(ext)s",
47 00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	97 "retries": 100,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	98 "nooverwrites": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	99 "call_home": False,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	100 "quiet": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	101 "writeinfojson": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	102 "writedescription": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	103 "writethumbnail": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	104 "writeannotations": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	105 "writesubtitles": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	106 "allsubtitles": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	107 "ignoreerrors": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	108 "addmetadata": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	109 "continuedl": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	110 "embedthumbnail": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	111 "format": "bestvideo+bestaudio/best",
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	112 "restrictfilenames": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	113 "no_warnings": True,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	114 "progress_hooks": [ytdl_hook],
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	115 "logger": MyLogger(),
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	116 "ignoreerrors": False,
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	117 }
00403c09455c Add channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: diff changeset	118
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	119 for i in load_split_files(args.database)["videos"]:
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	120 uploader = i["uploader_id"] if "uploader_id" in i else None
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	121 if uploader == channel:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	122 print("%s:" % i["id"])
70 eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	123 # :skull:
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	124 # todo: put this in a function?
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	125 if any(x in os.listdir(output) for x in [sanitize_filename(i["title"] + "-" + i["id"] + ".mp4", restricted=True),
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	126 sanitize_filename(i["title"] + "-" + i["id"] + ".mkv", restricted=True),
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	127 sanitize_filename(i["title"] + "-" + i["id"] + ".webm", restricted=True)]):
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	128 print(" video already downloaded!")
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	129 continue
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	130 # this code is really ugly... todo a rewrite?
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	131 with youtube_dl.YoutubeDL(ytdl_opts) as ytdl:
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	132 try:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	133 result = ytdl.download(["https://youtube.com/watch?v=%s" % i["id"]]) # TODO: add check for existing downloaded items and don't download them
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	134 continue
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	135 except Exception:
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	136 print(" video is not available! attempting to find Internet Archive pages of it...")
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	137 if internetarchive.get_item("youtube-%s" % i["id"]).exists: # download from internetarchive if available
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	138 fnames = [f.name for f in internetarchive.get_files("youtube-%s" % i["id"])]
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	139 flist = []
69 63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	140 for fname in range(len(fnames)):
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	141 if re.search("((?:.+?-)?%s\.(?:mp4\|jpg\|webp\|mkv\|webm\|info\.json\|description))" % (i["id"]), fnames[fname]):
63e6bc911606 Use regex instead of weirdness to filter archive.org names Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 68 diff changeset	142 flist.append(fnames[fname])
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	143 if len(flist) >= 1:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	144 internetarchive.download("youtube-%s" % i["id"], files=flist, verbose=True, destdir=output, no_directory=True, ignore_existing=True)
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	145 else:
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	146 print(" video already downloaded!")
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	147 continue
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	148 if os.path.exists(output + "/" + i["id"] + ".info.json"): # will always exist no matter which setting was used to download
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	149 for fname in flist:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	150 if os.path.exists(output + "/" + fname) and not os.path.exists(output + "/" + sanitize_filename(i["title"], restricted=True) + "-" + fname):
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	151 os.rename(output + "/" + fname, output + "/" + sanitize_filename(i["title"], restricted=True) + "-" + fname)
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	152 else:
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	153 print("ID file not found!")
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	154 else:
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	155 print(" video does not have a Internet Archive page! attempting to download from the Wayback Machine...")
70 eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	156 try: # we could use yt-dlp's extractor, but then we would need to craft a fake wayback machine url,
eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	157 # and we wouldn't even know if it worked. so let's continue using our little "hack"
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	158 headers = compat_urllib.urlopen("https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/%s" % i["id"])
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	159 if hasattr(headers.info(), "getheader"):
9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	160 contenttype = headers.info().getheader("Content-Type")
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	161 else:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	162 contenttype = headers.getheader("Content-Type")
68 a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	163 if contenttype == "video/webm":
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	164 ext = "webm"
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	165 elif contenttype == "video/mp4":
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	166 ext = "mp4"
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	167 else:
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	168 raise HTTPError(url=None, code=None, msg=None, hdrs=None, fp=None)
a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	169 compat_urllib.urlretrieve("https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/%s" % i["id"], "%s/%s-%s.%s" % (output, sanitize_filename(i["title"], restricted=True), i["id"], ext), reporthook)
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	170 print(" downloaded %s-%s.%s" % (sanitize_filename(i["title"], restricted=True), i["id"], ext))
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	171 except HTTPError:
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	172 print(" video not available on the Wayback Machine!")
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	173 except Exception as e:
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	174 print(" unknown error downloading video!\n")
61 c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	175 print(e)
c615532e6572 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 60 diff changeset	176 # metadata
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	177 with open("%s/%s-%s.info.json" % (output, sanitize_filename(i["title"], restricted=True), i["id"]), "w", encoding="utf-8") as jsonfile:
70 eafe13de3f76 Update channeldownloader.py Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 69 diff changeset	178 jsonfile.write(json.dumps(i).decode("utf-8"))
67 9636d5dee08c [channeldownloader.py] Python 2.7 compatibility Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 61 diff changeset	179 print(" saved %s" % os.path.basename(jsonfile.name))
68 a43ed076b28f [channeldownloader.py] Implement HTTPError to circumvent Python 2 weirdness Paper <37962225+mrpapersonic@users.noreply.github.com> parents: 67 diff changeset	180

Mercurial > codedump

annotate channeldownloader.py @ 70:eafe13de3f76