Mercurial > codedump

annotate kemonopartydownloader.py @ 116:205fc01d5eb4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
kmbscreens: replace bat+c with py committer: GitHub <noreply@github.com>
author Paper <37962225+mrpapersonic@users.noreply.github.com>
date Sat, 04 Feb 2023 13:46:06 -0500
parents f10492e8720b
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
1 """
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
2 Usage:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
3 kemonopartydownloader.py <url>... (--cookies <filename>)
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
4 [--output <folder>]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
5 [--proxy <proxy>]
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
6 [--timeout <seconds>]
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
7 kemonopartydownloader.py -h | --help
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
8
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
9 Arguments:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
10 <url> Kemono.party URL to download from
99
2bccbf473ff4 kemonopartydownloader.py: --cookies is an argument, not an option
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 98
diff changeset
11 -c --cookies <filename> A Netscape-compatible cookies.txt file
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
12
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
13 Options:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
14 -h --help Show this screen
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
15 -o --output <folder> Output folder, relative to the current directory
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
16 [default: .]
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
17 -p --proxy <proxy> HTTP or HTTPS proxy (SOCKS5 with PySocks)
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
18 -t --timeout <seconds> Time between downloads [default: 1]
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
19 """
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
20 import docopt
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
21 import http.cookiejar
49
717feded576e Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 46
diff changeset
22 import os
717feded576e Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 46
diff changeset
23 import re
717feded576e Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 46
diff changeset
24 import requests # pip install requests
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
25 import time
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
26 import math
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
27 import zipfile
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
28 import urllib.parse
115
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
29 import yt_dlp
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
30 from yt_dlp.utils import sanitize_filename as sanitize
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
31 from urllib.error import HTTPError
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
32 from http.client import BadStatusLine
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
33
51
6eb6c657c7eb Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 49
diff changeset
34
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
35 def download_folder_from_google_drive(link: str) -> int:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
36 takeout_domain = "https://takeout-pa.clients6.google.com"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
37 drive_id = link.split("?")[0].split("/")[-1]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
38 ses = requests.Session()
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
39 ses.headers = {
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
40 'origin': 'https://drive.google.com',
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
41 'content-type': 'application/json',
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
42 }
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
43 key = "AIzaSyC1qbk75NzWBvSaDh6KnsjjA9pIrP4lYIE" # google anonymous key
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
44 takeoutjs = ses.post(takeout_domain + "/v1/exports?key=%s" % (key),
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
45 data='{"items":[{"id":"%s"}]}' % (drive_id)).json()
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
46 takeoutid = str(takeoutjs["exportJob"]["id"])
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
47 storagePath = None
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
48 while storagePath is None:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
49 succeededjson = ses.get(takeout_domain + "/v1/exports/%s?key=%s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
50 % (takeoutid, key)).json()["exportJob"]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
51 if succeededjson["status"] == "SUCCEEDED":
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
52 storagePath = str(succeededjson["archives"][0]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
53 ["storagePath"])
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
54 time.sleep(1)
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
55 size = 0
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
56 for path, dirs, files in os.walk("./%s/Drive - %s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
57 % (output, sanitize(i["title"]))):
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
58 for f in files:
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
59 fp = os.path.join(path, f)
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
60 size += os.path.getsize(fp)
96
d2e0edd4a070 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 56
diff changeset
61 try:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
62 if size >= int(succeededjson["archives"][0]["sizeOfContents"]):
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
63 print(" %s already downloaded!" % (succeededjson["archives"][0]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
64 ["fileName"]))
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
65 return 1
96
d2e0edd4a070 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 56
diff changeset
66 except Exception as e:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
67 print(" %s download failed! %s" % (succeededjson["archives"][0]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
68 ["fileName"], str(e)))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
69 response = ses.get(storagePath, stream=True)
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
70 amountdone = 0
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
71 with open(succeededjson["archives"][0]["fileName"], "wb") as f:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
72 for chunk in response.iter_content(4096):
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
73 if chunk: # filter out keep-alive new chunks
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
74 f.write(chunk)
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
75 amountdone += 4096
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
76 print(" downloading %s: %.2f%%"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
77 % (succeededjson["archives"][0]["fileName"],
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
78 (amountdone / int(succeededjson["archives"][0]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
79 ["compressedSize"])) * 100), end="\r")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
80 print(" downloaded %s: 100.00%% "
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
81 % (succeededjson["archives"][0]["fileName"]))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
82 unzip(succeededjson["archives"][0]["fileName"], "./%s/Drive - %s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
83 % (output,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
84 sanitize(i["title"])))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
85 os.remove(succeededjson["archives"][0]["fileName"])
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
86 return 0
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
87
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
88
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
89 def unzip(src_path: str, dst_dir: str, pwd: str = None) -> None:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
90 with zipfile.ZipFile(src_path) as zf:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
91 members = zf.namelist()
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
92 for member in members:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
93 arch_info = zf.getinfo(member)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
94 arch_name = arch_info.filename.replace('/', os.path.sep)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
95 dst_path = os.path.join(dst_dir, arch_name)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
96 dst_path = os.path.normpath(dst_path)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
97 if not os.path.exists(dst_path):
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
98 zf.extract(arch_info, dst_dir, pwd)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
99
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
100
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
101 def download_from_dropbox(link: str) -> None:
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
102 responsehead = req.head(link.split("?")[0] + "?dl=1", allow_redirects=True)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
103 if responsehead.status_code == 404:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
104 print(" dropbox link not available!")
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
105 return
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
106 filename = urllib.parse.unquote(
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
107 responsehead.headers["Content-Disposition"].split("'")[-1])
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
108 if not os.path.exists(output + "/Dropbox - " + sanitize(i["title"])):
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
109 os.makedirs(output + "/Dropbox - " + sanitize(i["title"]))
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
110 filepath = "%s/Dropbox - %s/%s" % (output, sanitize(i["title"]),
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
111 sanitize(filename))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
112 if os.path.exists(filepath):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
113 print(" file(s) already downloaded!")
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
114 return
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
115 filesize = os.stat(filepath).st_size if os.path.exists(filepath) else 0
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
116 # will always be 0 if it's a folder...
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
117 if filesize == 0:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
118 with req.get(link.split("?")[0] + "?dl=1", stream=True,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
119 headers={"Range": "bytes=%d-" % (filesize)}) as r:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
120 r.raise_for_status()
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
121 with open(filepath, "ab") as f:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
122 for chunk in r.iter_content(chunk_size=4096):
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
123 f.write(chunk)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
124 filesize += 4096
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
125 print(" file %s downloading..." % (filename), end="\r")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
126 print(" file %s successfully downloaded!" % (filename))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
127 if filename.endswith(".zip"):
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
128 unzip(filepath, urllib.parse.unquote(os.path.splitext(filepath)[0]))
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
129 os.remove(filepath)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
130
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
131
115
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
132 def download_from_youtube(link: str) -> int: # int is the response
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
133 return 0 # just a stub for now
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
134
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
135
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
136 # https://stackoverflow.com/a/39225039
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
137 def download_file_from_google_drive(drive_id: str, out: str = "") -> None:
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
138 def get_confirm_token(response: requests.Response):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
139 for key, value in response.cookies.items():
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
140 if key.startswith('download_warning'):
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
141 return value
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
142
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
143 return None
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
144
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
145 def save_response_content(response: requests.Response):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
146 amountdone = 0
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
147 CHUNK_SIZE = 4096
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
148 filename = sanitize(
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
149 response.headers["Content-Disposition"].split("'")[-1])
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
150 folderpath = "%s/Drive - %s/%s" % (output, sanitize(i["title"]), out)
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
151 if not os.path.exists():
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
152 os.makedirs(folderpath)
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
153 destination = "%s/%s" % (folderpath, filename)
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
154 filesize = (os.stat(destination).st_size
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
155 if os.path.exists(destination)
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
156 else 0)
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
157 serverfilesize = int(response.headers["Content-Range"].split('/')[0])
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
158
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
159 if (os.path.exists(destination) and filesize == serverfilesize):
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
160 print(" %s already downloaded!" % os.path.basename(destination))
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
161 return
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
162
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
163 with open(destination, "wb") as f:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
164 for chunk in response.iter_content(CHUNK_SIZE):
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
165 if chunk: # filter out keep-alive new chunks
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
166 f.write(chunk)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
167 amountdone += CHUNK_SIZE
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
168 print(" downloading %s: %.2f%%"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
169 % (os.path.basename(destination),
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
170 (amountdone / serverfilesize)), end="\r")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
171 print(" downloaded %s: %.2f%% "
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
172 % (os.path.basename(destination), 100.0))
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
173
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
174 URL = "https://docs.google.com/uc?export=download"
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
175
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
176 session = requests.Session()
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
177
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
178 headers = {
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
179 "Range": "bytes=0-",
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
180 }
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
181
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
182 session.proxies = req.proxies
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
183
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
184 response = session.get(URL, headers=headers, params={'id': drive_id},
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
185 stream=True)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
186
55
4e5000c9b48f Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 54
diff changeset
187 while response.status_code == 403:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
188 time.sleep(30)
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
189 response = session.get(URL, headers=headers, params={'id': drive_id},
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
190 stream=True)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
191
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
192 if response.status_code == 404:
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
193 return # bypass when root folder has no files
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
194
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
195 token = get_confirm_token(response)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
196
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
197 if token:
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
198 params = {'id': drive_id, 'confirm': token}
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
199 response = session.get(URL, headers=headers, params=params,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
200 stream=True)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
201
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
202 save_response_content(response)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
203
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
204
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
205 def find_urls(s: str) -> list:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
206 url_regex = (r"""http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:""" +
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
207 """%[0-9a-fA-F][0-9a-fA-F]))+""")
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
208 urllist = []
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
209 for findall in re.findall(url_regex, s):
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
210 urllist.append(findall.split("<")[0].split(">")[-1])
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
211 return urllist
51
6eb6c657c7eb Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 49
diff changeset
212
49
717feded576e Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 46
diff changeset
213
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
214 def download_file(i: dict, x: dict, count: int) -> None:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
215 filename = "%s/%s_%dp_%s_%s" % (output, i["id"], count,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
216 sanitize(i["title"]), x["name"])
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
217 amountdone = 0
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
218 filesize = 0
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
219 if os.path.exists(filename):
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
220 filesize = os.path.getsize(filename)
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
221 serverhead = req.head("https://kemono.party/data" + x['path'],
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
222 allow_redirects=True)
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
223 for i in range(500):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
224 serverfilesize = int(serverhead.headers["Content-Length"])
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
225 if filesize < serverfilesize:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
226 amountdone += filesize
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
227 with req.get("https://kemono.party/data" + x['path'], stream=True,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
228 headers={"Range": "bytes=%d-" % (filesize)}) as r:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
229 r.raise_for_status()
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
230 with open(filename, "ab") as f:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
231 for chunk in r.iter_content(chunk_size=4096):
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
232 f.write(chunk)
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
233 amountdone += len(chunk)
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
234 print(" downloading image %d: %.2f%%"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
235 % (count, (amountdone / serverfilesize) * 100),
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
236 end="\r")
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
237 print(" downloaded image " + str(count) + ": 100.00% ")
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
238 return
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
239 else:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
240 print(" image " + str(count) + " already downloaded!")
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
241 return
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
242 time.sleep(10)
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
243 print(" download timed out!")
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
244 return
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
245
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
246
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
247 def parse_json(i: dict, count: int) -> None:
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
248 unique_urls = []
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
249 for url in find_urls(i["content"]):
115
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
250 # spaghetti
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
251 parsed_url = urllib.parse.urlparse(url)
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
252 if parsed_url.netloc == "drive.google.com":
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
253 if parsed_url.path.startswith("/drive/folders"):
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
254 if url not in unique_urls:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
255 download_folder_from_google_drive(url)
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
256 unique_urls.append(url)
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
257 elif (parsed_url.path == "/open" and
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
258 parsed_url.query.startswith == "id"):
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
259 if url not in unique_urls:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
260 download_file_from_google_drive(
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
261 parsed_url.query.split("=")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
262 [-1])
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
263 unique_urls.append(url)
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
264 elif parsed_url.path.startswith("/file/"):
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
265 if url not in unique_urls:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
266 download_file_from_google_drive(parsed_url.path.split("/")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
267 [-2])
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
268 unique_urls.append(url)
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
269 elif parsed_url.netloc in ["dropbox.com", "www.dropbox.com"]:
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
270 if url not in unique_urls:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
271 download_from_dropbox(url)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
272 unique_urls.append(url)
115
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
273 elif parsed_url.netloc in ["youtube.com", "youtu.be", "www.youtube.com"]:
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
274 if url not in unique_urls:
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
275 download_from_youtube(url)
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
276 unique_urls.append(url)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
277 for x in i["attachments"]:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
278 count += 1
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
279 while not os.path.exists("%s/%s_%dp_%s_%s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
280 % (output, i["id"], count,
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
281 sanitize(i["title"]), x["name"])):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
282 try:
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
283 download_file(i, x, count)
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
284 except (HTTPError, BadStatusLine):
56
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
285 while 1:
bde647ac9554 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 55
diff changeset
286 time.sleep(10)
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
287 download_file(i, x, count)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
288 except Exception as e:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
289 print(e)
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
290 time.sleep(timeout)
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
291
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
292
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
293 def get_amount_of_posts(s: str, u: str):
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
294 amount = 0
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
295 while 1:
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
296 data = req.get("https://kemono.party/api/%s/user/%s?o=%d"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
297 % (s, u, amount)).json()
54
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
298 if len(data) < 25:
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
299 return math.ceil((amount + 1) / 25)
5a5d47a795c6 Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 53
diff changeset
300 amount += 25
51
6eb6c657c7eb Update kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 49
diff changeset
301
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
302
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
303 args = docopt.docopt(__doc__)
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
304
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
305 req = requests.Session()
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
306
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
307 if args["--proxy"]:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
308 req.proxies = {
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
309 "http": args["--proxy"],
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
310 "https": args["--proxy"],
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
311 }
97
f1f4f6da04bd kemonopartydownloader.py: convert to percent formatting,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 96
diff changeset
312
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
313 timeout = int(args["--timeout"])
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
314
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
315 cj = http.cookiejar.MozillaCookieJar(args["--cookies"])
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
316 cj.load(ignore_expires=True)
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
317 req.cookies = cj
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
318
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
319 for url in args["<url>"]:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
320 if url.isnumeric():
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
321 print("do not input user IDs in --url! use a link instead")
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
322 continue
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
323
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
324 if url.split("/")[-2] == "post":
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
325 service = url.split("/")[-5]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
326 user = url.split("/")[-3]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
327 post = url.split("/")[-1]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
328 pages = 1
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
329 elif url.split("/")[-2] == "user":
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
330 service = url.split("/")[-3]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
331 user = url.split("/")[-1]
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
332 pages = get_amount_of_posts(service, user)
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
333
100
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
334 output = "%s/%s-%s" % (args["--output"], service, user)
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
335
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
336 if not os.path.exists(output):
b14e2a096ebf kemonopartydownloader.py: add --timeout, fix output
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 99
diff changeset
337 os.mkdir(output)
46
522ad91a230e Add kemonopartydownloader.py
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents:
diff changeset
338
98
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
339 for page in range(pages):
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
340 try:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
341 post
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
342 userdata = req.get("https://kemono.party/api/%s/user/%s/post/%s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
343 % (service, user, post)).json()
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
344 except Exception:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
345 userdata = req.get("https://kemono.party/api/%s/user/%s?o=%s"
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
346 % (service, user, (page * 25))).json()
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
347 for i in userdata:
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
348 print(i["id"])
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
349 count = 0
e4bf37150a3f kemonopartydownloader.py: lint,
Paper <37962225+mrpapersonic@users.noreply.github.com>
parents: 97
diff changeset
350 parse_json(i, count)
115
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
351 filename = "%s/%s_%dp_%s.info.json" % (output, i["id"], count,
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
352 sanitize(i["title"]))
f10492e8720b kemonopartydownloader.py: add youtube downloader stubs and dump json to disk
Paper <mrpapersonic@gmail.com>
parents: 100
diff changeset
353 json.dump(i, filename)