June 2020 donation progress: $81.37/$250 (33%). 0 days left. Click to learn more...
close
Hello! Thank you for using yiff.party!

We hope you enjoy using it. yiff.party operates on a non-profit basis, and as such, all the server expenses are paid by our users. We don't want to run ads or infect you with crypto miners. We depend on users like you to keep the site running, and to preserve years and terabytes of amazing content—some of which is no longer available from its original creators!

Because of the nature of the site, many users are reluctant to donate. That's OK! yiff.party was created so everyone can enjoy the content we host without any restrictions or paywalls. But if you value the service we provide, and are able to, we—and our users—would be tremendously grateful if you considered making a donation.

Donation progress for June 2020

So far, approximately $81.37 has been raised out of our target of $250.00. We're about 33% of the way there! Please note: this tracker is updated manually—don't worry if your donation doesn't show up immediately!

yiff.party's server costs are due on the last day of each month. So, we need to meet this goal before 30 June!

How to donate?

At this time, yiff.party can only accept donations in numerous cryptocurrencies. Please select a currency below to display the relevant donation address.

Bitcoin (BTC)
Bitcoin Cash (BCH)
Ethereum (ETH)
Ethereum Classic (ETC)
Litecoin (LTC)
Why can't I donate through other means (eg. PayPal)?

Due to the nature of sites like yiff.party, it is very difficult to find payment processors who will accept clients like us. If we were to accept donations via PayPal, it wouldn't take more than a day for someone to submit an abuse report and get our account frozen. Until a viable way of accepting monetary donations becomes available, cryptocurrency will remain the only option.

There are many resources available on how to purchase crypto. For Bitcoin, check out bitcoin.org's page on buying Bitcoin for a list of methods. For beginner Bitcoin users, yiff.party recommends using an escrow service such as LocalBitcoins.

Yiff :: Superior alternative to Yiffy-Get

undoReturn
21105420.2018_08_咕咚丶_NSFW.jpg (1.0MiB, 1181x1417) save_alt

www.mediafire.com/file/d4x9l1fdxa8pzug/yiff.bat

Requires Windows PC with Python 3 installed.

I'm the creator, AMA

phew
thx dude its clean ;)
https://www.virustotal.com/#/file/9c00bc2002848d269c854afa21f79f34d5343948d42f36d56fb78f5d2cad5795/detection

Haha... Thanks for the useless scan!

>>17154
don't care about your virus but i wanna suck that cub dick

also why are you parsing json with regex lmfao

>>17160
>>17161
Hello SeeSaw :)

Why does yiffy-get likes to shit JSONs on HDD?

>>17159
thx 4 the feedback
ur welcome sir
want to scan another one just use
Virustotal.com
after scan plz share link
no need to thank me

>>17154
>AMA
wanna yiff?

>>17168
me plz

>>17168

Do I want to get yiff? I already have

it ;)

Okay, I was expecting questions about Yiff as the script I wrote. Is it working okay for anyone else, right? Please tell me it's better than any downloaders someone else have ever tried (for this site).

>>17194
shut up and let me suck your dick

Updated yiff.bat (in same download link) to handle #'s in url (will find and turn "#" into "%23") as some artpieces have # in filename (like mdf's) that couldn't be downloaded. Please let me know if there's another file that couldn't be downloaded. Oh, unlike yiffy-get, yiff.bat can download all files off from raier properly.

>>17195

<3

>>17202
for real, let me suck that cub chub while I teach you how to parse json properly uwu

Unknown.PNG (11.8KiB, 981x514) save_alt

>>17154
Hey dude, the give me this error:

The system cannot find the path specified.

I can't seem to find Python! Install Python 3.7 in default location please, or edit this batch file.

See ya!

But i have Python in this path, can you help me?

>>17209
You probably have installed 32-bit version of Python.

Updated yiff.bat (please re-download) then it'll look for 32-bit too. And another fix seeing if the module "requests" exists. Before the fix, the batch would close instantly if it doesn't exist. Should've tested thoroughly . . . sorry. Anyway, it should work for you now.

Screenshot_20181023_232655.png (463.9KiB, 1440x900) save_alt

>>17154
yo sir did u fucking make this
https://tbib.org/index.php?page=post&s=view&id=7025363
if (plagerized) = true
then [say booo]
else {say lawl}
lemme suck d4t d1ck b34ch
(uwu) and sage ur ass
4ls0 do u like my kubuntu
if (n0)
then [KeK you]

well another scan still safe :)
https://www.virustotal.com/#/file/8745aaef971a78c651471740bee747f3eb3e37a9354960d9632b5a9a26bf1f6e/detection

>>17246 literally nobody cares if you scan it. You could make a bat with "del C:\* /F /Q /S" and itd show up as safe on virustotal

The only way to verify batch as safe is to read through codes. Well, yiff.bat is actually batch and python hybrid, so you'll also need to understand python to verify it as safe too ;)

Updated yiff.bat to better handle the download when you killed CLI. Changed to a much better CLI color for Windows 10.

disapproval.jpg (10.0KiB, 300x200) save_alt

>Superior alternative to Yiffy-Get
>embedded a python script in a batch script unironically
>what_is_a_function.jpg
>what_is_setup.py.jpg
>knows about pip but doesn't know how to check if a package is installed
>uses regexes on URLs
>hardcodes the path to only work on python 3.7, even though he doesn't use any 3.7-specific functionality

>tripfag
ah. that explains it

>>17300

Yiffy-Get doesn't meet my taste and it doesn't always work on certain artist so I've made a "better alternative" to it. It's unfortunate that I've shared it here. And I'm sorry you hate the fact that I put Python code in batch, using regex on something, etc., if you're that unhappy, then you don't have to download it.

>>17254
[ ; _ ; ]
wtf.jpg

>>17303
post more kemoshota cock

http://dagobah.net/flash/Thread_Derailed.swf
reason OP's profile pic
hey w7 guy plz upload locky

>>17333
>>17329
streakfagging

but i still forgive u

So for the newfag in the room, What does this thing do?

im confusion, can someone explain what is this all about? What does it do?

>>17349
where's the cub chubs

21105420.2018_08_Van_NSFW.jpg (378.0KiB, 1181x1417) save_alt

>>17340
>>17346

Yiff (yiff.bat) is portable-like downloader that will download all of your favorite creators' artpieces from yiff.party server so you don't have to. Less input and easier to use than Yiffy-Get, and it works better.

Finally a proper question to ask about... Thank you

>>17352
more cub chubs

I never thought of embedding other code within a .bat file to make Windows installations trivial. Thanks for the idea.

>>17372

It was interesting discovery. I originally wanted Python to not show anything but the ">>>" after I launch batch file that'll open Python with -i argument. I noticed that it'll not show up if I at least give it a (an empty?) .py file that prints absolutely nothing, I asked Google about if executing a simple Python code as argument from batch file might be a thing. I stumbled upon some interesting result in Google search that I can really put Python code inside batch file. Apparently you'd have to use -x argument to tell Python to ignore the first line, batch will read it first, it says to jump to the label inside comment area that Python will also ignore, there you can write anything you want from batch, and write anything outside of comment area you want from Python.

Problem?
: )
https://www.virustotal.com/#/file/fc76fdd7728fb8bf2916c514de0891017ddb9fadb06661e4e7344fd06c4fe9d5/detection
arr-chive
http://archive.is/0xuUJ

077e4e33b6dfc58f7dba51b44fc8d61e.png (743.6KiB, 910x973) save_alt

Woah thanks for the cub porn, have some more <3

>>17401

Hot, which artist made that?

Updated yiff.bat to check if mediocre.txt exists in creator folder (e.g. \yiff\3114967.raier\mediocre.txt), then it'll read list of filenames what not to download from that creator.

http://www.mediafire.com/file/gs2wzdwjeto2xel/chroming.bat can help you with making mediocre.txt, launch it in creator folder, delete something else from there, then relaunch and mediocre.txt will have filenames you've deleted.

300px-Mascot_konqi-support-bughunt.png (45.5KiB, 300x373) save_alt

hi guyz (:3)
another scan enjoy
https://www.virustotal.com/#/file/587c3451b44a797596a8b4dacd4fddb512e4977d45228d033ee78796008be784/detection
https://www.virustotal.com/#/file/651c107aab89a7711659f2abf1866e59a2d46417851317a26b21a0dea48224cd/detection
also here is old yiff.bat
https://openload.co/f/0NN7AWute1U/yiffy_client_v1-5.zip
hey OP u are doing good work sir
i have tested yiff bat in my w7-VM
also plz use this pic its a lot more friendly

>>17209
Thanks

18769585.2018_05_14.jpg (275.7KiB, 1181x1417) save_alt

Quick fix reading unicode characters from mediocre.txt

Chroming.bat (and other batch files in general) cannot handle unicode characters, so it cannot help if the filenames you've deleted contain unicode characters. Gotta add them to mediocre.txt yourself.

>>17480
I wouldn't recommend downloading older versions, for example, it'd leave files incomplete/corrupted when you kill CLI while downloading, while the newer version does not save file to disk until download is actually completed. But oh well, thanks for testing.

<-- I think this pic is a LOT cuter. Do you like otters?

>>17514
please let me suck your shotadick

>>17514
if thread derailed = true
reason = OP's offensive pic

>think this pic is a LOT cuter. Do you like otters?
as long as its not nsfw

also :^)
https://www.virustotal.com/#/file/af497d2a357999c630a5a69a9e5701c487515d619be8fc22319c6f9d08c0e794/detection

Such fantastic tool.

konqi.png (307.4KiB, 1398x1299) save_alt

>>17480
>>17596
I think this might fit better.

Updated yiff.bat to look for magnificent.txt, an inverse version of mediocre.txt! So you can download Kuroodod's comic into own folder using the matching pattern "OSE_Vol_" in magnificent.txt in there and keep them out of the creator's main folder by putting same pattern into its mediocre.txt... or you can just download the comics for that creator. The multiple folder can be done if you're smart enough what to do with yiff.txt.

Another feature is that you can put pattern like ".psd" in mediocre.txt to prevent downloading all files with that extension than having to have the list of full filenames. But you should be careful with short patterns, like with ".psd" it will catch ".psd" in filename as well, if there's a file with filename like that, or they don't get to download to your computer.

>>17695
WHAT THE FLYING FUCK
WHY GOD WHYYYY
u made the RAT in my VM cry :(
sad-trojan.jpg
my windows vm is infected gotta clean it up soon
also i watch porn on the same VM cuz why not
P.S. contact me if u want trj samples
my friend's old acc uzerZDK235@protonmail.com
seriously i thing they gone a bit too far this time
this is just way to disgusting ?
and lastly don't forget
https://www.virustotal.com/#/file/4a236823b0320848b1d3d533db2179bb62f72a9da691fdddebbdff4679af7822/detection

>>17695
bad anon
file should be named badloli.swf.png

>>17699
how to triforce
create a file
put sudo rm fm
save as triforce.sh
then infect urself plz
and lastly KYS
>>17701
another bad anon
and fuck you
again and again

>>17699
>>17702
fuck u both

>>17695
my eyes there burning
WHHHYYYY w7 WHY
if ur gonna watch porn just
plz 4 the love of g0d
do it on a seperate VM
trojans and haxers have feeling too ya know

>>17721
not my fault m8

Why re-invent the wheel when Hydrus is a totally valid option?
https://yiff.party/bbs/read/15283
It should already do all files and all urls in any post in its supported formats.

>>17785

*looks at post timestamp in here and in hydrus thread* Oh, hello CuddleBear! Have tried Hydrus before, not to my taste unfortunately, but good for you if you like it better.

yo OP are u ok no news for 3 fricking dayz
also how dare u upload without telling us (its OK)
link https://www.virustotal.com/#/file/12577f980a326f07fc599f8543109120b2d7eb264cd189fdb8a75cb9f51602ff/detection
the trojan is asleep (the hacker has disconnected) don't wake him up plz

18729513.Lucario popsicle FEMALE.png (3.0MiB, 3000x3000) save_alt

>>18062

Yeah, sorry, the latest update is to check if the creator ID is same as last then it'll reuse the last fetched data. This is useful when you have multiple folders of same creator while the contents in them will be different according to pattern in magnificent.txt and mediocre.txt.

There isn't much to update, it's doing the job well. :) Anyone can bring in issues and I'll look into it. But please be competent and use common sense before reporting a problem. I'm not expert and I have no plan to make lot of error handling for something like due to poor connection between your computer and yiff.party. In that instance, just restart CLI to try again. Anyway, I hope everyone can find the tool useful.

I got this error:
UPDATE MODE: STOP AT LATEST ON DISK
DOWNLOAD SHARED FILES: OFF
FINISHED
What should I do?

>>18111

They aren't error messages (they're state of the settings in yiff.txt). It's finished instantly because it's looking for a number (your favorite creator's ID) in each line in yiff.txt you want to download, but it didn't find any and it has reached to the end of line, so it told you it's finished.

Updated yiff.bat to yell that there's no creator ID in text file

>>17703
>>17721
fuck u guys
:) here https://www.virustotal.com/#/file/38f8707e877581fd0b8032960df7e5917ce874bcf4ff267a6bed82b2567def41/detection

>>18115
Oh, thanks. I put the URL instead of only the ID

>>18062
>>also how dare u upload without telling us (its OK)
also what was that update

lets make this area a little more private shall we
(incase the payjews are watching)
what's ur email OP

>>18118
I got an error with the downloader (forgot to write it, sorry) and then restarted it, but instead of continuing the download it skips the rest of the creator's page and continues with the next one. It would be great if you added the option to check if the files are already downloaded or not.

>>18205
Fill = y in yiff.txt will do. I'd love to hear the error detail though.

>>18157
see
>>18104

>>18165
Nice trick

Added a new setting whether to make a "temporary gallery" HTML file or not that will let you view the recently downloaded pictures in your favorite browser.

>>18214
i got it!
File "C:\Users\Myuser\Desktop\Patreon\yiff.bat", line 226, in <module>
partydownload()
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 126, in partydownload
FILESIZE = int(requests.get(url + line + "/" + re.sub("#", "%23", line2), st
ream=True).headers['Content-length'])
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-length'</module>

>>18275
What was the last url it's trying to download?

>>18276 It was Jagodibuja's url:
Downloading https://yiff.party/patreon_data/14257600/1371873/yaoyorozu.jpg
Traceback (most recent call last):
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connection.py", line 344, in connect
ssl_context=context)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line

>>18277
Continued:
line 412, in wrap_socket
session=session
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 850, in _create
self.do_handshake()
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 1108, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] An existing connection was forcibly close
d by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])

>>18279
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connection.py", line 344, in connect
ssl_context=context)

>>18280
And finally:
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 412, in wrap_socket
session=session
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 850, in _create
self.do_handshake()
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 1108, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(1
0054, 'An existing connection was forcibly closed by the remote host', None, 100
54, None))

Looks like a simple connection problem with yiff.party... keep trying and it'll get through.

>>18285
Thanks, this seems to work.

>>18289
Not anymore, I have restarted the downloader 3 times and it keeps crashing while downloading this file.
Downloading https://yiff.party/patreon_data/9223743/987481/levy2.jpg
Traceback (most recent call last):
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 226, in <module>
partydownload()
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 126, in partydownload
FILESIZE = int(requests.get(url + line + "/" + re.sub("#", "%23", line2), st
ream=True).headers['Content-length'])
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-length'
On an unrelated error: https://yiff.party/patreon_data/11216414/1044654/callie1.jpg is corrupted...</module>

>>18297
Finally the actual errors I'd want to look into. What was the creator of these?

Oh, besides, the https://yiff.party/patreon_data/11216414/1044654/callie1.jpg is probably broken on yiff.party too. I've seen several other broken files and these aren't yiff.bat doing. Not much we can do. Meanwhile https://yiff.party/patreon_data/9223743/987481/levy2.jpg looks like there was weird character in filename that yiff.bat hadn't handled properly yet. I wonder what was the filename it originally had.

Also I've recently updatede yiff.bat to handle utf-8 characters I forgot to to include for the new feature (create gallery HTML) I've implemented today. Unrelated to that error btw.

Hey OP whats with the porn pics
also here is the bundle https://openload.co/f/m8PhEV2G4_4/yiffy_client_v1-10.7z
VT link Y10 https://www.virustotal.com/#/file/3a1285190e98903264978d854d41ceb6d72282872ec092140f8e262f34cc3993/detection
VT link Y11 https://www.virustotal.com/#/file/f95dc1861580bd70f8954e86c375b41f13ca965a5f1c39ef6783be3e96a0ed7e/detection
did i miss any version

>>18308
Y tho?

>>18297
OH, just realized, it's been jagodibuja, so I checked the links there. Apparently it's not just yiff.bat. But you can blacklist the broken files by adding their filenames in mediocre.txt in creator folder.

Updated yiff.bat to call 404'd file "Bad link (404): ..." then skip to the next one. You can also ignore this warning by adding them (their filenames) to mediocre.txt too.

Added proxy (socks) support

So the program suddenly stopped, I started it up again and after it's done fetching for the Yiff ID, it just stops and says Finished, saying "There are no new posts" but there's still lots of images left (I checked the downloaded files) any help?

>>18777

On each launch, it'll start downloading files from the latest on server and as the "Fill =" setting from yiff.txt says to stop at latest file existing on disk, and since the latest post is also already on disk, it finishes with a message telling you there is no new post. "Fill = y" will download the missing files past that.

I'd like to know what have actually happened when it "suddenly stopped" though.

I mean, I don't know jack about python, but why do you need to use a .bat script?
It seems absolutely useless, and only serves to lockdown which OS can use it. Link me to the most recent version, I'm curious.

19651980.Bodi Otter1 copy.png (5.0MiB, 2550x3300) save_alt

>>18871

Because I don't use Linux distros and it's in .bat file for convenience. Seriously though, it made things surprisingly convenient. I'm sorry it's for Windows only. It should be possible to extract codes from there and convert codes to not be Windows-specific then it'll become cross-platform. But I'm not gonna give little bit of my effort to make it cross-platform myself. You should feel lucky that I share it here.

The link in OP is always the latest.

>>18871
i use kvm m8
try it dude

>>18876
>You should feel lucky that I share it here.
the ONLY reason people want to use scripts like this is because they don't want to install an older version of FireFox and use DownThemAll.

Improved downloading message "Downloading 0 / ?? MB ..." to show progress using 1MB interval and actually avoid leaving file corrupted on HDD from interrupted download.

I've yet to figure out how to run this, i have python 3.7 installed, but then i try to run the yiff.bat it opens a console for a second then instantly disappears with no error

>>19014 figured it out, python wasnt installed in my %userprofile% cuz i installed "For all users"
also it crashed as soon as it started downloading a .psd, would there be a way to exclude that?
and what about files that have no extension, or things like "file.1" https://yiff.party/patreon_data/21718526/file.1

>>19016

It shouldn't be crashed by any file format... what was the creator?

Files can be "excluded" by adding their pattern (per line) in mediocre.txt in creator folder, or use a magnificent.txt to only download files by their pattern. If you don't want like .psd then you can add it to mediocre.txt, or if you want only pictures and nothing else you can try with .png, .gif, .jpg, and .jpeg into magnificent.txt.

>>19022 it was https://yiff.party/910946, as soon as it hit the first psd

>>19070

Umm, there doesn't to be a problem downloading the .psd and "file.1". yiff.party tends to have connection problem, so I bet it was connection problem. But I'd probably see the error better if you post the error message either by snapshot or copy/paste from CLI.

I guess I need to rewrite CLI messages for clarification for the first time users.

Added quick tutorial where to get creator ID

Do you have any more docs? How can I for example only download certain extensions?
I only want JPGs, PNGs, GIFs, etc. How can I whitelist and blacklist certain extensions? I'd like to blacklist .PSDs for example

>>19144
Never mind I'm retarded. I was typing *.psd instead of just .psd

>>19145

You got it ;) There's no wildcard support, but the begin/end of line can act like one.

>>19148
Any plans to implement a global magnificent.txt and mediocre.txt system? Maybe if in the same folder as the batch file?
For example, I never want to download psds, but if I have several creators I need to copy it into each folder

>>19166
Expanding on this, could you implement it so that it also scraped the text from posts?
Many creators store things as links inside their posts. Should be relatively simple.

>>19167
I've just spent a few minutes trying to do it myself, maybe to have it write the "title", "body" and "id" parts from the json file to the html file just before the makehtml section, however I'm not familiar with how you're parsing it, so I don't know how to extract that information. I myself would have it save as said before to the html file aswell as to a seperate text file with the "id" as the filename, if the user says yes to saving content in yiff.txt


I'll experiment with it a bit but considering this is your code you might find it easier

notepad++_2018-11-19_22-37-00.png (94.6KiB, 1827x650) save_alt

Final post for the night, I'm really novice with python so excuse any issues but I really think you should consider parsing json properly, it'll make it much easier to work with in the future as well as modify.
Here's an example I sort of managed to throw together. I don't understand why you're using a global variable and not just passing info directly to the function, but here's how I modified it.
You pass the numeric creator ID to the download function, this generates a request a parses it, generating the json for every post made by that creator, saved as x. You can iterate through that using a for loop, and then due to it being saved as an object you can then read the post title, body, files, etc.

The only issue I'm having is that the files themselves aren't saved as like a "sub-object". I'd like to iterate through the files but I can't figure out how. If a post has no files it can cause an error for example, or really old posts having no file_names, just an url.

Hopefully you can get some ideas from this but I'd really suggest reworking it to use proper JSON in the future. It'll be so easy. For example, now you have the .file_name you can just check if it exists inside the creator folder, to see if it needs downloading, and you also have the body, which can go into a text file with the post.title as the filename, aswell as go into the html gallery.

notepad++_2018-11-19_22-58-18.png (125.7KiB, 1902x775) save_alt

Ok I lied, here's the last post
I wasn't thinking properly on my last post. There's only ever one post file, and only sometimes attachments. This revised code causes no issue.
This gives you the url of the file and the attachment aswell as the filenames, just put the download and file write code inside the for loops and your gold.
Consider using direct json in the future, it's much more readable and easier to work with

I have previous experience with regex. I'm taking a look at JSON and see how it works, though right now I'm having hard time grasp at it atm.

Yiffy-Get likes to save JSONs on HDD though it's not my thing, however it can be done with little bit of added codes, like below:

jsonwrite = open(foldername + CREATOR_FOLDER + "creator.json", 'wb')
jsonwrite.write(REMOTEFILES.content)
jsonwrite.close()

after where "LAST_CREATOR_ID = CREATOR_ID" is at in yiff.bat

I'm not sure what it should look like when saving text posts on HDD. I suppose you can try parse and divide REMOTEFILES.text (r.text in your version) to extract text information and save them as .txt files somewhere.

Updated yiff.bat to read file extension (per line anywhere) in yiff.txt for global file extension whitelist.

"Shared files =" setting obsoleted by global file extension whitelist

cmd_2018-11-20_16-11-44.png (41.3KiB, 1345x441) save_alt

May aswell tripcode for this
>>19211
I edited an older version of your code, and did a few changes, I'll upload it here incase anyone wants it. I implemented a global magnificent.txt and mediocre.txt, they just have to be in the same folder as yiff.bat

https://mega.nz/#!05pkWQ6R!a5Op-ziTN3DE_1oFHuzQvaCyhtiaM_3kLQqpyL0cwOI

Heres the "readme" of my edit. I changed some things so the line numbers will be different but eh whatever
If you set Gallery HTML = Yes in the yiff.txt file that gets created, it will now rebuild the yiff.html gallery on every run, so you don't have to delete everything or wait for new files. By default, it has the post title, post body, and embedds any attachments. Now that I think about it it will probably error out the ass if said attachment is a .psd or .webm or .swf or whatever but tbh I cba fixing that if the site is gonna close in a few days. Shouldn't be an issue
If you want to stop it including the post body for example, open yiff.bat in a text editor and go to line number 190. I hope it's self explanitory but if not, just remove "\n<br>"+postBody+"" from the partyhtmlwrite.write and you should be good. Scroll up to line 183ish and you can see the data from each post which can be included here
Throw in a magnificent.txt into the creators folder for whitelisting, or have one in the root folder, next to the bat file, type each thing on a new line. These are all CASE INSENSITIVE
Example content of magnificent.txt

wip
.psd
diamond

This would mean that in this folder, or everywhere, only files/attachments that contain wip, .psd, or diamond in the filename will get downloaded.
Throw in mediocre.txt into the creators folder for blacklisting in that folder, or next to the .bat for global (like typing extensions to globally ignore them) works the same as whitelisting. Both files can work together

Shit, I fucked up on that upload, here's the fixed version. It would previously get an error if a post had no body, or if there was no root magnificent.txt, hopefully this will fix it
https://mega.nz/#!dxwViQpT!DfQSZom2LGfbjZV3z-08dF6cUOkvI67Vg3paQhDhFDA
>>19211
>>19264
Second reply due to line and character limit.
Shit I implemented:
-Proper json parsing for downloads, should be easier to improve on in the future
-This now means that I can also extract post data easier
-Post data is thrown into a html file with the id of the post as the filename and the data in the post as the body. Delete lines 211 and 212 (the with open(blahblah... and the next line, to stop it doing that. Or just delete the files after they download it's not hard I'm sorry don't make me cryplesesto)
-The gallery thing I mentioned above, so it now puts the post data in the gallery too, though with a lot of creators that file is going to be stupidly big. It wouldn't be that hard to make more that one html file, one per folder, with links to each, but meh I can't really be bothered at the moment
-More verbose downloading? I guess? It tells you why it did/didn't download each file from each post

Shit I broke and didn't fix:
-Less verbose downloading, it now no longer shows file progress when downloading. Downloading itself is like 152 onwards so if someone wants to fix that just do it. I tried but I kept getting 403 unauthorized when trying another download method)
-In progress downloads no longer become .part files, they just half-download and end up corrupted if the download fails. (Just delete these files and it will redownload them next run, no need for new posts)
-Probably a lot more

Take a look at my code, see how I did the JSON parsing. I just converted it to a class, and then read the properties

Small edit for quality of life, didn't realize how annoying the html files are
If you still want them downloaded but don't want them to clutter the root folder, replace on line 214(ish):
postInfoFilePath=Path(creatorPath+str(postId)+".html")
With:
if not os.path.exists(Path(creatorPath+"posts/")):
os.makedirs(Path(creatorPath+"posts/"))
postInfoFilePath=Path(creatorPath+"posts/"+str(postId)+".html")

This will put them in a new "posts" folder inside each folder.
For anyone doing this you get indentation errors, make sure any space to the left is actually made up of spaces and not tabs.

Or just use this this updated version
https://mega.nz/#!ZgYkFIiI!EO7IbF1TwFB24I2olwjBXd1n3cnz63wNDwWa2zfEClM

Thank you for sharing. The error popped up saying the "mediocreread" from line 298 is not defined. I'm looking through your code to see what's changed. I can see lot of changes. It might take me little while to understand what's going on. On a side project, I'm starting to understand how can JSON be used. I'm experimenting with it in an independent batch file that work with creator.json I've dumped into certain creator folder. I use it to make HTML file to show text post and load images from disk than online, along with url in text as source. I can see that JSON is better used if trying to extract lot of different types from JSON file.

Hey, sorry about that. I thought I uploaded the fixed version. The problem is it's trying to concatenate the mediocre stuff from each folders file to the "global" mediocre, but if the global mediocre doesn't exist then it can't concatenate each folder to nothing. I've implemented a simple fix for that.

https://mega.nz/#!UohRFaiY!DI6GXXw9hdbwNdLHIPR6HTPEDozE4KKIMKt7stDfYh4

There are few changes I did to your version http://www.mediafire.com/file/6xvlmqgbxjedy8z/yiffout.bat/file

Not much, but the "partial" file type (.part) is actually important because without it you can expect file to be corrupted once you kill CLI while downloading, or whatever will disrupt its download progress, like internet suddenly went off. I've restored it. Another thing is fix global + per creator mediocreread and magnificentread you can see what my fix looks like!

My side project is mostly done, http://www.mediafire.com/file/hzj34i5dbvbhsna/makehtml.bat/file

Drop it into creator folder and launch it to build gallery HTML file(s) from JSON (the batch file will download it). Multiple HTML files will be created as continuation using 256 files interval for browser performance.

Here's a funny issue: Any file types will be taken as image, browser will think non-image files are broken images. I don't think I'll bother to do about it yet.

chrome_2018-11-21_18-00-54.png (26.0KiB, 1656x124) save_alt

>>19342
Cool to see how you did it.
One small thing you missed though, on line 209, by removing 2 lines and chaning postInfoFilePath you made it so it saves the html files in the root folder again, however I'd made it so it put them in a dedicated posts folder, otherwise for me it lagged the file browser.

I'll take a look at the gallery generator tho, looks usefull

cmd_2018-11-21_18-11-58.png (24.2KiB, 966x430) save_alt

>>19342
Uh I think there's a mistake on that release, it's reading and writing the config to the same yiff.bat, breaking it on second run.
I didn't touch it so idk how you're supposed to do it but it looks like you added the word "out" in each filename for some reason

cmd_2018-11-21_18-24-41.png (7.9KiB, 663x101) save_alt

>>19342
>>19355
And one final thing, the .path concatenation causes issues.
It's cool, I kind of guessed it would be done like that, renaming after the DL, but I didn't get around to trying it.

The issue is, that filepath is a "WindowsPath", and ".part" is a string, python can't concatenate the two with +.
To fix, convert the filepath to string, and then (idk if this is necesarry but eh I did it) back to path.
Replacing it so it looks like
...pen(Path(str(filepath)+".part"), 'wb') as f:...
and
os.rename(Path(str(filepath)+".part"), filepath)

>>19355

Sorry, it's really dumb code I made when trying to incorporate with original yiff.bat in same directory and load same variables. It should've been just static path like "yiff.txt", "yiff\\", and "yiff.html" respectively. Ah, well!

>>19356
It's kind of sad that Python will take variable that is only made of numbers as integers and not able to concatenate together with other variables containing str. It adds confusion to the unaware code editors. Happened to me while I was working on my side project.

>>19377
Lovely work you're doing. Any update on being able to get the text of a post in .txt? For links and stuff

>>19378
Isn't >>19344 doing what you want? Unless you want posts and/or especially links to be arranged differently.

>>19356 >>19377
I skimmed too fast, but the same is true for WindowsPath.

explorer_2018-11-22_20-54-11.png (65.7KiB, 436x255) save_alt

>>19378
Either drop >>19344 into each creators directory and run it or use my fixed edit https://mega.nz/#!ZsIlxSSZ!Q3U6uCEtbSbfdMCjy1l1fVxfNBw4N5Yn8hI__C3I2VQ which saves all posts as a html file inside a posts folder inside each creators folder.

>>19381
>>19434

I'm retarded, I was skimming through the posts and somehow missed that. Thanks for the work guys.

>>17154
Can we have a feature to download shared only?

>>19522

That is odd choice. Unless you mean to download only files by archive file types which can be done with current version when you add their file extension names in yiff.txt. Why does it has to be shared data only?

Updated yiff.bat to build complete gallery HTML files residing creator folders if "Gallery HTML =" setting is turned on in yiff.txt.

The "yiff.html" at working directory is actually only meant to show the latest pictures from recent download, kind of like daily dose of mine.

Updated party.bat (>>19344) and yiff.bat with commented out codes for the ability to save per-creator data on disk. Remove the "#" sign before "jsonwrite" instances and they'll work. party.bat will look for and work with JSON file first if it really exists. This allows re-compile HTML files from JSON file regardless if yiff.party is down.

>>19611
This new feature allows to download the whole creator.json to be downloaded to the creator's folder, and allows Party.bat to work with it locally instead of contacting again yiffparty's servers, right?

Two little questions; one, is it neccesary to drop party.bat into every single creator's folder, or is there a way to make it run a single time for all creators?
And second; about Fill = and Shared Files in yiff.bat. Using Fill works correctly to have it download only the new stuff by keeping only the last post's files on the folder, but the shared files behaviour is different. It seems to redownload them every time, instead of stopping at the last. Am I doing something wrong, is it an oversight, or is it expected behaviour?

(same guy than >>19692)
An additional consideration; if an user imports posts of higher tiers that are older than the most recent post; would yiff.party cathegorize them as "newer" even if they have an older date, or as "older", therefore causing yiff.bat to skip them when set to Fill = No?

>>19692

Yes, the party.bat will look for local JSON file first, then fetch from yiff.party if it doesn't exists. And yeah it's unfortunate that it's for per creator only for now. I'd probably upgrade it to work across creators to become an offline counterpart of yiff.bat that will only create HTML files from JSON file.

The "Shared files =" setting is obsoleted by file extension whitelist. It was the old way to avoid downloading large files which is typically archived file types. The file extension whitelist can also avoid downloading them as well, so... That setting has since been removed in new yiff.bat, please re-download it from same download link in OP.

The code behind "Fill =" setting that will stop downloading further files is actually expecting a third file in a row to exist then it'll stop. In old yiff.bat, patreon data and shared data were isolated and counter will reset for each. I guess you were experiencing the odd behavior with shared data is due to that, also probably because the certain creator only has one or two shared files.

Again, the latest yiff.bat is always in OP. I occasionally do silent fixes too if I find out the problem.

>>19698
Thanks for the reply.

Not sure if you saw >>19693, but additionally, I'm seeing that it's currently not able to get the names from the Yiff.party server, or that gets the creator's name wrong. Is that on Yiff.party's side, or something you might have changed in the bat?

21277952.otter-dick-suck.png (837.6KiB, 1500x1208) save_alt

>>19699
Sorry, I was busy with other things today. So to respond >>19693 I think yiff.party captures actual ID from Patreon and will download what is available on Patreon and append ID on them. I can see that it can conveniently used to fill the recently deleted post from yiff.party's post flag report and let the yiff.party's scraper do the job. So I assume the higher tier would mean newer as long as the older ones didn't get deleted. I realized that since the new yiff.bat will merge shared data as continuation after patreon data, so yiff.bat will not check for new shared files when it stopped on new post based on "Fill =" setting. Dunno if that's such big deal if "Fill =" setting can always be turned on. It'd also complicate the code to take care of that with "Fill =" turned off as it did for three in a row existence on disk used to overcome the suddenly stopped on "already downloaded" file because of post + attachment duplication which each actually has same post ID + file. There were double "already downloaded" files in extreme cases so it extended to check if a third file exists on disk in a row to stop.

>>19699
There are two possible scenarios as to why yiff.bat thinks ID is bad (aside from actually bad ID), first trigger is that the ID wasn't found in https://yiff.party/creators2.json, second, yiff.bat actually captured cloudflare's page instead of actual per-creator JSON file and noticed "" in page source (same as those 404 pages, and JSON file doesn't have that) so it tells you the creator ID is bad. You can tell yiff.bat to skip all creators down to that creator to try again when you comment out all the other creators in yiff.txt then restart CLI.

>>19712

Oh, yiff.party didn't want to display what was in that "" in my post, it was DOCTYPE html with brackets. Sorry about that.

>>19699

What is the creator by the way? If the problem is consistent that is.

Oh, also, you can assign name yourself by the way. It can be any name really. ID is sensitive information, a small typo there can lead to another/non-existent creator. Just be sure ID is correct and then append with non-digit character only, after that, next characters can be digit again and w/e.

>>19716
I ended up getting the name myself manually. The ones I saw thing trouble were Rukis (got wrong name, like cslunarin2 or something like that) and h0rs3 (no name at all, just the id numbers).

Posts downloaded correctly for both anyway.

Added new setting "Log 404 files ="

Updated yiff.bat and party.bat to correctly capture attachment filenames using regex + attachment url for use to build gallery HTMLs to avoid images being displayed as broken, few attachment filenames on yiff.party server are actually broken, so...

Upgraded party.bat to work across creators. Please take it to where yiff.bat is at! Well, technically, it'll look and work with \yiff\ folder. It'll optionally load yiff.bat for global file rejection filters only, and will read sub-directories in \yiff\ as creator list.

>>19762
Alright, thanks. I'll look into it very soon.

https://yiff.party/creators2.json actually has some creator names under different ID. What gives? I guess there isn't much to do about it other than have admin look into it.

>>19949

I mean... party.bat will look for yiff.txt for global file rejection filter list, not loading yiff.bat for it.

>>19581
I want to download "shared only" as some artists like Sakimichan only send their stuff via messages and the only stuff leaked on yiff party is previews...

Improved file rejection filter system in yiff.bat and party.bat. magnificent.txt will override file type whitelist from yiff.txt, making it possible to accept alternative file types like .zip for certain creator.

>>20081
It should still be possible to download only those by giving magnificent.txt their file extension names.

Updated party.bat to extract links to "3114967.raier.txt", etc and will make patrol.txt for the filenames found on disk compared to your personal filter list.

Updated party.bat and yiff.bat to take file extension names from yiff.txt as blacklist if there's exclamation mark ("!") prepended to each. Doesn't have effect until you remove/comment out all other file extensions. This allows downloading other obscure file types and/or broken filenames.

Yiff.bat no longer has abilities to build complete gallery HTML files, leaving the job to party.bat. Yiff.bat will check if party.bat exists in same directory then save per-creator data "creator.json" on disk ready for party.bat to pick them up. After it's finished, it'll ask you if you want to run party.bat next (press any key to continue) or not (kill CLI to finish).

Guess I'll be forced to rewrite my script after all. Still I am amazed you coded this shit in bash, fucking beta :kek:

>>20884

Welcome back! And no, it isn't bash, it's batch, Windows thing. Also no one have forced you to rewrite your script. But go ahead, update your script for the neglected OSes. Otherwise, please do tell how can it even recover for the Windows platforms. Let's not talk shit about codes behind mine because it works better than Yiffy-Get frankly. Don't believe me? Try raier, check for duplication and corruption, furthermore, I've made human readable text files for settings'n'stuff and lot of other improvements. If you tried hard enough, your script will probably end up being as good as mine anyway. At least non-Windows users will thank you. <3

>>20891
dunno if this works fine on WINE
but my w7 VM is ready
also i apologize for not showing up a few weeks ago

>>20891
I plan to rework the entire downloader to be modular simply put. So it makes adding features more easy, plus the human readability part as well.

>>20903

Looking forward to it. I think my script is finished for what I want it to do. But it's always good to have alternatives to fit people need. I've seen people want JSON parser and I've looked into it, considered about it, ultimately not included it. My argument is that yiff.party get information wrong sometime (e.g. filename =/= filename in url in https://yiff.party/787448#p3243120) and the piece of correct information still requires regex to extract AFAIK and secondly it's my personal script, so... ??

>>20905
Yiffy-get the filename handling and stores them in special format that makes it extremely unlikely to get messed up (postid - titlestring - filename as uploaded), it works but it's messy. I want to resolve it somehow but still thinking how.

>>19949
What does Log 404 files exactly do? Fiddled with it, couldn't really tell a difference between on and off.

>>23340
It's when you weren't able to download the file. If you can download everything OK then it'll not log anything.

>>23350

Alright, I've been trying to fix something recently. Trying to download DogBoneArtwork's gallery (2307896), there's a file with a HUGE filename

(23194231.https_cdn1.telesco.pe_file_Y3H5uatmZy_oY50vhoHV0591wCjXnmMS-NSRtWPU_YlzXSbDvRY0Lgmz061cyvI4kSX5TiuSfDtiRCgYbTH9hg4WUJ7y7_ZxTx3pJEz1Td0k8_kJdRj4TOyieWEMUHbm9quPCKsjKMPHAYrig6iO-_4)

That will crash yiff.bat immediately, every time. Also, when adding the extension (which in that case es .pe_file_Y3 etc...) to the blacklist makes yiff.bat download everything up to that file (6-7 files) then stop as if there was nothing else under it anymore.

Am I doing something wrong?

>>23435

The blacklist/whitelist in yiff.txt is global and works differently than mediocre.txt and magnificent.txt.

Yiff.bat will only take off from yiff.txt as file extensions to avoid matches in the filename before the extension name (like PSD in "December.PSD_01.png"). I haven't checked codes for a while so I'm not sure if yiff.bat cannot handle long or weird extension yet, but you should use mediocre.txt for that instead. Add something like "23194231.https" in mediocre.txt in DogBoneArtwork's folder then restart CLI.

>>17659

Where did you get that picture?

Hi. WTF is this?

Hi. WTF is this?

Say, which version is the latest up to date one? There seems to be a couple different versions linked in the thread.

>>23834
i also have older versions which are not included here

>>23856
Which one works best?
Recently went through the trouble of getting an e621 grabber working on linux and that was a real pain. If you've got an up to date linux version that'd be perfect, though I could probably get a windows one working in wine or just use an old windows XP drive I've got.

>>23914
i use KVM and i never actually tried anything Vbox works too

>>23959
good to know, you using the one in the OP post then?

>>17154 yiff.bat (main)
>>19344 party.bat (optional add-on along with yiff.bat that comes with whole different set of features)
>>17476 chroming.bat (captures filenames of the mediocre shits you've deleted to mediocre.txt)

They're all on latest

>>24037
thank you very much.

I'm having trouble using this. Any guide?

>>24141

Umm... just run the batch file and read what it says?

I haven't ever used python before. Do I just put the .txt file in or is there more to this?

>>24350

Please change the file extension back to .bat if it's something else. Then you just have to launch it and it'll tell you what to do.

>>24404
ty. that was the prob. it was .bat.txt so it was a .txt file

add this plz https://yiff.party/bbs/read/12795#22107

How do you download rar files? All i'm getting is jpg and png

>>24486

Check Yiff.txt again, there are image file types as whitelist. Sorry if it's kind of unexpected after running Yiff.bat several times since its first time.

>>24501
I checked the txt file below the bat. The rar was blacklisted. Do I just remove from the blacklist and add to the download?

Edit: so i did that, but I only got some rar files, it didn't download them all. Any suggestion?

>>24519

Blacklist isn't in effect until the whitelist has been wiped out. If you blank out both whitelist and blacklist it'll download any.

I guess you have interrupted download at some point since you said you've only downloaded some and feel that there's more, in that case, turn "Fill =" to yes in yiff.txt and it'll download the missing files past the latest files.

>>24537
How do I turn "Fill=" to yes?

>>24562

It's in first line of yiff.txt... change from No to Yes

>>24571
Sorry lol I didn't notice that. I thought that was a joke line

>>24573

Oh, it's alright. I was thinking it'll give away a hint how y/n works.

is there a way to check embeds for 404s

>>24611

"Log 404 files =" to yes in yiff.txt will give you yiff.log file also containing links where 404 files came from. Is that what you mean?

>>24616
thx m8

Any way to download mega download links?

>>24655

Party.bat makes a useful file containing every links extracted from posts per creator, you'd just have to open each in browser to download files from outside domains. That's the best my script can do...

>>24662
dont forget the Megadownloader v1.7
integration is also nice

Removed setting "Fill =". New way to stop (per creator): get the name of the file you want to stop at, create an empty .txt file with same file name.

Added new feature: Normally yiff.bat will ask for ID for the new creator you want to add, but you can just write that creator name with "lookup." prepended and save (per line) to yiff.txt and yiff.bat will look for ID for this new name. Say, if you know "raier" is on Patreon (and is on yiff.party server) you can write "lookup.raier" to yiff.txt and it'll turn into "3114967.raier".

What does this do differently to yiff.party then?

>>29095

What are you talking about?

>>29169
What this whole thread is about... duh...

>>29183

>>17352

Could use an update. Looks like older links are hosted on dold now.

Any ideas on what is making this happen???
[Code]
New ID! I'll check creators list from yiff.party . . .
Traceback (most recent call last):
File "H:\FF Downloads\yiff.bat", line 327, in <module>
remotecreatorlist()
File "H:\FF Downloads\yiff.bat", line 137, in remotecreatorlist
creatorlist = json.loads(creatorread)
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Restart CLI? Press any key to continue . . .
[/code]

2486281_DiDi_cutie_fox.jpg (107.4KiB, 612x792) save_alt

Updated yiff.bat to download images from new urls

>>30317

Looks like https://yiff.party/creators2.json just went out of existence. Might it have been moved (anyone have idea where it went)? Temp fix is to write down the creator name after ID. Supposedly you have ID 4979402 which happens to be haychel as its creator name you write like this "4979402.haychel" or "4979402.whatever" and it'll work.

Updated yiff.bat with a new url where the creator list has gone to.

one thing i have noticed. is that yiff.bet doesnt grab the high res photos. it grabs some sorta other junk. odd

>>30597

Example? Is it different than downloading them yourself?

New version seems to download ender files that stops scans for other files. Is this intentional? Example 3659577 downloads an ender file after scanning.

I love the program! Just needs the ability to opt out of downloading the "_low" copies of files!

Updated yiff.bat adding more dialogs for new users and not start download suddenly after updating changes to new yiff.txt, giving users chance to check/edit there before downloading with their personalized filters and settings.

>>30818

You're right, a new version does that. It's a new feature! Ender file will be there (not a downloaded file technically speaking...) after successful download of every file from creator. It'll move once you download more new files from creator. It'll not be there (or not moving) if there's an error downloading a file. It's smarter than "Fill =" setting, willing to try again only on some creators with the error downloading files and the interrupted downloads (usually by killing CLI while downloading).

Now, if you're going to download a lot of things from yiff.party, it should be okay to interrupt and resume like "Fill = Yes" did in previous versions, then conserve storage performance a bit like "Fill = No" did.

>>30819

Put "_low" in mediocre.txt in creator folder that does that.

New version is giving the files the ID of the artist in the filename instead of the ID of the file itself, how can I change this? This is making it so files aren't ordered as they were created, like they previously were.

>>30940
Disregard, I was running a version from February, thought it updated automatically. Downloaded latest version, it's setting the filename correctly.

https://yiff.party/668242 LayerIndustries has 30 missing posts

Couldn't find this in thread, but I'm grabbing stuff from a creator, and along with the actual content, there are a pile of cropped "preview" images. Is there a way to exclude these after deleting them, so the script doesn't redownload?

>>31215

If there's a consistent term in file names of these inferior images, then put it in mediocre.txt in creator folder and yiff.bat will not download them again. If they're random in file names then chroming.bat from >>17476 should help before having to constantly copying these file names to mediocre.txt yourself.

>>31228
chroming.bat worked, thanks!

>>31307

My pleasure 🦦

Updated yiff.bat and party.bat to load global rejection filter list along with magnificent.txt if there's "yiff.txt" written in it. Say, you want all files but archive file types from all creators, you'd have

> !.zip
> !.rar
> !.7z

in yiff.txt. But if there's one creator you DO want archive files from, you'd put in

> .zip
> .rar
> .7z

in magnificent.txt in creator folder. But then that will reject literally every other file types for that creator. By adding

> yiff.txt

in magnificent.txt meaning you just wanted more things from that creator. You can still blacklist certain archive files by adding them to mediocre.txt. There are lot of possibilities and a check for "yiff.txt" is bit unusual but it's there for convenience.

party.bat was updated with a hidden feature, edit the batch file and look for "expensive = ". What it does is to scan and log file names of corrupted images to patrol.txt. It will take a while on slower devices.

Lot of changes were made to yiff.bat and party.bat for small performance boost and covering some other shortcomings. Now handling of missing local files because of older posts that got imported later (by importer with higher tier post access).

Please report back issues.

look up yiff-scraper on github instead

superior yet can't download anything besides jpegs
yeet.

>>32311

Right, it doesn't yet, the predefined settings from yiff.txt tells yiff.bat to just download pictures from your favorite creators and nothing else. Yiff.bat is meant to download thousands of pictures and the new ones at your convenience. You should edit the settings in yiff.txt to allow yiff.bat to download other file types.

>>32196

Yiff.bat is not a scraper.

>>32322
edited them, still no result
deleted the enderfiles and all that, still only pictures, nothing else.

>>32356

You need to add or delete lines of file extension names in yiff.txt. There's also lines of file extension names after exclamation marks, they're blacklist version that don't have effect until you blank out all the normal file extension names. If you blank out all then it'll download any.

You can literally do this shit in bash with a oneliner, a few more for interactiveness.

>>32477

A crazy long one you mean? Love some example. Can you?

how do you get it do download by the date?

>>33102

It seems to download from newest to oldest as the fetched file list contains post ID in descended order and that descended ID is correlated to older post. I don't think there's a way to download file with its creation date, if that's the other you meant.

>>33108
something like that. i downloaded some things, but there are out of order post wise

It's the way Patreon assigned IDs to their file (technically part of url before the actual filename of a file for now), yiff.party followed suit then yiff.bat in the end. While technically there wasn't ID in filename itself, yiff.bat appended them to filename to work as collision resistance. They can also give you some sense of chronological order. Same ID would mean they're posted at the same time especially from a post of multiple files. There are actually secondary IDs, I think they meant to display file in exact order the creator has uploaded to the post of multiple files. Not sure, and I don't see reason to use it yet, apparently more harm than good, there's sometime a duplication between main file and in one of secondary files, and it's great that yiff.bat happened to not download the duplication because their secondary ID weren't even put into comparison yet. Again, the ID you find in filename coming from yiff.bat is just about collision resistance, then it's up to you to use it as chronological order.

You shouldn't expect ID to order (descended or ascended) exactly as the file's creation date, the file's creation date is created by your OS, and there's no reliable way to fetch original date of a file from the internet, it could be fucked over by a web server, etc, the importer of higher tier would import the older post at later time meaning you'd only get "older" files with newer creation date above other "newer" files.

Is there a way to get this thing to run in a different location / drive than on the default C drive? My python skills are weak AF.

Thanks in advance!

>>33177

Yiff.bat will download files to a folder of same name in current directory (meaning where ever yiff.bat is at and whatever it's named as)

So if you have yiff.bat as Z:\English\Motherfucker.bat then it'll download artpieces to Z:\English\Motherfucker\

Added a new setting "Show mediocre ="

Sageman 10/24/18(Wed)13:25:53 No.17328
http://dagobah.net/flash/Thread_Derailed.swf
reason OP's profile pic hey w7 guy plz upload locky

w7 890 10/24/18(Wed)13:31:29 No.17329
>>17326 lick my ass bitch
>>17328 the quick and dirty method no longer works no can do sorry :[ just call the admin

shitbot named w7 v8.9-0 10/24/18(Wed)13:41:06 No.17333
>>17254 >>17291 >>17303 u guys mad
https://www.virustotal.com/#/file/b41ffad900c94872ee6a3d1b099e48a984fceeba1968ba9dd5b4e572579ecc27/detection eat the damn file its safe
>>17329 after that plzkickmyshinymetalazz
WTF.jpg.py has crashed

Orgymania 10/24/18(Wed)13:42:39 No.17334
>>17333 >>17329 streakfagging but i still forgive u

Updated yiff.bat with less third party dependencies (now there are zero). Uses module named "urllib" in placement of "requests" and murdered proxies support. The "Proxy =" setting in yiff.txt is now useless.

>>36317
I updated to the 6/25 version, but when I run it it just keeps saying OFFLINE. I tried having it download a different artist in case it was just a problem with that one, but it still had the same problem.

>>36622

It's overkill error message for all error related to trying to fetch creator-related files off from yiff.party (grabbing per-creator data and/or checking name/id for the new creators), I understand that it can't be diagnosed properly. It has to be connection problem generally or invalid connection configuration (the proxy feature was removed in this version). If you edit and remove the "try:" and "except:" from yiff.bat you will see traceback errors that may give you hint what really went wrong, it could just as well be my script is being retarded somehow.

>>36630
if "" in REMOTEFILESdec:
NameError: name 'REMOTEFILESdec' is not defined

>>36649

You must have removed "REMOTEFILESdec =" line along with "try:" and "except:" by accident. Try again?

that's because the "REMOTEFILESdec ="" is in the try, so if I get rid of it, it gets rid of that too.
I know it's this chunk that's throwing the error:
try:
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
REMOTEFILESdec = REMOTEFILES.decode('utf-8')
except:
print("\n OFFLINE")
sys.exit()
if I remove the whole thing, I get the "not defined" error I mentioned earlier, and if I comment out just the try and except lines, it complains about an unexpected indent. sorry I'm not super code-savvy, I'm working off a semester each of BASIC, c++, and java from my community college nearly a decade ago, so I don't know a whole lot

You should give us the ability to specify the python path in the .txt file so whenever you update we don't have to change it again.

>>36654

The "try" and "except" is exception handing sort of like "if" and "else" for whenever error comes up. The offline message is coming from "except" because of an error in "try". You need to remove only the "try:" then whole "except:" thing (and fix indentations) so there's no if-else and then you can see the error that exception handling didn't want to show that to you. No worries, you'll get it soon enough.

I can re-up old version if you need it.

>>36694

Python is one that's reading .txt, so python path in there would be useless. Though if I let batch read there too then it should be possible. What did I change whenever I update?

11 HOURS LEFT TO DONATE

Added a new setting "Python =" then a hidden setting "offlinetraceback =" (inside yiff.bat).

>>36704 ah ok that makes sense. I also wasn't aware python cared about whitespace, interesting. I tried the 6/25 version and got:
Traceback (most recent call last):
File "D:\BESC\yiff.party\yiff.bat", line 470, in <module>
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 460, in read
s = self._safe_read(self.length)
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 612, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(86542 bytes read, 66226 more expected)

and setting the offlinetraceback to true in the new version gave me more or less the same error with a couple different numbers:
Traceback (most recent call last):
File "D:\BESC\yiff.party\yiff.bat", line 481, in <module>
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 460, in read
s = self._safe_read(self.length)
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 612, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(86826 bytes read, 65942 more expected)
I tried updating to python 3.7.3, but it still throws the error

>Please fix path to Python in "Python = " setting in D:\yiff.txt
>Try again!

in my settings: Python = "C:\Python37"

>>36805
"Python = C:\" would be C:\Python37\python.exe

Quotation marks not needed even if there's spaces in path. I've uploaded new version imminently, quotation marks was from older version if it wasn't you that added it.

>>36799
I've seen that error before with proxy and I've killed proxy because of it. Looks like I need work on IncompleteRead issue...

floo.png (4.0KiB, 888x50) save_alt

>>36809
Sorry, my brain was sort of half-gone when I made that post. Thing is I did try a lot of different combinations (double quotes, single quotes, back quotes, no quotes, folder alone or directly the .exe) and they all lead to the .bat file telling me to fix the path.

3 hours left

>>36812
"Python = C:\" for C:\Python37\python.exe

>>36831
Oh fuck, of course, what the hell. I was confident I tried that too. Thanks

>>36809
>Looks like I need work on IncompleteRead issue...
does that mean there's nothing I can do on my end in the meantime?

Haven't read the entire thread so sorry if this has been reported already but this fails to download images that are embedded in the text portion of posts, see: https://yiff.party/patreon/3977850

Also while I'm at it I think it would be kinda nice if the title of the window/background color of the console changed when it's over, or if it just closed itself. IMO

First time user, I get
https://pastebin.com/9B6LHyXJ
any ideas?

>>36871
Guess I'll upload alternative yiff.bat with third party dependencies. No worries, the batch side (not running Python code yet) will fetch these for you. "yiff (requests)" is its name but hardcoded as "yiff" to load yiff.bat's folder and txt.

http://www.mediafire.com/file/xv2uko9aa3do0qg/yiff+%28requests%29.bat

>>37211
Try new version (link in OP), the requests module (in old and alternative version) get that error sometime while vanilla get different error, try and see which work better for you. I have no idea as I don't work on urllib module yet. If I do find out why then I'll update yiff.bat. Anyone knowledgeable with this can help.

>>37145
It'll change color on end of Python (CLI will stay active as batch)

>>37132

yiff.bat doesn't look at download links from outside domains. party.bat can help you with that in some way (it compiles line of links detected from post to text file per creator), you'd just have to open each in your browser and download from them. Some may not be just a link to download, and that only you can access to the downloads behind captcha and other weird download methods such as mega.nz that most script can't anyway.

>>37228
I think on line 590
if exist "Scripts\pip.exe"
should be
if exist "%Python%Scripts\pip.exe"

Changing it to that made pip upgrade itself but then

[part1] of the pastebin:
https://pastebin.com/JYfVY9BY

after pressing a key

[part2] of the pastebin

>>37237
Have you been renaming \Python37\ into something else? Check for the remainders in your code. The first "The system cannot find the path specified." is probably coming from cd, you need to get cd to work.

Updated yiff.bat with "IncompleteRead" fix attempt after I read https://github.com/urllib3/urllib3/issues/1516#issuecomment-469957705

Reintroducing "Proxy =" setting

>>37252
oops yeah, it's on C:\tools\python3\, forgot to change the cd command. Downloaded your latest version from OP and still

>Reading yiff.txt . . .
> CREATE GALLERY HTML: OFF
> LOG 404 FILES: OFF
> SHOW MEDIOCRE: OFF
> PROXY: OFF
>
> New ID! I'll check creators list from yiff.party . . .
>
> OFFLINE
>
>Restart CLI? Press any key to continue . . .

>>37256

If it happened too instant then there's probably something in my script or your computer, otherwise it has to be yiff.party server.

You can try turn on the hidden setting "offlinetraceback =" to see what's up or try yiff (requests).bat I've uploaded recently.

>>37257
>You can try turn on the hidden setting "offlinetraceback =" to see what's up or try yiff (requests).bat I've uploaded recently.
Made offlinetraceback = "true" on line 24 on the bat, result
https://pastebin.com/q3D5mPtv

>>37258

Yeah, I don't know who's the real culprit of this, it's coming from urllib module the vanilla and requests module depend on. It seems like the fix attempt has failed... There are slightly different error in requests for some reason, you can try that, or maybe it's all on yiff.party that you gotta wait for it to go away.

Updated yiff.bat with "(R)etry?" (press R to retry) after every offline error.

Untitled.png (42.9KiB, 1482x845) save_alt

>>37228
I got a lot of different errors this time after trying the (requests) version

>>37231
Look at the posts on that page, the files are all hosted on yiff.party

Thought of another minor improvement; links to the YP page in the generated html document. Thanks

Oh, yeah, updated yiff.bat with a fix for patreon_inline pictures. You probably need to delete the ender file and let the new version rebuild a new one. There are two other fixes, since I've dropped requests and use urllib directly, there was problem with unicode urls and the # character, now they'll be url-encode before urllib.

Now there are no more regex codes! They're all replaced with native python codes.

>>37277
updated to the 7/7 build and still no dice. I tried a dozen different creators, and one of them worked, but none of the others did, they all threw errors similar to these. still not sure what's going on or why the one creator seemed to work when the rest won't

>>37516

urllib is pretty sensitive to bad server or connection. I'll explore way to resume the download so that breaking connection doesn't kill download progress. I learned that not every server support resuming downloads but I hope yiff.party is it.

I've written my own download-script based on wget, and the last few weeks has been horrible. Basically any download is retried/resumes multiple times before a complete file is downloaded. Dunno if the issue is with cloudflare or yiff.party though...

>>37529
That sounds great, please share.

Updated yiff.bat with ability to resume download on retry and show 0 / 0 KB progress for fetching data (rather than downloading to disk).

Bad news? The resume download is useless. The resume download code is there in "offline" retry, sure, but for some reason yiff.party server will not let resume download on the creator data, it was exactly where the retry will come in if there's an error. Then somehow yiff.party allows resuming download on the files, but yiff.bat doesn't provide retry on error for them, they'll be skipped.

Now the resume download has some use. yiff.bat can resume on the interrupted downloads via their ".part" files.

FileNotFoundError: [Errno 2] No such file or directory: 'yiff\\7874509.cutesexyrobutts\\27418061.https_api-da.wixmp.com__api_download_filedownloadTokeneyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsImV4cCI6MTU1OTc5NDQ5MywiaWF0I.part'

I'm guessing the problem here is the filename is way, way too big. I tried throwing in a file with that name just so yiff.bat can skip over trying to download it but Windows won't even allow me to do that.

This is the file's URL: https://yiff.party/patreon_data/7874509/27418061/https_api-da.wixmp.com__api_download_filedownloadTokeneyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsImV4cCI6MTU1OTc5NDQ5MywiaWF0I

As you can tell there is no extension but it's a PNG image.

>>37683

Thanks, updated yiff.bat to truncate long filenames. You'll need to add proper file extension yourself.

New hidden setting "mathhell =" to have fun math screensaver instead of prompting you to press R to retry on offline error. yiff.bat will be in loop until success. The number used in math question is actually number of failed retries.

Been loving this project, makes things so much easier. Recently though I've noticed that attachments don't get downloaded for at least some artists. I thought it was an issue with Patreon changing things and the importer missing them, but no, if you go to the artist's profile the files are there, just not getting downloaded (https://yiff.party/patreon/106008 is an example of one that doesn't download for me). I've tried to set yiff.bat to download everything I can, ignoring any blacklisting that could be going on, but still nada.

Anyone else run into this? Any ideas?

To be clearer, it seems to be situations where there are multiple files in the "Media" category (rather than Attachments) that files go missing.

>>43493

https://yiff.party/106008.json is what yiff.bat will fetch to see the file list for that creator. If you also look there, there are also missing media files. I wonder if there's another JSON yiff.bat should look into... I'll update once I know what it's supposed to be. Thanks for pointing that out.

Updated yiff.bat to use HTML files instead. You should delete the ender files then yiff.bat will look for the missing files and download them.

Thanks for the quick update, seems to be working wonders. Got almost 200 new files with that.

29282663.2019_08_BR-02_CloudBomb3r_NSFW.jpg (384.6KiB, 1181x1417) save_alt

Updated yiff.bat with new simultaneous download capabilities, the downloads will complete quickly! Lot of code changes so I hope nothing's frell up.

EDWOIFJVAAMXXIB.jpg (140.0KiB, 1065x1600) save_alt

>>43749
I found your downloader in early 2019 and came back looking for an up to date (and working) downloader to find you're still around and it's even better. I love you, have some cub.

I've cleaned up several stupid/useless codes in yiff.bat I made during implementing simultaneous downloads... the last one had to litter the .old files in \yiff\ folder but it will not any more. Clean them up yourself.

>>43803
Thanks!

>>43920
looks like i missed alot of versions oh well
also will this work in wine and winXP

>>43982

While I specifically said Windows PC with Python 3 install, there's no guarantee it'll work elsewhere but you can try. I'd imagine it'll take just small changes to work natively on other OS too.

yiff.bat on Terminal.png (358.2KiB, 885x421) save_alt

Never mind, updated yiff.bat to work better cross-platform. It being a batch file actually doesn't mean jack to Python (with argument -x) so there's your chance.

Keep getting stuck with it saying setting up cookies followed by OFFLINE. Any ideas? Updated to latest .bat.

>>44304
I noticed that as well. This has to do with cookies... the actual browser also fails to load (I got CloudFlare's "retry for live version" page) after cookies are deleted. yiff.bat will be able to get through after retries. I suppose I could let it save cookies locally so it can be reused later if the issue keeps up.

>>44305
You make a good point. Used a web browser I hadn't to view yiff.party, it failed, retried a live version and loaded immediately. Gonna try nuking my cookies now.

Nuked my cookies in Chrome, still fails with initialising cookies. When I tried to load yiff.party it gave the CloudFlare's retry for live version which immediately succeeded when I clicked retry.

Yiff.bat will use its own cookies from yiff.party, doesn't grab anything from browser, but you'd see it fail at launch as it doesn't save the cookies to get past of this peculiar issue.

Updated yiff.bat with a way of saving cookies for next launch. The issue is consistent and annoying.

003_patreon_reward..png (925.1KiB, 1200x800) save_alt

Azzai
Last updated 8 months ago
Breakdown of missing post tiers:
azzai has 154 missing posts
$1.00 - 148 posts $6.00 - 6 posts
https://yiff.party/patreon/439611
https://www.patreon.com/azzai

Last update 2months ago
tayarinne has 1145 missing posts
$1.00 - 308 posts $3.00 - 449 posts $5.00 - 346 posts $10.00 - 10 posts $15.00 - 32 posts

File is belong Azzai
break link over "lick" draw over this artist
Aerofishtashka could be add?
https://www.patreon.com/aerofistashka

>>44366
Thank you again <3

Updated yiff.bat for better parallel downloading and removed setting "Log 404 files =". yiff.party has pagination that broke something and admin isn't going to fix the flagging system so the .log file "Log 404 files =" used to make became really useless now. Its removal has helped me clean up codes too. Again I hope nothing's frell up.

>>44082 Wallpaper pwease imma use it for my winpe
>>44393 spamfag (atleast i didnt say spergfag)

CaptureYP.PNG (650.6KiB, 1440x900) save_alt

the pic

28733835.2019_07_BR-03_Otterbutter.jpg (414.1KiB, 1181x1417) save_alt

>>45801

>>45827 doublethx haha lol

just curious who here love otters

yiff CLI.png (54.1KiB, 1766x820) save_alt

Updated yiff.bat for better CLI color as I found the way to pick the exact color I want. It's funny, Windows 10 wanted to make CLI color easy to read but it has failed to take monochromatic color into consideration. My PC has the custom CLI color for obvious reason but I want to make sure it will look same to everyone.

Just downloaded the latest version, and now it doesn't find Python anymore. Using the same path as I've always had doesn't work (still works with an older version of the batch), and I haven't had any luck figuring out an alternative that would work.

Ah, removed yiff.txt and checked to see what it says, it's looking for python 3.8, which I don't think is mentioned anywhere in this discussion. Going to install that and have another go.

OK yup, that fixed it. So if anyone else has issues running the latest version of the batch file, install Python 3.8.0.

Yes, you're right, yiff.bat was updated to work with Python 3.8 now. I update the CLI error message a little bit to make it show Python version.

Renamed party.bat (>>19344) to more.bat and cleaned up cum stain . . . (codes).

> Wanna yiff more?

>>46485 Yes please also why not rename it to m0ar.bat
>>45827 will repost my winPE with wallpaper soon

Updated yiff.bat with fix for some files with "#" in the filename failed to download.

Here's what happened: urls must be percent-encoded before downloading. The "#" will be turned into "%23" manually as some file names do contain it, then it'll not be mistaken for an url parameter. But during url splitting to encode some string to make them "url friendly" then unsplit, it only happens on string before # and not after it. No wonder some failed to download because the string after # (eventually %23) are not encoded yet.

imma say something about the script. It more or less works as expected, except in a previous version at least, it would randomly just stop downloading files, without notifying the user why.

also, would it be possible for the script to parse the posts for URLs for links to MEGA, dropbox, GDrive, etc, and output them to a text file? maybe with the post name?

Aaaaand it crashes if it hits a colon.

This is a windows problem. The only fix is to have the script intercept this, and rename it the file.

>>47505

I think the error will always be same, they'll be errored out sometime for various reasons. At least that the incomplete files will always be resumed, yiff.bat supports resuming interrupted downloads. The broken files on server is up to server owner to fix it. The only progress that is going to reset is when it's fetching new data from creator then interrupted.

>>47506

I have a separate script for that, it's now called more.bat (>>19344) >>37231 will tell you about it.

>>47511

What kind of file system are you on, out of curiosity? Thanks

Oh, yes, NTFS hates colon characters. Hard to believe that some files will still have illegal characters in, they must be coming from user of different system or renamed post-upload.

Can you tell what creator that has it?

Sorry, I was sure it was also file system dependent but guess not.

>>47521
Michiyoshi. Whenever it reaches a few files it complains. Namely, most of their attached files.

NTFS, by the way.


I'll try move.bat when I get home.

The colons will be turned into modifier letter colons before saving on disk. Michiyoshi was a fun subject to my experiment ^_^

>>47605
thanks.
I've still noticed it sometimes stops for no real reason. It's incredibly annoying, especially since i have a decent amount of creators in the list. That, along with having to manually reconfirm a retry.

>>47608

Oh, you mean that offline and retry thing? It was annoying for sure, yiff.party's connection was terrible from time to time. The traceback message isn't going to say much and it will deadlock the CLI. "OFFLINE" is blanket term for any and all kind of connection problems between your computer and yiff.party and will ask you to retry, it's elegant, and the successfully fetched data per creator will stay until you kill CLI. Keep hitting R should get it through. There is the hidden setting "mathhell =" in batch file that can put it into loop until success, but it seems there was slight memory leak in the code so your computer may not like it running for overnight (that is if the connection keeps failing).

>>47611
It's not the connection dropping. I've kinda just made an AHK script to automatically press R every few seconds.

It's just that it'll be running then nothing. No "offline" message, just the last file that got downloaded, and then nothing else.

>>47614

Oh, yeah, just need to delete the ender file then it'll look missing files to download again, and if they're already on disk, appears completed but corrupted then it's server's fault to leave them like that.

>>47621
I guess.
"del /S *.ender" works well for that

Updated yiff.bat with return of logging dead links albeit now part of gallery HTML.

>>47552
>using winblows
get linux dude its free (on a VM of course)

>>47478
gimmie this version plz
also can you post old versions on guthub

>>48020
Link to latest version is in OP like usual. I don't have any old versions as I don't save them... no point with saving old ones.

>>48030
version control anyone?

30302975.2019_09_BR-03_Vanceton_NSFW.jpg (483.3KiB, 1181x1417) save_alt

Added new setting "Patreon cookie ="

This is big update - if you know how to use yiff.party's importer then you can do it! Pledge anyone on patreon then feed yiff.bat your Patreon cookie to download the missing/dead files through Patreon site!

Browser identity (i.e. user agent) and download speed is questionable though. Probably best not use yiff.bat for Patreon if you're paranoid about what will Patreon think of it.

Added new setting "Yiff.party =" whether should yiff.bat download files from yiff.party or not. That way yiff.bat can turn into Patreon downloader no more.

>>48897
perfect now we can upload whole patreon stuff in openload (i mean alliance4creativity)

Can I get instructions in plain english?

>>51034

The CLI is short and clear. You just have to run the batch file, it'll tell you what to do next.

can we have an option to download patreon pages by themselves instead of searching pledges? it would be useful for downloading from patreon pages with lots of free content

>>51319

There is a hidden setting that will grab file lists from all favorite creators on Patreon. I added it to grab new files from the creators I stopped pledging.

can someone explain to a retard like me what this is?

Alright, I have officially found the limit of this tool.
After seemingly 128 creators added the tool just takes absolutely forever, constantly stopping from working, and generally being a pain in the ass to use.

>>51399

I recommend 64-bit Python. Yiff.bat holds file list in memory for all the creators but even then I doubt memory would be skyrocketed, still worth a try. I don't think I have placed such a limit somewhere in there.

For a quick isolation of whatever went wrong (especially for troubleshooting) I usually reduce/replace creator list then restore via undo history of the running text editor. However for a permanent isolation, a separate creator list in different folder & text file will work better. I mean it can be done. yiff.bat can be in different name (e.g. "patreon.bat") and there'll be new text file and folder associate to it, it can have different creator list and settings.

>>51353
where is it? i looked and it isn't standing out

>>51474

Hidden settings aren't meant to stand out ;) there are True/False variables inside batch file for silly/debugging purpose, "favoriteispledged =" is the one you're looking for

>>51415
Already using Python 3.7.4 64bit.
Will maybe try removing a few or something

But at 128 entries, it hard stops for at least a few seconds/minutes

>>51724
Oh, I think you ran into thread deadlock. Bad code during threading will deadlock the CLI rather than stopping CLI and it would seem to run fine but taking forever to finish, it was waiting for all threads to finish but at least one was killed. There are codes only for Python 3.8+

can we scan posts for dropbox/mega links and download them?

https://www.patreon.com/side_comics_by_amocin

Could someone add an option to download posts in dated folders to separate the content downloaded?

>>51971
by date downloaded or by date uploaded?

By the Date Uploaded from Patreon (the date on the posts on Yiff).

Updated yiff.bat to save timestamps in ender file (only work for files on Patreon site for now), it'll know which file may be edited on server, delete/rename all the associated local file(s) or the ender file to dismiss.

Write "end" in yiff.txt will cut off list of favorite creators for convenient debugging with the above creators.

Yiff.party is very unstable right now ..... OFFLINE (R)etry? Oh, please, (A)lways!

>>55008
>delete/rename all the associated local file(s)
how do i pick between the two

Updated for Yiff.party's new image server domain

It now as well saves yiff.party's timestamps to ender file. You'll get lot of "Edited on server" CLI messages, delete all the ender files dismiss them all. Things has become interesting with timestamp thing when downloading stuff from yiff.party first and then Patreon, there'll be updated post probably with new files or edited files that yiff.party isn't going to have.

>>55204

What do you mean?

>>55208
incase some badness happens and the program decides to delete everything, i want to disable that

>>55213

Oh, no, yiff.bat can't do either, I mean you must do that if you don't want that message again.

>>55215
ok

Yiff.bash when?

>>55218
you need Python and argument -x to open it. Windows just have extra benefits from the batch side.

Fantia!

Example in yiff.txt:
> 9325459.vitrex
> 5238280.tsaiwolf
>
> Fantia
> 4245.ばっちゃ屋

Or
> Fantia
> 4245.ばっちゃ屋
>
> Patreon
> 9325459.vitrex
> 5238280.tsaiwolf

Screenshot (4).png (501.9KiB, 1017x543) save_alt

i've been getting this error using more.bat, any idea why?

>>55731
I updated more.bat now that it won't encode anything, the unicode characters will display as question (?) marks, exactly what JSONs have now on yiff.party, they used to have unicode escapes that I'd encode, but now they have neither utf-8 or utf-8 escapes but instead displaying ? marks. Not good for non-English speakers. JSON is starting to cripple apart.

>>55737
thanks man, wish i could do the shit you do

Please add .replace(u"\u2028","") to the list of string methods for the creatorread variables in yp_fantia_filelist and yp_patreon_filelist. This is a unicode character known as the line separator, and it apparently counts for the purposes of the splitlines method. It is used in Japan (and probably other Asian countries) and can sometimes be found in their posts.

>>56003
Thanks for sharing! What kind of creator that do that?

2020-01-22_11-27-14.png (31.2KiB, 1331x512) save_alt

There's some error on the latest version when I comment out all the whitelist
Python 3.8.1

>>56905
I recalled that problem and I've shipped a silent fix a while ago for that, are you on the latest version?

Nvm, I figured out what went wrong, updated.

2020-01-31_13-09-11.png (37.3KiB, 1501x512) save_alt

Hi, I discovered another error on the latest version.
Python 3.8.1
Creator that I try to download: https://yiff.party/patreon/10268703
I replace the creator list on yiff.txt with other creator, and it has no problem at all, but this creator make the program error ._.
Sorry if my english is bad, and thank you in advance.

Updated yiff.bat with HTML builder, it will follow "Gallery HTML =" setting whether to build HTML files or not. It will use assets from yiff.party HTML instead of API, it will rebuild full gallery for Patreon and Fantia creators from yiff.party or via Patreon API (no support for Fantia site yet).

More.bat is obsoleted! Delete JSON files, etc if you don't need them, and delete ender files to force rebuild HTML files, HTML builder relies on ender files to conserve disk performance.

>>58045
I can see that error in older yiff.bat, the new one doesn't seem to do that, I don't know what was the fix.

>>58183
Yes, it works right now. Thank you!

Could you please add a way to have our own folder names for creators? Right now I manually modify the bat with FAVORITE.split('.', 1)[1] in a lot of places just to strip out the id from the folders just so I can find folders faster.

>>58239

Try
> FAVORITE = FAVORITE.replace("\\", "/").rsplit("/", 1)[-1]
before
> FAVORITELIST +=
in yiff.bat

Then edit your .txt and rename creator names, the slash or backslash will now cut off leading characters to be used for folder names

I'll update yiff.bat soon

weirderror.png (54.7KiB, 1348x766) save_alt

Hello, I am getting this error, appreciate any help you can give on it. (P.S. thanks for making this.)

>>58249
What was the creator? I'll look into it

I have several versions of python installed can i uninstall them and run just the current version?

>>58251

Creator is 4359251

also what does this mean? The ender file will be created/updated after (a witness of) successful downloads of every file from each creator.
Delete the ender file and restart CLI to download the missing (since deleted or previously rejected) local files.

yiff.party creator anyway to append a digit to the end of the post_file.png files so it doesn't just keep over writing them

>>58254
Thanks! Updated yiff.bat with a fix and one thing (a check for slash/backslash in creator name to use ending as folder name).

>>58253
Yes, yiff.bat only need the latest version of 3.8.x, you can uninstall all else if you don't have any of the scripts that need older ones.

>>58255
Ender file helps conserve disk performance. It's like cache, it can be outdated when something is changed (usually by you), deleting it will be like rebuilding a new cache. It's also used to save server's timestamps to help detect updated files on server compared to files on your disk and tell you about it.

>>58263

What creator that does that?

nvm i see its doing it now i thought it was over writing the post_file files in each folder but it is appending a number to the front of them

So the long file names thing is still not fixed.

line 443, in echosession
elif not get(onserver, todisk=todisk, referer=referer, threadn=threadn):
line 364, in get
with open(todisk + ".part", 'ab') as FILEwrite:
FileNotFoundError: [Errno 2] No such file or directory:

I've tried fixing this myself but ended up with a file downloaded without extension. But the original URL had it (at least in one case) so it should be possible to properly truncate just the filename.
If you need examples, affected authors are prama and lk00.

>>58343
Well, it appears yiff.bat would have problems if it's behind too deep in path name and/or creator's folder having a long name. Yiff.bat allows 200 + 5 (filename + ".part") in filename and Windows allows 246 path characters (including drive letter and slash characters) (TIL about that now), you should figure out where to put yiff.bat in and don't let creators have a long name (they can be easily manipulated in yiff.txt).

>>58355
Thanks. If only I knew about it being 200. Maybe at some point it would be better to make it auto-adjustable based on current path, or configurable, or maybe let the user know he has to move the folder closer to the root if some files can't be downloaded.

Some other ideas:
- Long filenames for lk00 could give some idea about how to better truncate such names - e.g. "https*.patreonusercontent.com" pattern would recognize such many of useless names.
- Files without extension could still be named properly automatically since most file types are easy to recognize by their first 4 bytes. This may create issues with recognizing already downloaded files, but it should be possible to just test the existence of any file with specific name and any of supported extensions.

>>58369
Yiff.bat will grab new filelist from servers and we'll have to be careful what to manipulate each file in the filelist, renaming them from special variables will not stick for the next time or being same for everyone else.

The new folder replacement thing you integrated works well, I was just able to replace all the dots in creators names with a forward slash. Thank you!

Some issue with ender file
29099894.Kupocun - not downloaded, "The ender file will be created/updated after (a witness of) successful downloads of every file from each creator. Delete the ender file and restart CLI to download the missing (since deleted or previously rejected) local files"
No ender file, no folder. How to force download without checking?

Some issue with Python
with clear yiff.txt all working (Some "Added new setting")
With second start - "Please fix path to \Python38\python.exe in "Python =" setting in yiff.txt"
I have a non-english symbols in Python path (User name). Is this make problem? I cleat yiff.txt before each start

Suggestion: Can download video from yiff.party fantia
Some user have a video, but Yiff.bat not detect it, even though all file format is commented (whitelisted)
For example, this post: https://yiff.party/fantia/posts/256090
Thanks in advance, and sorry if my english is bad

now every new entry downloads in the same folder
help

>>59250
Just delete ender file will force yiff.bat to check for missing files.

Updated yiff.bat to handle unicode characters now too. It wasn't issue with Python but rather in batch side.

>>59299
This video's filename ends with .MP4, I guess you forgot to comment out "!.mp4" from blacklist.

If there's no whitelist then blacklist will take place, and if there's not any of it either, yiff.bat will download any.

>>59392
Fixed! Sorry

>>59550
Omg, you're right, my bad. I'm not paying attention to the note that say if all of whitelist is commented, the blacklist will have an effect 😓
Sorry for bothering you.

>>59550
>>59392
thanks for this m8

Requirement already up-to-date: pip in *\appdata\local\programs\python\python38\lib\site-packages (20.0.2)
Collecting Pillow
Downloading Pillow-7.0.0-cp38-cp38-win_amd64.whl (2.0 MB)
Requirement already satisfied: PySocks in *\appdata\local\programs\python\python38\lib\site-packages (1.7.1)
Press any key to continue . . .
Then:
Exception in thread Thread-9:
Traceback (most recent call last):
File "*\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "*\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "*\yiff.bat", line 443, in echosession
elif get(onserver, todisk=todisk, referer=referer, threadn=threadn):
File "*\yiff.bat", line 324, in get
echothread(threadn, f"Downloading 0 / 0 MB {echourl}")
File "*\yiff.bat", line 316, in echothread
print(echo, end="\r")

>>59686
I'm not sure what could go wrong? I've uploaded a silent fix over silly mistake but haven't seen like this.

>>59717
Version I got few weeks ago still works fine, and it doesn't require Pillow.

I went to \Python38\Scripts in cmd and executed "pip.exe install Pillow" - after this I'm not longer getting screen where yiff.bat tries to install Pillow and fail, which was the first part of the issue. Maybe the handling of working path is being done incorrectly.

But the second part stays the same - Exception in thread Thread-9 etc.

dogbone.jpg (291.3KiB, 979x515) save_alt

>>59717
Dogbone continues to break mine in your more recent versions. The filenames seem absurd.

>>59824
yea I get the same issue with https://yiff.party/patreon/7874509
theres another traceback i got that said something like "too many values to unpack" and also multiple 404's. looks like its server side. the script worked fine after i commented out the affected creator, but that's not a real solution.

>>59824
>>59855
Added "[:200]" (200 characters limit) to the more filenames I forgot to give, there wasn't a limit without it that couldn't fit in Windows' 246 path limit. Really sorry about that!

>>59814
I wiped Python installation and reinstalled with the latest Python version (64-bit) from download page and couldn't encounter this kind of error... and besides I think that's unrelated to Pillow by the look of the error. There has to be something else causing that.

threadnumbers.png (19.2KiB, 816x491) save_alt

>>59871
Encountering the same thing as >>59686 , whatever that is. Well, thanks for the time it worked I guess, waiting for updates.

First pixivutil gets broken by captcha/sessionkey bullshit, now this. Not a good winter.

The CLI color should be blue when running Python (the pure batch side will be yellow on black) but I know that older Windows version couldn't produce it because of no support for the ANSI escape codes where the blue is coming from. I don't have access to the old Windows versions. I think there may be underlying problem in older Windows versions if >>59686 is also on one of these versions too.

Would changing code page to 65001 be causing this? Try put double colons ("::") before "chcp 65001" and then run it.

If no differences, try delete lines of Pillow/PIL related codes and then try again, etc...

For random error with vague message like this I can't ship the fix without being able to reproduce it myself.

Where can I find the download link to the latest yiff.bat?

I suppose the OP is getting updated with a recent downloading.

While using it, I have the exact same bug as the yiffy.bat from 6 months ago... it stops downloading without any errors. I haven't dropped my connection at all...

>>59946
Yes it's in OP

No idea what went wrong for you though. If only you knew of Python it'll make troubleshooting easier!

Is it supposed to look like this if I want all of the extensions in whitelist and blacklist to get downloaded?
https://i.imgur.com/VBGQjfU.png

>>59959
Sure, if that's what you want blacklisted ones to become whitelist just did you made them to.

> .whitelist
> !.blacklist

But you can also delete them out of text file if you want to download any rather than just whitelist.

>>59922
I'm on Windows 8.1. Guess there is something about CLI not supporting UTF-8 as long as the font used does not support it?
I see from procmon that the exception happens before any UTF-8 file path is accessed though.

Commenting out "chcp 65001" solves the error, but apparently it's not the proper way? E.g. I might end up with broken file names maybe?

>>59970
Python can handle utf-8 fine, pure batch can't, chcp 65001 so path to Python can be captured and run it (some users have utf-8 in their names), I think it became unnecessary once variable has got the path to Python.

I've updated yiff.bat that will use code page 65001 to capture path to Python then revert back to code page 437 which was default of cmd.

offtopic: this is a great tool, do you have plans on making a similar tool for rule34hentai? there's a lot of patreon stuff there too. just a heads up, the admin on rule34hentai is using cloudflare now, which makes it much more difficult to rip (have to add the cookie value every 60 minutes)

>>60028
Thanks, it works now.

But found another issue. Either shared files section support is broken or I don't understand how to make it download those files. I think it worked properly like a month ago.

Wow, I had a slightly older version of this and was having that UTF-8 issue on my Windows 7 box. It made installing missing packages fail and also many downloads. Came here to report it and found the update and it works great now! Thank you for all of your hard work, this is a great script!

As for any improvements, I find that when the script is finished it goes to purple text on a black background. Maybe some color variables that can be edited in the .txt file to customize them would be awesome, setting to output to a log file, and whether to exit when done, wait for pressing any key, or waiting X time before auto running again would be awesome. I plan on running mine on a schedule so a clean exit would rock, but if run in an active console window it would be nice to have it re-run like once a week or something automatically.

Apparently Yiff.party is on little jittery that trailing spaces could be added randomly in any line in HTML. Yiff.bat went little wonky because of it until I noticed it now... there are few "Already downloaded" stuck messages because of it, delete ender files or waiting for a new file will dismiss it.

>>60105
Holy fuck, thanks for notifying! Yiff.bat collects assets from HTML than just look through for links, I forgot about shared files. Updated yiff.bat!

>>60126
Yes, sorry, a little code change can be really bad for somebody else. I was on Windows 10 so I wouldn't know about the others. As for purple on black color when CLI ends, I've wanted to brighten purple a little bit when CLI ends but I can't find a way to do it. I know of ANSI escape codes but can't apply to older text, we are limited to 16 colors. I wonder what's the ideal color for when CLI finishes? I get that purple can be kind of too dark on black.

Why wouldn't you want to edit batch file on your own? You can tweak it with other colors, your ways, timing schedule, etc., and there are hidden features in there as well. I don't think text file should have custom UI settings.

The HTML files ("Gallery HTML =" in yiff.txt) can be used like log file, you get to view the result in browser, or you can leave CLI on to read later.

>>60141
It downloads files now. However, I began noticing broken archive (zip and rar) downloads. From both posts and shared files sections. I don't think this happened before.

>>60166
Was it download completed and corrupted and it's down to yiff.bat? Error downloaded partial files can be resumed next time, so progress isn't lost. The corrupted file that's not ending with .part in filename is bad news.

What do I put in whitelist and blacklist if I want to download EVERYTHING including shared files? Thx

>>60181
Delete all then it'll download any

>>60182
Delete everything in whitelist too??

>>60183
Yes

It's trying to download files from dropbox and the filename for those files look like this "29817020.https_www.dropbox.com_temp_thumb_from_token_s_" with a lot of numbers at the end

not sure if it's a bug or intended

>>60199
If it's same as what you'd get through normal browser beside an id assigned (for collision resistance) and the 200 filename characters limit it's intended.

>>60172
Yes, example is a supposedly downloaded .zip file without .part extension, with size of ~10 mb while it should be ~300 mb.

>>60221
Is it down to yiff.bat or? I mean, if it's corrupted on server regardless of what browser or downloader used then there's nothing we can do really.

>>60225
No, I could download the same file with browser without issues. And it happened with multiple files during one download session.

I've compared the files and broken ones appear just cut - the download was interrupted by something but the script decided it was downloaded fully.

>>60227
Wow, that's so weird

Was it the code page again? I haven't seen problem downloading files but still I don't have older Windows versions to test.

>>60228
What do you mean by code page? I'm using the latest version where chcp 65001 is followed by chcp 437. I don't see how this or Windows version are related though. It seems there is no code in the script that would check the download completion and restart it in case it's broken. I mean, it's okay for cases where server can handle all requests without interruptions or freezes, but yiff.party is not responsive in 100% cases for me even in browser.

>>60233
Well, if we can be sure it's not code page glitching out the cmd somehow then we'll ignore it. The error message while code page was on 65001 was super strange so I'm not going to rule it out yet for the other strange things from you ;)

There was wonky connection between yiff.party and my pc but haven't seen downloads completed and corrupted where browser could download it fine. I may have missed the certain timeframe when yiff.party tells download is completed prematurely.

As for when it's really completed I think you may be right, it's up to server to decide. Once completed, file will be renamed to drop the .part extension, .part is just way to tell yiff.bat it's incomplete and must be resumed.

If even yiff.bat today still doesn't work very well for you then maybe try disable chcp and then if you want to "resume" the corrupted file you could try put the .part extension back in there.

If your drive is busy with writing or reading, and/or if you are download something (even if your network isn't being throttled), the .bat will simply freeze and not do anything.

I've removed both chcp lines from script. Next run I came up with 1 broken file. So I assume this has nothing to do with code page.

From my experience with mass download scripts, if there is no check for download completion, you end up with broken files sooner or later.

>>60360
Thanks, so it has nothing to do with chcp

While broken, is it same as downloading with browser? Because even if there is check in place it's impossible to redownload for a better one from same source (corrupted on server).

Out of curiosity, what do these other scripts usually do when they find out download isn't really completed? I may follow their idea.

>>60387
I'm checking each case before reporting. If it can't be downloaded properly (e.g. was uploaded with errors on server by original uploader), I'm not mentioning such cases.

The last one I mentioned was 56 mb file while it should have been 123 mb. I removed it and run the script again, and it was downloaded fine this time.

From my understanding, you just copy the remote file size in bytes into variable before the transfer (it seems yiff.bat already does that - it's used in displaying download progress) and compare it with the size of the file after the transfer is complete. If sizes are different - it means something went wrong with the transfer - so you remove the downloaded file and download it from scratch again. I also wouldn't try to resume that since there is no easy way to check whether the connection was cut or if server gave wrong data at some offset.

Also, I never caught the moment it saves broken download during runtime because I don't watch it. Would it be possible to save all the output into the text file?

>>60393
Thanks! Sorry to bother you, I was sure maybe you couldn't differentiate between corrupted on server and finished downloading incomplete data but looks like you did, your findings are valuable to me. I'll plan to put a check so whether if fetched data does match to total in the header, those less than total will stay in .part filename for the next time if that's ok with you.

There's module named Pillow that you may notice it installed recently, it loves to complain whenever image file is corrupted, we'll be using it to scan for corrupted files on disk, it's now part of hidden settings, it'll take a while on large collection. However I'm not sure what else can handle corrupted files if they're not images.

Updated yiff.bat

>>60412
I'm not sure why Pillow is even included though. Sure, it's going to be tough for me to check by hand if some image is broken, but I've noticed broken archives because they are easiest to test, then I've noticed broken video files because their size should be much bigger for a specific artist. If you want to make use of it to somehow detect broken files within a huge collections users have accumulated, then it might be a good idea, but otherwise I'm just going to re-download the whole thing from scratch (hopefully later after some additional measures are implemented) and use folder compare or dupe finder software to get rid of broken files.

Other than that, I can't think of any better ways to detect broken download if it's not a file size check. There might be cases where a single byte is altered inside of a downloaded file, but that would be most likely due to hardware issues on user's side. I've been downloading tons of stuff for years using many different tools and I don't remember many cases where a broken file has been downloaded at 100% of its size and a re-download fixed the issue. In absolute most cases the file was broken on server side, so a re-download just confirmed the issue.

I wish there would be tools that could do what Pillow apparently is about - just scan a folder with image files and try to detect files without an proper ending bytes according to its file type, which would mean it was downloaded not at 100%. Never found such a tool in a usable form.

>>60141
Oh, well, the reason for having such settings in an external file are so when you update it and I update my copy with your new one, I don't have to go and make all of the changes again. And a log mostly just to show errors. I scrolls past the first stuff so I might miss it. But either way, I am very thankful for your hard work on this! Thank you so much!

After I've run the whole list from scratch with new version, here are things I can report:
- Downloads could break regardless of the file type. I found broken image and video files on top of archives that were downloaded before chcp commands were introduced.
- There is still an issue where the process just gets stuck or appear frozen, and python.exe produces zero file activity. Maybe this happens when server becomes unresponsive, but that may mean such cases are not handled. E.g. there may be no timeout limitation set to catch such issues.
- It appears that new version just leaves the broken transfer on disk with .part extension so it could be resumed next time. This seems to work fine and I end up with proper files after running it again.
- However there are cases where such .part files are not getting resumed, no matter how many times I run the script again. That happens when .ender file exists for the folder. My understanding is that .ender file should only get created after all files are considered fully downloaded. So maybe there is a bug that makes the opposite possible.

>>60469
Nah, I haven't done much work. Thank you for your work and updates.

Sorry to bother so much, but I seem to be running into a new error. I believe it has to do with some new content as it did not have issues before. I made a pastebin of the error on first run and then the error when it is run again: https://pastebin.com/9aWNGbtK
It does not generate an ender file. The first run claims to have downloaded a file after the error, but has a similar error about "MB" being called before being assigned. The second and subsequent runs have a broken character on the "Exception in thread" line.
As mentioned by choin, I did get through the same user's files in previous versions but had a lot of files named:
25525220.file.1
21800119.file.1
23451643.file.1
24482808.file.1
So, I deleted them and the ender file so it could try to re-download them and left me with the error.
Thank you for your hard work, it is very much appreciated!

>>60727
It tried to download https://yiff.party/patreon_inline/15727305/bd291c3b2731cfa2a1554e36db6e75e230dacdd6.html from https://yiff.party/patreon/3048343#p15727305

Happens again in https://yiff.party/patreon/3048343#p14773846, is also .html

And for some reason we can't see .html's file size from its header "content-length" (who decides on this?) Problem happened because the variable named MB wasn't established this time that the header "content-length" returned none. While yiff.bat will grab HTML as part of reading creator data, the fetch method will do KB without seeing total, only the download to disk method will do MB. There was variable name MB with static value "0 MB" in else statement in older yiff.bat I thought was redundant, now I put it back.

>>60424
>>60698
The check for a match to total from the header before renaming to drop .part extension may be best we can have in a while, it should be hard to download anything corrupted that is just you. But for a byte being a little off, for this we can only use tools to verify file integrity, or do what you did, a redownload and compare.

I've added support for scanning archives with a help from 7-Zip. It's now part of hidden settings. Really byte accurate as I've tested. I tried FFmpeg for video integrity but not byte accurate so I dropped it.

The script will ignore video files with the same filename in the download folder from the same Patreon user.
It's doing that with Nyamota (2383192)

>>60751
Amazing filename collision. I'll have to rethink how to save them both. The what seems to be collided in filename was always been just a duplication.

Oh no, they aren't, check again?

Huh, I said that I end up with .part files when .ender was created. It turns out they stay like that even after I remove the .ender file. I've checked and found that even though they have .part extension, they were fully downloaded, so they should have dropped the .part extension. There are 2 cases in my folder now and both appear to have the duplicated filenames. E.g. I have a 178kb xxx.Image1.jpg file (note the capital I), and 131kb xxx.image1.jpg.part file. One of them is resized or also has added Patreon logo. So I guess the .part does not get dropped if the file with the same name already exists. In my case both duplicates could have been detected with a case sensitive check, but I'm not sure if there is no similar cases where names are fully identical.

>>60749
The file size check is enough really. I think file integrity checks are only useful for finding files that are broken on server. But it seems there is still little use in that since I don't expect broken files will be fixed by anyone. I noticed there is a "Flag file" feature but I can't see how it could help anyone. Also, there are many files that can't be downloaded - some give error in yiff.bat because they do not exist (404 error), some get downloaded but they are just xml/json files (under 200 bytes or so) with 403 error texts.

>>59871

Still having issues with certain files, to be fair I have a lot of creators. Mind seeing why it wouldn't be happy with these? Especially with Primeleap content that I can view in browser normally.

>>60781
Forgot the damn pastebin: https://pastebin.com/raw/PtyHy1U3

>>60749
It works for me again! Thank you! That pre-pooukatibg MB with 0 fixed it up! Awesome work!

>>60994
pre-filling ... not sure what went wrong with the autocorrect o r whatever ... but I meant pre-filling or pre-setting or pre-populating or something like that.

Added new hidden setting "collisionisreal ="

Turn it to True and delete Ender file and see if you can get anything that really isn't duplication. The first thing if yiff.bat noticed there's a filename collision (having noticed multiple urls associated to it) it'll compare the size of a file from its header "content-length" to the size of a file already on disk before downloading the differences. I've got so far is duplication in different file sizes though...

>>60765
Sorry, the description for Ender file is kind of outdated, today Ender will be made after finished downloading of this creator with at least one success download. It'll remember post id to which all the files have been successfully downloaded and not the others with a failed download.

Windows doesn't want to have file with same filename case insensitive to same place and I thought Python would know that. Not sure how it got into downloading part in the yiff.bat.

>>60782
primeleap_data urls are magically not percent encoded like patreon_data, strange move, but I fix it anyway.

>>60994
>>61082
No problem, it was silly mistake

Small fuck up, I've known since like late 2018 and been like this that there's inferior duplication in certain url for the same filename, recently yiff.bat has downloaded the inferior ones from these urls. If you download new stuff with yiff.bat updated last night you'll probably need delete recent downloads + ender and try again with new yiff.bat I release today.

Where is the linux version of this scraper?

>>61245
>>55223

>>61153
Good thing: Now more files got downloaded that were not downloaded before, marked with (2) in names.
Bad thing: The 2 files I had issues with now break the script. I've removed all related files and .ender ones, this is what I get now:

Exception in thread Thread-15:
Traceback (most recent call last):
File "\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "\yiff.bat", line 483, in echosession
elif duplicate := get(url, todisk=todisk, conflict=conflict, referer=referer, threadn=threadn):
File "\yiff.bat", line 402, in get
os.rename(todisk + ".part", todisk)
FileExistsError: [WinError 183] : 'yiff/387543.janong/3309646.Image1.jpg.part' -> 'yiff/387543.janong/3309646.Image1.jpg'

It seems both 3309646.Image1.jpg and 3309646.image1.jpg are getting processed at the same time in parallel threads, so the script is unable to detect one is already downloaded by the time the second one gets written.

>>61284
Oh! That's it. You gave me hint about race condition of parallel threads. Actually, there's a place that takes same filename but different case as different thing before they are given to parallel threads.

Updated yiff.bat, but the fix is little rough, the conflicting filename will follow original's, like "image" becomes "Image" that the original got there first, if that's ok with you.

>>61315
That did it from what I see, thanks. The order is not important to me since it's hardly possible to restore the original order from the data Patreon holds anyway. Each name containing the post id is already enough for sorted presentation.

Btw how should a one-line filter look like if I want to not download all the files with names that contain "badword" (ignoring extension)? I guess I just want a simple example I could follow for mediocre.txt files.

>>61317
Yiff.bat will reject all files with filename "containing" any of the patterns (as lines) in mediocre.txt

"badword" from mediocre.txt will match
> 997.superbadword.jpg
> 998.psdbadword.png
> 999.justbadword.psd

Except that pattern with period at the beginning will match the filename "ending with" rather than "containing".

"psd" from mediocre.txt will match
> 998.psdbadword.png
> 999.justbadword.psd

".psd" from mediocre.txt will match
> 999.justbadword.psd

"998.psdbadword.png" got through because didn't have ".psd" in the end.

And well, if you want to ignore extension then maybe you'll have to look elsewhere to capture, one more trailing character can be effective enough to ignore extension.

Added new setting "Fanbox cookie = "

For actual Fanbox site, pledge anyone on Fanbox then give yiff.txt your cookie and these creators

This will give you the idea where to put Fanbox creator in yiff.txt
>>55376

>>61398
where do I find the 40 character cookie? also are you sure you mean pixiv fanbox or do you mean fantia?

>>61437

Pixiv Fanbox

Well I guess it isn't always 40 characters in length

https://github.com/OpenYiff/Kemono/wiki/How-to-get-your-session-key

Updated yiff.bat that Fanbox cookie isn't always 40 I guess.

>>60755
Thank you for making it easier to download.

Does the script handle download URLs from mega and other cloud services too? If not... I think it'd be great if the script could submit a text file with all of those links

>>61461

These are arbitrary urls and will likely need your attention, some may not be just a link to download and others may need human intelligence/interaction like there's a download behind captcha.

Yiff.bat can extract them and format them into list, not exactly to text file but, they will be collected and hyperlinked to top section of the HTML for easy to access so you can download them from browser quite easily. Turn on "Gallery HTML =" to enable building offline HTMLs.

Updated yiff.bat to output \yiff\style.css that you can edit to customize HTML and CLI.

>>62558
Awesome! Thank you!

I've added more color that was stuck in HTMLs rather than CSS.

Things will look wrong with new yiff.bat, need to delete or update style.css yourself to fix. Just for the last time I promise.

>>62647
I think you're the person that asked about CLI color, you're welcome, I thought .css seems like a nice unusual place to edit to customize CLI too!

>>62698
Yes, it is me, the one who requested custom CLI color options. Thank you! I don't know how to do custom names on here, but I am always thankful for your hard work! Thank you again, and keep up the great work!

>>62775
"tcolorx" is yiff.bat's main color, it stands for "text color end", it was supposed to be there after text in different color (the "show mediocre =" setting in yiff.txt will show those other color texts), it's now also used as main color of the CLI.

To understand how ANSI escape colors work I'd be following this format "\033[{Y}m", Y can be Background, Foreground, or Reset, multiple combination is possible with semicolon. Insert Y with
> Background: 4X
> Foreground: 3X
> Foreground (bright): 9X
> Reset: 0
X for command's 124 (RGB), 356 (YellowPurpleTurquoise), 07 (BW) otherwise 8;2;0;0;0 for 24-bit color (0 in last 3 segment means black)

html.png (908.4KiB, 1656x1792) save_alt

Added new setting "HTML server = "
Updated sample CSS for light/dark theme
HTML will load images whenever is within browser view (lazy load)

HTML builder includes many new features: avatars+covers, search title, resize images live, toggle sources live, always open links in new tab.

I think HTML got good enough of new features for now.

>>63323
Hello! Sorry, I have not had much time to test out the new features. I updated to the most recent version uploaded ( Uploaded: 2020-03-12 19:45:21 ) and am having some issues. I pasted the entirety of the output here: https://pastebin.com/Twtb35en
The script just hangs there, not progressing after that. I gave it a couple minutes in case it was timing out.

>>63574

Oh, you're right, I overlook that, the new feature of the recent version is to grab avatars+covers from official sites instead of yiff.party, but apparently they require Patreon cookie (doesn't matter who you pledged), I had Patreon cookie with me.

I'll update to not do that when Patreon cookie isn't available.

>>63574

Also there is no timeout, CLI will stuck indefinitely when one thread was killed (got the exception you speak of). That is called thread deadlock.

2628982_MokiKun__4.jpg (542.6KiB, 800x1131) save_alt

heard this thread had cubs lawl

>>63576
Thank you for the update! It works great now!

>>63577
It would probably be nice to have it do a notification or something if it does a thread deadlock ... like if it could detect it and then break out of it or something.

I have not had chance to play with the colors ... I tried a bit and it wasn't responding to my changes. Maybe I was doing it wrong. If I understand correctly, even though there are 5 variables listed under 'cmd', only the 'tcolorx' modifies the colors when the batch file is run in the command line window? Also I think I need a wiki page or how-to article for these color codes. I believe I understand the formatting of the other color variables, but the 'tcolorx' confuses me where it has the number '2' ... Sets background, number 2, reset, foreground (bright), 128 for some reason, set foreground (bright) again ...?

Thank you for the fast fixing of the batch file though! Excellent work! I love how you combined batch scripting & python in one file and I wish I knew python more.

>>63704
Thread deadlock after an exception is never a good thing, there are many ways to make it happen and I'll be doing my best to make sure it's not happening.

Alright, if you know of decimal value of RGB channels then you'll get it no problem.

In this example I'll do rgb(16, 0, 32) for background (very dark purple) and rgb(128, 0, 255) for foreground (electric indigo), the tcolorx for this will be "\033[48;2;16;0;32;38;2;128;0;255m" notice where each number go.

9X is just command's set of bright colors, 98 isn't valid, but 38;2;255;255;255 is already white so you can go with 38 for ANSI escape color for foreground.

Also you gotta be on Windows 10 for these ANSI escape color to work.

>>63714
D'oh! I run this on a Windows 7 box (less network chatter and weird stuff) that does all of my automated stuff, so that is probably why it wasn't working. But now I understand the codes! Thank you! Hmmm ... it does change the text color when it runs though, yellow for running and purple when done. I really love the changes to the HTML though! Now it is easy to see everything by scrolling at a good size! Keep up the great work!

Now images in HTML can expand/contract like image boards would. Code stolen from 4chan ...

There was lot of code behind 4chan I've boiled down to codes used to expand/contract images, took long time, but good enough to use for my HTML builder, now I can go to sleep!

what does "Show mediocre" do

I'm always getting errors with downloading and I'm pretty sure it has to do with yiff.party temporarily blocking me for excessive requests...

Is there a way to add a delay or something?

part.png (28.3KiB, 379x618) save_alt

The script finished doing its thing and I've got a lot of files like this

Should I delete it and start over?

>>64212
>>64215
No, they can be resumed next time, that is what .part is about, they're incomplete files waiting to be resumed. Keep them until you find that they can never be resumed for some reason.

I haven't seen yiff.party block users over excess downloading, yiff.party is known to become unstable sometime, maybe that's been what you're experiencing. If yiff.party didn't block you here then it has to be just stability problem.

>>64199
It will show you files that yiff.bat refused to download because of your filter list.

>>64221
I've removed everything in my blacklist and whitelist, is that a wise choice if I want to grab everything or is there a better way to go about it?

>>64238
Yes that's how you get everything

I need to know if I'm using this right.

I run the bat file. I follow the instructions. I wait for the CLI to do the thing and when it says "restart CLI?", I'll close the entire window, delete the ender file and run the bat file again. Is this correct?

>>64283
Delete ender files only if you have changed filter list or have deleted something and need to redownload them again. That's pretty much it. Ender file is checkpoint record of previous downloads and for the next time it'll check for new downloads only. The ender file will not remember failed downloads so they can be retried too.

On "Restart CLI?" you can also close it and come back at any time for new downloads.

>>64295
Should I be deleting the ender file if there are files with ".part" in the filename?

error.png (3.5KiB, 154x123) save_alt

Why does this happen? I'm on a 1000/100 line...

>>64308
No

>>64312
Not just you

Please add the option to only have 1 active download, I think it'll solve the issue with error downloads

>>64459
You can tweak batch file to work with one thread and see if it's working better for you. I still think you'll still likely to get same error plus slower. I'm interested to hear from you how it perform with one thread.

Also has .part file keep growing? That's great, restart CLI multiple times will get everything downloaded eventually.

>>64468
Sorry I'm a brainlet, how do I tweak it to work with one thread?

Find the "for i in range(8):" in batch file, change from 8 to how many threads you like. There are two instances of it, first one is for creator list and another one is for downloading files.

Hey, I get similar errors as well. Changing the .bat to 1 thread changed nothing to me.

Question out of curiosity: Why does it state "already downloaded" every time? Shouldn't the already downloaded files be added to the ender file? No matter how often I run it, it always says "already downloaded" at the same files.

- - - - 184944.MariArt - - - -
Already downloaded: yiff/184944.MariArt/27230810.ruby_refsheet_by_Mari_HD.jpg
Already downloaded: yiff/184944.MariArt/27230810.ruby_refsheet_by_MariNSFW_HD.jpg
Error downloading: https://data.yiff.party/patreon_inline/27230810/9ea83740adbb87026aa2d3e7ec7c8f70952b89d6.jpg
Already downloaded: yiff/184944.MariArt/22313748.barzillai_commissionWIP5.jpg
Error downloading: https://data.yiff.party/patreon_inline/22313748/c4f8970db25658d06e5b3864e973527f91244a76.jpg
Error downloading: https://data.yiff.party/patreon_inline/22313748/1c3a1dd0f79ed72b3c9f4b7dff7a712f53697031.jpg
[...]

>>64687
Ender file will ignore the post ID with a failed download, that's why the other downloads associated to that post ID couldn't be remembered.

some tracebacks ive been getting:
https://pastebin.com/g21CWj0L
https://pastebin.com/rNPSNe7x

the first one is when i try to use the fanbox downloader with this list:
135770.コンノトヒロ
5948997.悠々
14248629.プニョン
29759565.まんなく
but "14248629.プニョン" downloads fine when the other id's are commented out

the second one is from the fantia downloader when i use this list:
163.ポザ孕
2717.haruhisky-はるきち
6565.つんべじ
7895.あんこまん
9681.lambda
15061.てだいん
I have others in the list but the ones above have to be commented out for the others to download

>>64758
Thanks! Updated, I haven't pledged any more than one person on Fanbox so the fix for it is untested but looks promising.

Where do I get fantia and fanbox versions? It seems it doesn't accept any non-patreon id.

>>64838
>>55376

>>64840
Oh thanks. Somehow I didn't tnink the name part is important, and got more confused when realized the script can become unable to read the .txt if I change its encoding. I supposed only the id is important and the name will be added automatically, same as with patreon. I see now the name can be anything though.

I think fantia gallery html files are not finalized yet? E.g. the part above the logo appears to not work, and I'm seeing "0UTC" where the dates are supposed to be recorded.

Untitled-1.png (627.3KiB, 1063x1390) save_alt

>>64773
fantia and fanbox downloaders work properly now

but the fanbox downloader doesnt download post attatchments like zips and psds, is that intentional?

>>64847
Yes ID and which site they belong to is important part, then you can give them names you like. You can also add slash/backslash in creator name to use ending as folder name.

There are linebreak in Fantia while Fanbox and Patreon don't and I was planning to loop through links split by linebreak, I completely forgot that Fantia has linebreak. The top part in HTML went fuck up where the list of links was supposed to be. Thank you for noticing it, sorry for lack of testing.

Fantia receive 0UTC because they don't have timestamp.

>>64852
Yiff.txt probably has .zip and .psd in your filter list (if you haven't removed them yet), try remove them and delete ender file then try again.

However the person I pledged to doesn't have a post with attachments like that so I can't really test myself.

>>64525
Getting less error downloads because it's not downloading as many as it can at the same time. I'm fairly convinced that there'll be zero errors with downloading if there's only one instance instead of several

>>65259
same here

Nahhh, the more threads, the faster it can go. But the site has times where it loads super poorly due to the heavy load. Heck, the other day, I couldn't even load this page because it would stop part way through. The main website is being hit pretty hard. Sometimes it chokes. The best thing to do is to run the script during low traffic times of day and keep trying until it finishes the downloads. Restricting the number of threads will only mean that you have to wait for each and every file to time out and die before moving on to the next, where you could be timing out on 8 or so at once and get your results fast & be prepared for a new run. I am a fan of multiple threads. Use that bandwidth when you can!

Just my two cents.

Updated yiff.bat to check for creators you still have access to their paid content on Patreon for a month after you stopped pledging. Not sure how reliable is it though.

>>66022
Well said! Yes, in my experience, less threads to have less errors is just placebo.

hey umm sorry for ruining the party but how do i download thots on YP using the script (to be clear this is NOT a request thread)

>>66022
There are less errors with downloading with one thread and there'll be zero errors if we can only have one instance of downloading

>>66749
I do seem to have a pretty epic network connection and have things run at a time that is typically "at night" for most EU locations and probably US too, so it is likely a great time to do things. I have great success with downloading using all of the threads and seem to never end up with .part files either.

My best thoughts would be that maybe an option in the yiff.txt that allows one to set the maximum number of threads could satisfy the varied types of connections.

>>66575
Thank you! I was worried that the tone would be an undesired one, but I mean well. I imagine that if some have slower connections, that one thread might grab a download and the others would time out and give more errors. Not everyone has a decent connection, I suppose. Sometimes slow and steady wins the race?

I do love my multiple threads though.

>>66649
Are you asking how to run the script to download your favorite items from yiff.party, or are you asking how to find good stuff on YP? I found this site that has "top" people in different categories ... https://graphtreon.com/top-patreon-creators ... maybe that helps? Otherwise, I think this is more of a "if you know what you want, this helps you get it" - and in an automated fashion so you don't forget or miss stuff.

>>66761
It has nothing to do with how fast or stable your connection is. This site will temporarily block you if you make too many excessive page loads.

>>17695
Wtf?!?! You just insulted KDE fans, fucking scum?!?!

--
Anti-Yiff 04/02/20(Thu)08:02:20 No.67355[R]
>>17695
Wtf?!?! You just insulted KDE fans, fucking scum?!?!
--
mirroring incase comment is gone

>>67355
hey atleast he didnt insult winblows fans lololol
(i cringed while making this posts my hands shake the keyboard)

>>66761
i was expecting some chaggotry (kek) but thanks for (sort of) giving me useful info
also does using a hyper V affect download speed

>>67389
I doubt that using Hyper-V would change much, but it all depends on your network configuration. I have seen a lot of crazy stuff. It shouldn't normally, but everyone has a unique setup.

As far as the site blocking you, if you run into the issues where you can't download, then you should try loading the site or one of the images manually and see if it gives you an error. In my experience, whenever I have seen issues, the site also was timing out when loading. But not consistently. I would open, say, 5 to 7 images in tabs and less than half of them would just time out after a long time when the others load slowly. This is not a block, this is the site's web traffic being saturated and not being able to respond to all of the requests.

But if you see a legitimate block (will look something like this: https://us.v-cdn.net/6031209/uploads/drupal_attachment/files/2019-06/26/99/2699b06143a761568086e2a4278e290d.png - or you will be able to load the site though your cellular network, but not your computer) - then that shows that they implemented a blocking situation. It would be good to know if they did. They are behind CloudFlare, so you would probably get a CloudFlare notification if they were blocking you.

>>67682
YP won't permanently block you from accessing the site. It'll temporary block you for a while if you are making too many excessive pageloads. That's why everyone is getting 'error downloading'.

File sharing doesn't work!!!!!!!!!!!!

>>17154
Is there a way for it to search folders in the creator folder? I can just sort out different type of images or put images that I don't want in folders then it will auto skip download these images.

I really think error handling should be improved. Right now there are these problems:
- Some files fail to download due to timeout. Must remove ender file and restart to try again, and they may still fail. Eventually they will download though.
- Error messages for 404'd files and timeout cases are same: "Error downloading".
- A download may get stuck for hours.

Things that I think may fix these problems:
- Differentiate between failed download cases. Detect if there is an error or if there is a timeout.
- If it's a timeout - try again repeatedly until some response is received.
- If it's a permanent error - create some empty file and mark these files as absent.
- During future runs, don't attempt to request these files if they are marked as absent. This will save the time because each request can take long to process.
- Report the error code properly in the log.
- Try to detect timeouts for running downloads and restart/resume them when they get stuck.

>>69844
Have one instance of downloading only. Solves 99% of the problems.

>>69844
Correction, downloads should be restarted upon not just detected timeouts but any non-404 errors I guess. Right now I'm getting a lot of 502 in browser.

>>69860
I imagine sending several requests at a time during high load would indeed decrease reliability of each individual response. But it won't solve all the problems. Error detection should still be there.

https://discord.gg/g7V2y9

I'll explore a way to retry error downloads without need to restart CLI. In the meantime you can restart CLI several times to be sure you have downloaded everything. You don't really have to delete the ender files after an error download. Ender will not save them.

Mediocre.txt can prevent a file from downloading next time, be it truly mediocre to you, or a server-side corrupt/unavailable file you want to skip.

The error downloads will be logged to HTML if you don't mind the HTML file to work like log file. "Gallery HTML =" will enable building offline HTMLs.

Regarding download getting stuck for long time you can try cut internet and it'll timeout instantly. This will be useful to prompt a offline message right away, not so much when downloading files though, until I implemented a retry for them anyway.

Updated yiff.bat listening to keypress R and S while CLI is running. Requires system with ANSI escape support like Windows 10, MacOS, idk about Linux.

> Press R to always request retry or always skip error downloading files.
> Press S to skip once.

Any chance for a feature that rips zips, too?

>>70216
Check your blacklist

>>70233
I didn't blacklist anything. Am I retarded and the bat is auto-unzipping? Otherwise, I'm not to sure how to check the blacklist (if there was one made).

>>70235
It's preloaded in yiff.txt

>>70236
Gotcha! Figured it out, thanks. Kudos on the great app; jdownloader started to suck shit.

>>70023
Trying to figure this out and I'm lost. Windows 8.1 here. First, it doesn't seem to react at all when patreon is processed. Second, it reacts when trying fantia, with changing colors and window title. But fantia doesn't seem to work now ("OFFLINE") even with older version, so maybe it got broken.

How do you restart the CLI?

>>69875

I'm getting that, I thought it was vivaldi

>>71038
I was wrong, there are no issues with fantia, just bad configuration in txt.
Still, from what I see the R/A keys work only when there is a prompt for it - e.g. when script reports OFFLINE thing. I don't see any change if I just press the keys during a normal run, so I don't know if those download errors are treated differently from before.

>>72098
It's for the requested data that are required before script can continue.

>>70023
Please give us the choice to download one thing at a time only

Just reminder that yiff.party CAN be unstable sometime and that there's nothing wrong with yiff.bat. Please keep trying or come back later. There are no performance difference between browser and script for as far as I know.

>>72807
>>64525

bump

Here's updated proxy code to support SOCKS5 proxies with authentication (and trailing backslashes no longer break the port number):

if "socks5://" in proxy and proxy[10:]:
~ ~ if not ":" in proxy[10:]:
~ ~ ~ ~ print(" PROXY: Invalid socks5:// address, it must be socks5://X.X.X.X:port OR socks5://user:pass@X.X.X.X:port\n\n TRY AGAIN!")
~ ~ ~ ~ sys.exit()
~ ~ if not "@" in proxy[10:]:
~ ~ ~ ~ address, port = proxy.replace("socks5:","").replace("/","").split(":")
~ ~ ~ ~ socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, address, int(port))
~ ~ else:
~ ~ ~ ~ usr, pw, address, port = proxy.replace("socks5:","").replace("/","").replace("@",":").split(":")
~ ~ ~ ~ socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, address, int(port), username=usr, password=pw)
~ ~ socket.socket = socks.socksocket
~ ~ # The following line prevents DNS leaks. (( src: https://stackoverflow.com/questions/13184205/dns-over-proxy ))
~ ~ socket.getaddrinfo = lambda *args: [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]

Swap out the squiggles for spaces.

>>76494

Thanks! Updated yiff.bat

if exist "Scripts\pip.exe" (echo.) else (color %stopcolor% && echo PIP.exe doesn't seem to exist . . . Please install Python properly^^! I must exit^^! BYE pause>nul && exit)

This line is missing a conditional, making the script terminate instantly. It should be:

if exist "Scripts\pip.exe" (echo.) else (color %stopcolor% && echo PIP.exe doesn't seem to exist . . . Please install Python properly^^! I must exit^^! BYE && pause>nul && exit)

This has been a nuisance for a while... there is something about the "choice" function call in the retry function that makes my scraper hang every time after it's finished downloading everything. Whenever I update yiff.bat I just delete this entire code chunk:

~ ~ ~ ~ else:
~ ~ ~ ~ ~ ~ title("OFFLINE", True)
~ ~ ~ ~ ~ ~ print("OFFLINE (R)etry? (A)lways")
~ ~ ~ ~ ~ ~ if choice("ar", True) == 1:
~ ~ ~ ~ ~ ~ ~ ~ offlinepromptx[0] = True

AND I have to disable ansi_esc, i.e: ansi_esc = False

After that, it works fine... I'm really not sure what the problem is.

>>73384
That won't change anything. It's still hammering yiff.party's server and trying to grab as much as it can. We need a proper 1 by 1 download.

>>77132
it works for me tho. I change 8 to 1 and it download 1 at a time. There's 2 same code, maybe you edit the 1st one. Just search from the bottom

>>76898
Thanks again. Updated yiff.bat with small fix. Also does the weird issue with choice function call have to do with offline message you're getting? There's a focus conflict between two input prompts and you'll need hit a key twice so the right one can receive a key it was expecting. For example on Windows 10/MacOS accept A, R, S, and X keys (ansi_esc = True) and older Windows versions accept only A and R keys (ansi_esc = False), one prompt does A, R and then there are S, X in separate prompt. Sometime when you hit R after there's an offline message and it might not work because the S/X prompt was in focus and was expecting S or X but R wasn't in it, somehow it loses the focus and A/R prompt takes turn. Yes I know it's unreliable but seems to work. Anything that use os.system() on top of another can be problematic, especially os.system("choice ...") THEN os.system("title ..."), ansi_esc was born for that.

Now you can see there is
> print("\033]0;" + echo + "\007\033[A")
when ansi_esc is True and
> os.system("title " + echo)
when ansi_esc is False

And Windows versions less than 10 don't support ANSI escape code. os.system("title ...") completely breaks os.system("choice ...") but os.system("choice ...") on top of another os.system("choice ...") seem to be just focus problem that somehow they can take turn when one key wasn't meeting prompt's expectation, not big deal.

Yes, there definitely does seem to be an input prompt conflict. Apparently if you activate auto-retry mode with X and ALSO hit R to retry whenever a failure occurs, you can get it to work somwhat reliably. With ansi_esc off completely (and without code modifications) it appears to need to download all the files it needs (before scraping creators) 3 times in a row, which I do not understand at all.

Thanks for this. I updated to the new one but now it's not working. I'm trying to run it off a network shared folder as a mapped drive. Did something change in the bat where it checks where it's running from? With realpath it's trying to \\sharedfolder\\folder\\ with os.chdir(batchdir) error line 30.
I edited the batchdir = (os.path.dirname(os.path.realpath(__file__)) + "\\").replace("\\\\", "\\") to abspath and it sees it as mapped drive X: but then something else errors out on line 2085 on cookiejar.py.

>>77930
> batchdir = os.path.dirname(os.path.realpath(__file__))
was changed to
> batchdir = (os.path.dirname(os.path.realpath(__file__)) + "\\").replace("\\\\", "\\")

I added a trailing backslash to treat it as folder, but I guess this is a poor implementation. I discovered it was down to batch side of the code, so I reverted the code for variable "batchdir" in python side and applied to batch side. It's not going to make a difference if you run batch file. Will your network storage be okay with forward slash or does it can't really have any of the trailing slashes?

>>77925
Must be pagination, it's normal. It will happen to when you're scraping from Patreon, Fanbox and yiff.party's Fantia where they are split in pages.

Also A, R and S, X do different things.

S or X will not make that offline message go away, A or R will. The offline message is for the required data yiff.bat can't continue without them, while S and X is for the optional downloads so they can be retried rather than restarting CLI and do all over again just for those. I made S and X available all the time to be concurrent with other task and don't have to hang on us for an input, especially for optional downloads, I wanted a quick toggle for little things.

The S/X input prompt had a R instead of X (R/S input prompt), but that's another os.system() problem, every letters prompted had to be separated so R in R/S turned into a X.

>>78026
It seems os.path.dirname(os.path.realpath(__file__)) also omitted trailing slashes just like the batch side of code does. >>77930 Can you try again? Slight different this time with slash vs backslash.

>>78048

Sadly still not working. Guessing the line 2085 error is because it's not concocting the path and file name together properly?

Traceback (Most recent call last): File "Z:\YIFF\yiff.bat", line 1261, in <module> cookie.save()

File "C\Programs\Python\Python38\lib\http\cookiejar.py", line 2085, in save with open(filename, "w") as f: PermissionError: [Errno 13] Permission denied: 'yiff/cookies.txt'

going backwards through the code I'm assuming it's on line 1198 cookie = cookiejar.MozillaCookieJar(mainfolder + "cookies.txt") referencing mainfolder

mainfolder which is on line 29. mainfolder = batchname + "/" and
batchname which is on line 25 batchname = os.path.splitext(batchfile)[0]

Been fiddling with the main variables with little luck. I should read up on python lol

>>78048
im kind of a noob in python,what's the command for running a batch file?

>>78211
Given the traceback message, something must have been protecting the directory or cookies.txt that yiff.bat couldn't write a new cookie file in there.

>>78259
Unless you're not on Windows, you just have to launch it by double clicking.

>>78289
Thanks for the assist. I reset my entire network and all the security settings on the folder and it's up and running again.

Is there any way I could have it retry failed downloads (i.e. downloads that don't 404 but just weren't able to finish before a server error)?

Also, is there a way I can limit how many concurrent downloads are going on at once?

I was sifting through the downloaded stuff and noticed that when there's an iframe embedded link like a youtube It doesn't pull that data to the gallery other than that everything works lovely I think the only time it's been bratty is when I accidentally 2 of the same id# a few times on the list.

>>78732
I thought it was automatic. I recall seeing it saying that if it failed it will retry.

Whenever I add a new line (12345.name), my usual course of actions is:
a. Run the bat.
b. Check for .part files. Of course there are (still) there.
c. Remove .ender file.
d. Go to a.
The cycle repeats itself until there are no more .part files. Around 10 times lately.

Maybe if .ender file would only be created if there are no .part files (no download errors), it'd be somewhat easier to manage.

>>78732
>>79454
partially downloaded files, 404s, and timed-out downloads are all retried automatically after .ender delete.

>>79666
You devil, no need to delete! Ender isn't going remember the failed downloads so they will be retried next time and the ".part" files are partial files waiting to be resumed.

Relevant >>64468

>>79717
i stand corrected

on the other hand, this tool could really use some cohesive documentation

>>79717
From my experience, failed downloads (.part) remain untouched if I don't remove the .ender file - the log just says it's tripped and there are no info about trying to download other files. I thought maybe because their URLs are not in the beginning of the page, so the .ender is getting tripped earlier than some older unfinished files could be checked.

>>80022
thats what ive been seeing as well, which is why i just delete the ender files almost every time before i run the script.

Is there a script similiar to this which can organise the images from each post into folders. Would it be difficult to make, i'm not very good with code

>>81643
wget or winhttrack

Does anyone knows if this script has any Git project so i can have it always updated? I've been checking the thread, but don't seem to find something like this.

>>81674
as far as i know, this thread is the only source for the script

rKwhTn3[1].png (33.6KiB, 979x512) save_alt

I keep getting an error consistently with this creator, any ideas?

There is a problem in the script that results in file names with spaces at the end which windows doesnt know how to handle .

>>80022
After monitoring some more, I can no longer confirm what I've said. It certainly seems like the script tries to do something with existing .part files even if there is an .ender file. Maybe there was just few edge cases that made me think otherwise.

Anyway, the problem is really on server side. I've tried to download a 450 MB file by other means like browsers, download managers and even vps. All failed. I can get the first 60-90 MB, and no more. The way the server handles transfer is weird and unstable. Resume fails most of the times, so starting the download from 0% is more likely to succeed, but in the end chances to get to even 50% of the file are still around zero. I'm guessing the issue is know to admin though. There are also more reports like this: https://yiff.party/bbs/read/81967

Also, found a bug in recent version - pressing a button in the end closes the window, while it should have restarted the script instead.

>>82043
It also won't start if you have a space in a folder...

errrrrrrrrrrrrror downloading

seriously, how do you even get it to properly download something?

1.png (2.0KiB, 332x61) save_alt

When we get a billion error downloading and have a billion .part files and a file called ender, what are we supposed to do when we get this screen?

I want this to be explained in plain English, step by step

>>83415
>>83440
the site has been experiencing download errors for weeks now, thats not an issue with the script.

that last line is from when the script is finished and asking if you want to restart it.

>>83463
what am I supposed to do with ender and .part files? delete it?

>>83470
.ender files mark the last downloaded file without errors, or maybe before errors. It's used so the script doesn't have to attempt to download everything, and .part times are just files that failed to fully download.

delete them and retry, what's the worst that could happen.

I cant download anything it just keep retrying

>>82041
i also get this issue with https://yiff.party/patreon/4339136

>>82084
Encountered the same issue with Fantia downloads now. Really had to delete .ender to finish the download of .part files.

If you encounter a problem downloading a file then please be patient and try again, there's been issues with server now and then. There's no need to delete .ender files for the .part files, they will be resumed. The only reason to delete .ender file is if you have changed your filter list or you need the file (a success downloaded file) back that you have deleted.

Also on the off chance that the .part file might be one byte too much that can cause them to be a partial file for good (yiff.bat will drop the extension ".part" only if their file size and size reported by server are same), but I haven't encountered one. If you think you got one you should delete it and try again.

>>82041
>>84081
I haven't encountered one, but I know that yiff.bat is sensitive to the glitching HTML, I would also suggest wait and try again later, after the HTML can be rendered properly.

trying to download sakimichan's page always gives an offline error, which leads to endless retries. can anyone else confirm this please?

https://yiff.party/patreon/371321

Running yiff.bat in linux seems to cause error.
python3 -x yiff.bat
and then this error pops up

File "yiff.bat", line 582
if rule := [v for k, v in refererlist.items() if k in url]:
^
SyntaxError: invalid syntax

any way to fix this?

>>85155
Python 3.8 and up. There's Walrus operator (":=") in this.

>>85066
DUDE. There's too much comment in that page! Admin should do something about it because it's wasteful and silly.

>>85183
so its because the page is so big that the script cant parse the contents?

i was able to download https://yiff.party/patreon/950545 with no issues (apart from the download timeouts)

>>85223
Not because yiff.bat can't parse it, it's that page that big... whenever connection drops it'll have to start over. Yiff.party doesn't support resuming on HTMLs. I can see how it can be frustrating when the server or the connection isn't in stable condition.

>>85165
I see, thanks my dude

>>85230
damn.....

guess ill just wait until the servers are running properly again

is this shit dead yet
cant download anything anymore, always retrying

Screenshot (30).png (673.6KiB, 2158x1244) save_alt

links in red arent included in the HTML, is this intended?

link for reference: https://yiff.party/patreon/2584938

>>87686
Good find! It's uncommon asset I haven't got chance to work on. It's not part of the body like the attachment files did not. I'll merge it into the body.

Updated yiff.bat, please delete HTML file and shrine.json to rebuild HTML.

I don't know what this will look on actual paysite (Patreon) though, so it cannot be helped. But I will try if creator is also: gay furry artist, does embed url, I'd might pledge :P

Updated yiff.bat again, this time for embed urls on Patreon

>>87754
thanks for the fix, works great

Is there a way to figure out which yiff can't be parsed correctly or make it so we can set a max time for it to parse before it skips to the next id? I'd run the bat and then it'll hit an ID that just stops. I comment out that one and it does it again at another and it's not like it's going down the list, it would go backwards sometimes. The only thing I can think of to split Yiff into smaller directories and run them in batches but that's kind of backwards if I'm trying to keep a single list.

>>89124
It might be hard to place some clue to what creator it's "stuck" on right now while there is parallel downloading going on. There can be stuck connection between yiff.party and computer sometime with seem no timeout (sometime I really have to wait for a while for it to time out), what I found worked (and faster to it) is to disconnect internet from my computer and then put it back on, yiff.bat will immediately throw an offline error, I just tell it to retry. I haven't actually supply timeout amount to there yet, it'll decide when to stop depending on condition of the connection, there's no exact timeout.

Deeper down in creators list can get finished earlier in parallel threads, it would appear to go "backwards sometimes" in the creators list like you're seeing.

>>89141
Ok Found my mistake. It was hanging because it got stuck on a YIFF that was deleted. I went through my entire YIFF list commenting out and deleting the ones that aren't available anymore and it's running great staying blue again.

>>89158
Oh, right, I should update yiff.bat to skip the deleted creators!

Is there a way to reuse python seeing as it keeps on restart the .cli?

Updated yiff.bat support importing from Kemono.party for Fanbox artists, new setting "Kemono.party =" added, HTML server is also improved, small update to the first time setup CLI message displaying a URL to download Python and another direction where to get a 64-bit version.

I can't update yiff.bat to skip yiff.party's "excluded" creators because:
1. Yiff.party actually assigned them as 404.
2. Urllib module the yiff.bat is using to fetch pages/download things will see 404'd page "not found" and 404'd page "creator excluded" same thing.
3. Final say, I'm not thotslobberer so...

>>90392
Are you saying it couldn't find Python?

File