Welcome to Inkbunny...
Allowed ratings
To view member-only content, create an account. ( Hide )
smaxattax

JAID (Just Another Inkbunny Downloader) v0.2

Download: Removed (Get v0.21r3 instead)

Big thanks to
starling
starling
for getting back to me on my API suggestions. In a recent announcement it was mentioned that API suggestions could be submitted in a ticket. I didn't expect to be taken quite so seriously though.

Issues discovered with 0.2 (Rare <0.1%):
* Submissions whose files contain special characters may fail to download and/or validate. Usually this is because of filesystem-imposed limitations, but a comma will cause validation to fail due to an oversight. The URL is stored unecoded in a comma-delimited field.
* Submissions with ridiculously long filenames may fail due to filesystem limitations.

Most significant changes since v0.1:
* Use of cURL. Faster, friendlier, and much more error-resistant on the network side of things.
* Command line parameters! Well, two. Most important is: artists=artist1,artist2 to choose specific users instead of loading userlist.txt
* JAID2Shim.php - Import downloaded submissions to a Shimmie 2 imageboard! This will let you browse with thumbnails, search by tags, etc. Much more organized than folders of images.
* Validator.php - Validates downloaded submissions. No longer do you have to take it on faith that the submissions were downloaded successfully.

Next release will focus on cleaning everything up, separating the processes into individual functions, and merging the functionality into a single script. This will make it much easier to maintain and build upon (it's an absolute disaster at the moment).

Readme:
" JAID (Just Another Inkbunny Downloader) v0.2
By smaxattax: https://inkbunny.net/smaxattax

What it does:
* Downloads all submissions from a list of users
* Downloads submissions from command-line specified users (usage: php JAID.php artists=artist1,artist2)
* Runs on multiple platforms (requires PHP and Aria2)
* Uses HTTP Keep-alive to grab multiple pages from a single connection.
* Checks for and downloads new submissions on subsequent runs
* Checks for updated submissions (slower - can be turned on and off)
* Can keep old files from updated submissions
* Downloads and stores submission details (tags, etc) in CSV format
* Some sanity checking: Won't run multiple instances simultaneously, can handle almost any forced/abnormal termination
* Verification of downloaded files with MD5 checksums.
* Export images and tags to Shimmie-importable CSV file

What it should do in the future:
* User-defined organization of files based on submission details
* Have more downloader modes (e.g. "I just want the files!")
* Have resume capability
* Have some semblance of a proper script (Convert individual steps into functions, allow extensions)
* Faster/Automatic updated submission checking (Waiting for Inkbunny to add sort by update_datetime to API)

What it could do in the future:
* Support other sites?
* ???

Known issues:
* If Aria2 halts (very rare) the script has to be manually restarted.
* Be aware of filesystem limitations. Using FAT32 limits the subinfo directory to ~33,000 files (0x80070052 - can't make new files).

/\/\How do I Install/use this?/\/\

!*_ENABLE API ACCESS IN YOUR INKBUNNY ACCOUNT SETTINGS_*!

*****Windows*****
Grab and install PHP 5.3 from php.net (thread safe is fine)
Ensure the following settings are set in php.ini:
allow_url_fopen = On
extension=ext\php_openssl.dll <-- note the lack of ; at the beginning. These extentions are enabled.
extension=ext\php_curl.dll    <--
Grab Aria2 from aria2.sourceforge.net (i686) and extract the EXE to the same folder as JAID.
Open config.php in a text editor and set your login information.
Type or paste usernames into userlist.txt.
Run JAID.php from a command prompt like: php JAID.php

*****Other Platforms*****
Build (with OpenSSL support) or apt-get install: php5
Build or apt-get install: aria2 php5-curl
Ensure the following settings are set in php.ini:
allow_url_fopen = On
Open config.php in a text editor and set your login information.
Type or paste usernames into userlist.txt.
Run JAID.php from a terminal like: php JAID.php

In both cases, ca-certificates.crt from the Ubuntu package ca-certificates is included for convenience (aria2 doesn't know what certificate authorities to trust.)

***Changelog***

---v0.2---
* CLI Parameters!
* Download/update specified artists via command line (usage: php JAID.php artists=artist1,artist2)
* Update stored submission info via command line (usage: php JAID.php mode=subinfo)
***This should only be used if you have a reason to do so. You should run Redetails.php then Validator.php afterward.
* The userlist is now sorted and filtered of duplicates
* Now uses cURL for page grabbing (persistent connections like Inkbunny likes, faster page loading, better error detection/handling)
* Aria2 now grabs 50 submission details at a time. They are later split
* Submission details now include the submission's description
* When downloading an updated submission, the old files can be kept
* JAID2Shim.php - Exports certain (configurable by type+rating) submissions to CSV for Shimmie bulk_add_csv
* Redetails.php - Rebuild details.csv from individual submission info (in case of deletion or corruption)
* Validator.php - Validate new and/or existing downloaded images with stored MD5 checksums

---v0.1---
* First public release
Viewed: 66 times
Added: 6 years ago
 
Shokuji
6 years ago
Just Another Inkbunny Downloader? I didn't know there was already one. =p
I may have to give this a try sometime. Though what I think would be even more useful (and less costly on bandwidth) is being able to batch download your favorites.
smaxattax
6 years ago
This would be the third I know of, the others being ArtHawk by Looq and IB Downloader by BunnyFoxglove.

Grabbing yours or anyone's favorites can be done in the next release, if you're interested :)
Gimlao
5 years, 11 months ago
Huston, we have a problem !

" cmd.exe wrote:
php JAID.php artists=Delicious
User(s) specificed manually via argument: Delicious
Making artist folders...
Attempting to log you in...
Got SID: fd72ba4afba6addd1274ef4798c5d189
Checking user #1/1: Delicious
3 new submissions found!
Downloading submission details...
Downloading submission files...
Exception caught
Exception: [AbstractOptionHandler.cc:73] errorCode=28 We encountered a problem while processing the option '--checksum'.
  -> [OptionHandlerImpl.cc:371] errorCode=1 Unrecognized checksum

All should be well! Logging out and closing down.


Could you fix that ?... =/
smaxattax
5 years, 11 months ago
Oh hey, actual feedback from a user! Thank you :)

This problem is mostly not my fault. The submission causing the problem is this:
https://inkbunny.net/submissionview.php?id=230890&p...

The second image is broken. If you open sub_230890 you will see the following:
" <full_file_md5><![CDATA[]]></full_file_md5>


The MD5 is blank, so Aria2 is confused.

You can sort-of fix this by changing line 503 of JAID.php from:
" fwrite($todlf, "    checksum=md5=" . $currentmd5 . chr(13) . chr(10));


To:
" if (strlen($currentmd5) == 32) { fwrite($todlf, "    checksum=md5=" . $currentmd5 . chr(13) . chr(10)); }


This will stop the broken image from stopping the download process, but submission 230890 will still fail validation because of the blank checksum.

Many thanks for the bug report. I was unaware that a minor IB error like that could be so damning. I've been using an old version of Aria2. Instead of throwing up an error, it ignores the blank checksum.
smaxattax
5 years, 11 months ago
The suggested fix/workaround is now in v0.21, along with some other minor changes. The submission will still fail validation, which it should as there's clearly an error, but if it bothers you then you can manually "validate" the entry by finding its line in subdetails.csv (before running the validator) and changing "n,n" at the end to "y,n".

Thanks again for the report!
New Comment:
Move reply box to top
Log in or create an account to comment.