Welcome to Inkbunny...
Allowed ratings
To view member-only content, create an account. ( Hide )
smaxattax

JAID (Just Another Inkbunny Downloader) v0.21r4

Download: http://www.mediafire.com/file/cxvy4xk13u2eln6/JAID_0.2...

October 11 2017: SID length changed to reflect site changes. Thank you
kavc
kavc
and
Foop
Foop
for the heads up.

This tool is launched with the command line and in its current state is much more suitable for power users. Its purpose is to not only download submissions, but as much information about submissions as possible through the site's API. If you find this tool hard to configure/use or straight-up cryptic, I highly recommend looking into IB Downloader by BunnyFoxglove .

--

Well damn, this thing is over 3 years old now! Inkbunny release 74 caused a minor hiccup by changing the timestamp format a bit, prompting a quick hack-fix so JAID wouldn't think every submission had been updated. I also threw in a barebones backup feature (enabled by default - read over the config) since I've have had the 'pleasure' of suffering through code flubs, drive failure, file/filesystem corruption and all that other good stuff over the years with only stale backups at hand (For backups on Linux, piping TAR into SREP is amazingly efficient for redundant data).

Nothing much has changed over the years because, well, it's done what I've needed/wanted it to do and only required a couple of minor fixes to keep working (SID format, this new timestamp format). I was quite surprised to see that, across its versions, JAID has been downloaded over 250 times! Makes me feel shitty about neglecting it so much. Poking through the script for the first time in a good while... boy is it ugly! It's mostly still what I threw together in a sleep-deprived afternoon after ArtHawk by Whippy was discontinued for technical reasons, but it seems to have lasted.

This won't be the case for much longer though. Inkbunny release 74 marks an important change that fundamentally affects the operation of this tool and others like it: usernames can be changed at will. This was always the case with the aid of a support ticket, but as a rare occurance it was easy to manually account for. I've personally seen 32 changes out of the 1,441 people I watch in the past 24 hours which is over 2%. What does this mean? Well, it's annoying. JAID can't download anything new if you specify an invalid username (it will try to help you out by throwing a notice in the window+log), but if you do update the username then a new folder will be created for that username. It won't re-download old stuff, but there will be no apparent link between the two folders unless you manually merge them. Thankfully this can be rectified by poking those small subdetails files a tiny bit :)

So now I've got a reason to do what I've always wanted to, more-or-less rewrite the whole thing. I was also unaware at the time I wrote the script that user ids could be utilized. Go me! I'll have to ponder file organization a fair bit, write a script to update old versions and move stuff around, and generally make things suck less, but it should be fun.

---v0.21r3---
Workaround for a rare bug affecting submissions ending in "\". Curse you, PHP!
Quick hack-fix/workaround to cope with timezones in timestamps, introduced in IB release 74. More eloquent solution to come.
Barebones db backup feature. Mostly as insurance against bugs with newer versions, but can save yer butt if you do full backups infrequently.
Update included certificate bundle (old one hadn't worked since the site changed SSL settings - update your Aria2 as well if needed!)
--
Readme:
" What it does:
* Downloads/synchronizes all submissions from a list of users
* Downloads submissions from command-line specified users (usage: php JAID.php artists=artist1,artist2)
* Runs on multiple platforms (requires PHP and Aria2)
* Uses HTTP keep-alive to grab multiple pages from a single connection.
* Checks for and downloads new submissions on subsequent runs
* Checks for updated submissions (slower - can be turned on and off)
* Can keep old files from updated submissions
* Downloads and stores submission details (tags, etc) in CSV format
* Some sanity checking: Won't run multiple instances simultaneously, can handle almost any forced/abnormal termination
* Verification of downloaded files with MD5 checksums
* Export images and tags to Shimmie-importable CSV file

* Barebones db backup. Creates a copy of the main db (details.csv) after each run. Enabled by default mostly because I cannot guarantee the stability of versions going forward, so you'll have something to fallback on should something go awry. 0.21 is solid enough to have lasted me years. Setting the target directory to an alternate physical drive can give you some degree of fault tolerance, but don't mistake this as a replacement for manual backups.

What it should do in the future:
* !!Internally keep track of user ids and resolve them from usernames instead of relying on (now) easily-changed usernames
* User-defined organization of files based on submission details
* Have more downloader modes (e.g. "I just want the files!")
* Have resume capability
* Have some semblance of a proper script (Convert individual steps into functions, allow extensions)
* Faster/Automatic updated submission checking (API is in place, just need to write it)

Known issues:
* If Aria2 halts (very rare / version-specific bugginess) the script has to be manually restarted.
* Be aware of filesystem limitations. Using FAT32 limits the subinfo directory to ~33,000 files (0x80070052 - can't make new files).

/\/\How do I Install/use this?/\/\

!*_ENABLE API ACCESS IN YOUR INKBUNNY ACCOUNT SETTINGS_*!

*****Windows*****
Grab and install PHP 5.3 from php.net (thread safe is fine)
Ensure the following settings are set in php.ini:
allow_url_fopen = On
extension=ext\php_openssl.dll <-- note the lack of ; at the beginning. These extensions are enabled.
extension=ext\php_curl.dll    <--
Grab Aria2 from aria2.sourceforge.net (i686) and extract the EXE to the same folder as JAID.
Open config.php in a text editor and set your login information.
Type or paste usernames into userlist.txt.
Run JAID.php from a command prompt like: php JAID.php

*****Other Platforms*****
Build (with OpenSSL support) or apt-get install: php5
Build or apt-get install: aria2 php5-curl
Ensure the following settings are set in php.ini:
allow_url_fopen = On
Open config.php in a text editor and set your login information.
Type or paste usernames into userlist.txt.
Run JAID.php from a terminal like: php JAID.php

In both cases, ca-certificates.crt from Mozilla is included for convenience (aria2 doesn't know what certificate authorities to trust.)
Viewed: 371 times
Added: 2 years, 6 months ago
 
CosmicMutt
2 years, 6 months ago
good thing im too dumb to read
GreenReaper
2 years, 6 months ago
Thanks for the update - and looking forward to more to come! I've refreshed our wiki's link to JAID.
GrandApollo
2 years, 2 months ago
Oh hey, I'd realized the old one was broken and was wondering if it was ever updated. Glad that it's still being taken care of in some regard, even if it might be minor! Thanks for being an awesome person!
GrandApollo
2 years, 2 months ago
And I added that comment exactly ten minutes before I wanted to edit it, so I guess I have to make a new one. Was just wondering if you're really planning on updating it fully? I don't object either way, honestly, because even with people changing names, I'm only watching a bit over a hundred people. I can handle the name changes, and everything else works just fine. But I'm definitely curious, and your little project here has been a staple of my Inkbunny experience for quite some time!
smaxattax
2 years, 2 months ago
Glad to hear you've found it handy :)

I do indeed, just non-IB things have come up. In addition to name handling I've wanted to experiment with things like SQLite to see how much of a performance increase that can yield along with some other minor improvements.

Have you tried PHP 7.0.x yet? I've been using JAID with it and have found it to be ~2-3x faster than PHP 5, which I find to be quite amazing!
GrandApollo
2 years, 2 months ago
Haven't tried it. Guess I'll upgrade soon!
Foop
1 year, 11 months ago
Oh hey, you say you're using PHP 7. What dark magics do you use to have it running? :)
Foop
1 year, 11 months ago
Nuu, it stopped a-workin'! I upgraded(...) my Ubuntu to 16.04, and had to reinstall php and aria2. Now I have PHP7, and running 'php JAID.php' dies immediately with error code 255. Sob, sniff! Will there be a new JAID, or should I grab an older PHP?
Foop
1 year, 11 months ago
Okay, my bad - I only installed PHP, not 'php-curl'. With that package reinstalled, JAID seems to zip-zoom along just fine again!
New Comment:
Move reply box to top
Log in or create an account to comment.