Welcome to Inkbunny...
Allowed ratings
To view member-only content, create an account. ( Hide )
Whippy

ArtHawk - slim and pretty

by
Whew...been coding all night. I cut out pieces of code from various ArtHawk versions and pieced it all together in a nice little bundle. My goal to to simplify the thing before I went insane. And I did!! I trimmed it down from 1,836 lines of code to 548 lines!!! Squeeee! All I really need to add right now is the code for checking InkBunny and DeviantArt.  I know those pieces are already working so it should be easy.
^__^

So far, changes will include the following:
*removal of renaming, because it was a HUUUUGE pain in the ass to maintain and do correctly

*Removed the choice of the folder saving pattern, it now manually assigns and creates "InkBunny" or whatever site as the root folder for your artists of a particular site. I couldn't seriously see people being bothered by this one XD

*Added a big ol ListView to show filenames, sizes, KB downloaded, percentage, etc so you can see what the heck is going on.

*Double clicking a filename on the ListView will open that file

*Simplified the interface (GUI) A LOT!

*Re-added the text box for editing of the artists list in place of the listview

*Planning to add something to append the watchlist of any user for IB,FA, and DA.

Here's a comparison shot:
http://i.imgur.com/cCDVX.jpg
Viewed: 157 times
Added: 6 years, 6 months ago
 
ForeRest
6 years, 6 months ago
Nice! can I have a copy? pleaseeee!? :3
Whippy
6 years, 6 months ago
^_^ all in good time! Hehe!! It only works with FA right now. Poke me on skype and I'll send you an exe of it.
Whippy
6 years, 6 months ago
ok ok...here's a lil version I call FA_slim
http://dl.dropbox.com/u/43927622/apps/ArtHawk/AH_FA_SL...
KaitoFenikkusu
6 years, 6 months ago
Soooo ... when do we get to play with this new version???
Whippy
6 years, 6 months ago
Soon ^.^ It shouldn't be long now. I'm hoping to have it complete in a day or so.
KaitoFenikkusu
6 years, 6 months ago
Actually, I've been meaning to ask ... do you have instructions on ArtHawk? cos I never seem to get it to ... er ... work
Whippy
6 years, 6 months ago
1. Fill in your username and pass for the site selected in the dropdown menu.
2. Browse to the site and copy your watch list.Paste it into notepad or whatever text editor you use.  Make sure there's no spaces or tabs. It needs to be a simple list like this
artist1
artist2
artist3
artist4
3. Push the edit artist list button. Paste the list you have into the box for the site. Push 'save'.
4.Set your base path and choose and save pattern.
5. Hit the "Check Site" button and the downloading will start.

Your question (and a few others asking about how to use arthawk) are part of the reason I wanted to simplify it. With a little luck, I'm hoping to make it this simple:
1. Hit "set save", pick your path
2. Fill in your user name and pass for each site.
3. Hit the "get watch list" button, enter your name or another and hit "append list" to add those names to your artist list.
4. Hit "download site"
KaitoFenikkusu
6 years, 6 months ago
Okay ... so I'm guessing arthawk doesn't enjoy artist names with weird characters then ... because it logs into FA ... and then says it's done ...
But the first artist on my list is --- ...
and yes, I checked my password in the lil .ini file just to make sure it went in properly

*rechecks everything and removes the odd 3 special character names* ...
hmmm, same result ... it logs into FA then says 'download complete' ...
Whippy
6 years, 6 months ago
The version I have up right now doesn't work with FA at all. They changed their code.
in the meantime, try this: http://dl.dropbox.com/u/43927622/apps/ArtHawk/AH_FA_SL...
It downloads one at a time and is a bit simpler to use.
KaitoFenikkusu
6 years, 6 months ago
ahhh that explains it, oh and that works great btw ...

so ... to be an ass because I'm not just on FA, IB and DA ... what about FurPaws and SoFurry?
Whippy
6 years, 6 months ago
lol
i've never heard of FurPaws. SoFurry is a bit complex looking and I don't use it (yet anyway)
KaitoFenikkusu
6 years, 6 months ago
Well, since I already dumped an error in IB's coding to the SoFurry username system on em ... *coughs*

FurPaws is new - Shuik started using it and I followed
Whippy
6 years, 6 months ago
Uhmm...dumped an error?

*checks FurPaws AUP* Okay, looks like I'm sticking with InkBunny. FurPaws may as well be another FA:
" C3: [Artistic representation] Artistic representation is assumed and encouraged in works submitted to Furpaws.Net, however any works depicting a minor or representation there of, engaged in any sexual acts or mature situations is not tolerated, and a violation of user rights as per this agreement. This violation MAY result in immediate revocation of account access and termination of service to the user.
KaitoFenikkusu
6 years, 6 months ago
Oh yeah, see you can link to your SoFurry account etc using the shortcuts on IB - however the glitch I found was that IB's linking system doesnt allow spaces in user names, where as SoFurry allows spaces and pritty much any special character in their user names - i.e. on SoFurry my user/log in name is Kyte Pheonix not kytepheonix as it would be most anywhere else

Oh and yeah, FurPaws does have that minor limitation in there, kinda forgot about that ...
KaitoFenikkusu
6 years, 6 months ago
Oh hey, another thought ... what about good old VCL?
bencoon
6 years, 3 months ago
Questions regarding this: Does it work with the new thumbnail layout, what does it download, and what's the delay? If it connects like a normal browsing session, that's perfect. If it connects like it's been doing with IB, well.. slightly less perfect. <:)
Whippy
6 years, 3 months ago
It doesn't work with the html of the site much at all. The only time it actually fetches html is when you add your watchers to the user list. Other than that, what it does is use InkBunny's API in order to find links to full size images. Unfortunately, it has to parse through a bunch of data it doesn't need, so I've made it request all the pages in one artist's gallery, then parse it for links, then queue those to download. The number of simultanious downloads is adjustable, but I plan to make the connections adjustable too. As I told Starling below, it's not supposed to be making hundreds of requests to a site at once, that's a bug for sure.
bencoon
6 years, 3 months ago
Sorry, I was actually referring to the FA slim downloader, since there's really no documentation in it. :o
Whippy
6 years, 3 months ago
Oh, okay. Well..with FA, it used to be kind of easy to get links to submissions because the full link was actually part of the thumbnail URL. But they changed their code, so now I have to download every single submission's HTML page in order to parse it for links. Before, all I had to download was the HTML for the gallery page. Sooo...that little code change they made multiples the number of connections to an insane level (ie. 1,000 submissions = 1,000 HTML pages to download = 1,000 submission links). I would be happy to work with them if they would make this process easier, but I dunno if they're up for that....
bencoon
6 years, 3 months ago
Yak HATES downloaders with a passion, to an unreasonable level. He never really articulated why, but I suspect it's something like the standard BOFH mantra, "Fuck the users". I suspect he's never likely to be interested in working with anyone on, well, anything.

As far as the downloader, though, something that loads the pages one at a time with a randomized pause would probably be perfect. It'd be slow as balls, but the traffic would be pretty easily lost in the background.
Whippy
6 years, 3 months ago
He hates downloaders huh? :P Well...you know what I hate more? Wasting time right clicking and hitting "save" for every file in a user's gallery lol. It's great they have a site up to show all this great stuff, but it's so much better being able to keep it on your own hard drive. Not having to worry about their site going down, an internet outage, an emo artist deleting their gallery, and so on. If he's really like you say he is, it sounds like he's not even worth trying to reason with.

Load the pages one at a time with a randomized pause? Uhmm..eeek! While that is a good idea to avoid detection, I don't know if I could stand doing that. I built ArtHawk to be able to QUICKLY download whole galleries. How big of a pause are you talking about? I could randomize the page order it tries to download.
bencoon
6 years, 3 months ago
I'm talking 1 or two seconds, really, not anything *huge.*  Have it be something you can leave running while you have your dinner or something. Speed is handy, but you still want to be gentle on the server.
Whippy
6 years, 3 months ago
*giggles* Okay, that's reasonable. Yeah, I can do that ^^
DaddyDuckyBE
6 years, 6 months ago
What next buckling? You gonna add an automatic cleaning feature, even? ;)
MichaelJBear
6 years, 6 months ago
*Really really big smile.... from the left ear to the right one*
ButtercupSaiyan
6 years, 6 months ago
This is interesting... what is this?
Whippy
6 years, 6 months ago
A mass-downloader for InkBunny, FurAffinity, and DeviantArt. You fill in your username and pass for each site except DA, then copy the usernames of your watch list into the tall edit box at the bottom left. Set your save directory, and hit the download button ^_^
I'm striving to make it as simple as possible. Oh, it also downloads files 5 at a time.
MichaelJBear
6 years, 6 months ago
Eeeehh Héhéhé Yeah... it's like a FA SLIM version but it's for DEVIANT, FA and InkBunny... look really darn good Looq...and I can say (because I know it personnaly) that your FA slim version and your big version Downloader (FA, Deviant, InkBunny) works like a charms *thumbs up*
starling
6 years, 3 months ago
There seems to be an issue with the way it connects to sites. I see it opening hundreds of simultaneous connections to IB but we block above about 20+ connections. The semi-official standard for HTTP is no more than 8 simultaneous connections. The connection rate may also be an issue as many sites will limit this.

It's best to write scripts that have deliberate delays between each request and wait for each request to finish before moving on to the next (even if you have 8 request threads they should each wait for the connection or request to finish before moving on to the next).

Bandwidth and request speed limits might be a good thing to let users configure as they may need to set them quite low to avoid getting blocked by sites that are grumpy about scripts. FA has nearly zero tolerance for scripts. We'd rather support scripts and let script writers know how best to connect without causing resource issues on our side! :P

My suggestion for now would be setting limits for connecting to IB so there are no more than 8 simultaneous connections at any one time and a maximum of about 20 requests per second. That should still be fast enough so that users can get the files with no noticeable difference in speed (but way less pain for our poor firewall and server).
Whippy
6 years, 3 months ago
Hmm...hundreds!? 9__9 That is definately a bug, it's not supposed to do that many at once. There's two parts to the process. The first process parses the XML for full image links, while the second part is the actually downloader. Right now I have the "page fetcher" set to make a request for each page of results from the API. That's likely where things are going haywire, because I know that even artists with a ton of submissions (arineu for example) return something like 30 pages of results.  

Right now, I have the link for the OLD version of ArtHawk up on my page here, but I'll remove it and it will be replaced with a newer one. I'll see about making the page requests user configurable, that's a very good idea. So something like a range from 1-20 is a good idea then?  

The code would be for a maximum number of connections, like..
if (connections < 20 AND page !> last_page)
request(page)
starling
6 years, 3 months ago
Yeah I seem to remember the majority of the connection floods were "get full details about these 20 submissions" and there'd be like 100+ of those open from one IP at a time. :O So it must be that step which tries to open them all up. It could be to do with how the connection is opened and closed. You know you can do all that via just one connection? It might be obvious or automatic already but you just open one HTTP connection and then you can make as many requests as you like via that one, one after the other. Don't open/close once per request!
Whippy
6 years, 3 months ago
Hmmm...really? When you say 'requests' are you actually talking about the submission ID's that it sends through the API call? When I get the RID results I set it to submissions_per_page=30 then I do the same when fetching submission IDs.  Would it be better to just get all the submission IDs from one request, then put all those IDs into the request for the submission details?

I don't really have a way of keeping a connection 'open' or 'closed' since I'm using an outside function to make these requests for me.
starling
6 years, 3 months ago
Oh the other limitation I should mention is that you can only ever get one request back from IB at a time anyway. Because every request involves a session, IB will lock down all other requests with the same session ID until the first request has finished being served. That could actually be where the bottleneck is coming from. If you ask it for even 8 (or 30 or 100) pages of full submission dtails, it'll open up all those requests but they will all sit there hanging open as it answers them one by one. If it takes 1 second per request then the others will be open for quite a while. 100 requests at once would therefore keep the hundredth connection open for 100 seconds or more. :O
Whippy
6 years, 3 months ago
Ouch 0_0
Ahh...well since I'm only using a function to make these requests, there isn't really a way to tell it to make multiple requests on the came connection. I could ask the creator of the function if it's possible though. The only way I see is to make something like submissions_per_page=(however many submissions an artists has). Otherwise, I'm forced to make another request for another set of IDs and would have to wait for each.
PrinceLeon
6 years, 3 months ago
do you have one for inkbunny
PrinceLeon
6 years, 3 months ago
i cant get the ah slim to work its says my password failed but i know its right
Whippy
6 years, 3 months ago
Sorry, that link you downloaded from here or my user page is an older version. I'm working on a newer version and working out some bugs atm though.
PrinceLeon
6 years, 3 months ago
ok thanks looq
New Comment:
Move reply box to top
Log in or create an account to comment.