Welcome to Inkbunny...
Allowed ratings
To view member-only content, create an account. ( Hide )

Americas served 'til 2019 by dedicated cache

We gladly received support from French and Quebec hosting provider OVH earlier this year, helping us to improve service in the Americas. But good things can't last forever… and once our funding expired, we found ourselves hunting for a Cyber Monday deal to replace it, within the budget set by our members.

Fortunately, we found the perfect system, which we've leased for the next two years:

HP DL120 G6 - Dual-Core Pentium G6950 2.8Ghz - 16GB DDR3-1333 Multi-bit ECC RAM
4x1TB SATA HDD (3TB RAID5) - 10TB/month transfer @100Mbps Full Duplex - $39.78/month

So far we're very happy with it, and we hope you are too!
Our more technically-minded members may have a few questions…

A six-year-old Pentium, really? Closer to seven! And feature-wise, it's a decade old. This is a cache. We need disks, RAM and network bandwidth, not CPU power, and we don't want to waste your money.

The server's already handled 3x traffic bursts while serving bulk downloads - close to its 100Mbps limit - and it's pushed out almost 300GB in one day, so it should be fine.

To ensure good performance, we maxed out the RAM, reconfigured the Linux kernel to strip out features we didn't need, and recompiled it for this specific CPU. It seldom ventures above 15% usage.

As a bonus, rolling out similar changes out to our SSD-based caches in Virginia, Tokyo, Sydney and Arizona freed up ~5% of their RAM. Most of these only have 1GB to play with, so every little helps.

Why so much storage? Our members' content is growing, especially after the image and thumbnail upgrades earlier this year. Most caches only hold part of our library; but our goal is to be able to serve every public file from North America, without having to go to Europe.

Upgrading to 1TB disks also gets us larger internal buffers, higher data density, and lower latency. Leaving a little spare capacity means most data will be at the start of the disk, on the fast outer edges.

Why not SSDs? We'll be storing a copy of most public content in Inkbunny's 1.5TB (and growing) library - almost 8 million files, including thumbnails. It's already soaked up 570GB, and that'll double by March. SSDs this size are not yet cost-effective to lease; spinning rust is still the way to go 'til 2019…

[Most content will be stored and read sequentially, so random access speed is less important here. Some smaller caches peak at over a hundred requests a second, but this one averages less than ten.]

10TB/month is a lot of data… It was the minimum we could buy for such a server. While it's ~160% of current traffic there, we need to handle growth over two years, plus visitors currently served via Chicago.

Inkbunny preloads content you may need - such as the next image in a pool, or the next search page. This improves user experience over slower connections, at the cost of extra bandwidth on our end.
We're fine with this - even if we'd had to pay more, the cost is low - we just need to account for it.

What about Europe? The rest of the world? Our secondary server is more than capable of handling it! Traffic has grown as members enabled wide mode/huge thumbs, but it's still well within capacity.

Why not stick with OVH? We closely considered OVH's range of offerings; but for this box, at this time, it made sense to go with our existing main provider, LeaseWeb.

Their Black Friday offers had limited availability in Quebec. Their closest SoYouStart server had only two drives - so we'd only get 2TB after mirroring them - while their Kimsufi models lack redundancy entirely. Other caches rely on this server; we want it to stay up up even if a disk drive fails.

LeaseWeb hosts our servers in the Netherlands; their Virginia datacenter is 10ms closer than Quebec. Our cache's upstream connection is also their connection home - so they'll fix any problems fast.
Lastly, donors can fund our services via them, and know their money is going directly to our hosting.

While we decided not to go with OVH this time, we may use them in the future - particularly where LeaseWeb lacks VPS coverage, as in some places we do not need a dedicated server.

So you don't need any more money, right? We do have ongoing expenses totalling over $4000/year, and you're welcome to help with that. [Many have done so for several years, and we thank them!]

Our next major planned expense is the renewal or replacement of our European servers, in late 2017. That'll cost a bit… but for right now you might prefer to commission our hardworking members. After all, they have their own expenses to cover this season!

P.S.: If you haven't seen the Inktober 2016 roundup yet, be sure to check it out!
Viewed: 1,417 times
Added: 7 months ago
Site News Item: yes
7 months ago
A thought, and maybe a curiosity:

I know that PNG images were being reprocessed to reduce their impact on storage footprint and bandwidth use after upload (the reprocessing actually broke an APNG I uploaded).

However, are JPEGs receiving the same kind of treatment?
If so, has it been of any help in curbing any storage or bandwidth needs?

Just really curious about that. :)
7 months ago
Some JPG files are limited to a certain quality level. But it's a relatively high one,  as we're sensitive about not degrading the images we're hosting. That's kind of the point of an art site, after all! ;-)

There are potential opportunities for improvement in terms of the processing libraries, and it's on our list of possible improvements, but we want to be sure is supported as far as possible in our OS, rather than having another piece of code we have to maintain directly. We do compile ImageMagick from source to reduce the potential attack surface, but the underlying JPEG libraries are maintained by Debian.

As for Animated PNG, we're considering support for it in the future, particularly as Chrome is planning to support it natively. Right now it does not survive processing by our optimization and scaling software, so we'd have to patch them or split APNGs into separate PNG files for reprocessing, then recombine.
7 months ago
" GreenReaper wrote:
JPG files are limited to a certain quality level. But it's a relatively high one,  as we're sensitive about not degrading the images we're hosting. That's kind of the point of an art site, after all! ;-)

True! I was thinking more along the lines of jpegoptim for stripping EXIF data and probably 95% quality level, but I see what you mean on minimizing attack surface, for sure.

Also, Chrome's devs are finally pulling their heads outta the chocolate starfishes and considering APNG? About time. :)
7 months ago
We do strip out extraneous chunks using jpegoptim - both for the size savings, which can be significant, and because they can cause images to display incorrectly on some browsers.

I don't think we apply a quality alteration to original files, but "medium" screen-size images are restricted to 95 quality, while thumbnails and other reduced-size copies are at 85.

I thought you might have been talking of improving the compression ratio of JPEG files. See discussion of this by libjpeg-turbo's developer, noting the trade-offs - it also converts to progressive JPEG, which we've avoided due to the way such images are perceived by users.

It remains to be seen whether APNG will actually be merged. Apparently the developer is leaving the Chrome team soon. But it's something to hope for.
7 months ago
" GreenReaper wrote:
As for Animated PNG, we're considering support for it in the future, particularly as Chrome is planning to support it natively.
Firefox has supported it natively for years so I don't see any reasons why you shouldn't add support for it when your processing and scaling software can handle APNG files.
7 months ago
That software can't handle them, though - that's the point. If they did have support, it'd work. Requests have come up regularly with ImageMagick, but it hasn't happened yet - so if we implement support, we have to balance that against other things we want to do.

[Or perhaps we can convince them to reconsider, since that Webkit has implemented it in a way which does not require PNG library changes, and they are under a compatible license.]
7 months ago
You guys do so much work, making sure this site stays in great working order. I don't know how much of this stuff people read, or understand sometimes, but there's a lot of planning that goes into all this. Not to mention all the work configuring it once the decisions are made.

So thank you all for your dedicated hard work. <3
7 months ago
A curious question in the interest of optimization:  Is it possible to disable the pre-loading of images on the user end?  I'm not sure to what extent this pre-loading goes, but I very rarely browse by pools, and I hate to think I'm munching on bandwidth and resources when I don't have to be!

Thank you guys for all your hard work!  Keep on rockin' in 2017~  <3
7 months ago
It is possible to turn such features off on a browser level - see links in the journal relating to this feature ("Performance and Infrastructure" section). Adjacent images in galleries are also prefetched (and are often the same ones as those linked by pools); turning prefetch off generally will turn that off as well, along with preload of subsequent pages in search results for Chrome.

Unless your performance is impacted by the feature (and we've tried to make sure that doesn't happen), or you have pressing cost-related reasons, we do not recommend disabling prefetch. Bandwidth is relatively cheap for us. However, doing so should not prevent anything working; it may just work slower.

The cost and potential benefit on slow connections is greatest if you have wide default mode and huge thumbnails enabled. If not, it should have less of an impact. Some browsers do not preload content over metered connections (e.g. cell phones), or at least heavily restrict the feature.
7 months ago
Awesome, good to know!  I'll leave it on for now at your recommendation; my only real interest was in potentially helping IB out, even if just in a tiny way.  :]
7 months ago
yeah, this sounds like a good system =^~°=
7 months ago
Why not have a checkbox option in the profile settings to preload images, then? I mean most people would keep it on but still
7 months ago
Mostly because as noted above, browsers supporting this feature have a setting to control predicative downloads, and we figure if members want it off here, they'll want it off elsewhere too. It may be restricted or disabled automatically if the browser can detect it's on a strictly-metered connection - otherwise, there's be a setting for it under "prediction" or "prefetch" (Chrome places it in "Privacy", probably because search engines often use it to preload websites at the top of the results page).

In general, it's good to have pages load quicker, which is why we implemented it in the first place. The cost to the site is low - additional data got a lot cheaper once everyone started hosting video - it's just something we need to account for in our purchases.
7 months ago
The Americas?
There is only one America. O.o
7 months ago
[The United States of] America is a country, but "the Americas" is the combination of North America and South/Latin America. We have members in both, and a cache in Brazil served from this cache.

There's various ways of dividing the continents up, but "the Americas" covers it all.
The word "America" was first used for what's now South America.

Of course, Hawaii is not in "the Americas", despite being in the USA!
7 months ago
I don't consider the U.S. as America.
For me (and I'm pretty sure the whole Latin America and Spain), America is a single continent.
It's always been a single continent, who made it two or three separate continents?
I can't truly believe it was used just for South America, when the first countries to be conquered where Cuba and Mexico (the last being in North America).
7 months ago
In Spanish-speaking culture (and many other languages, including Portuguese) the common term is "América". However, in the English-speaking world, the common term for this same area is "the Americas", and North and South America are considered to be very separate - two land masses, two continents, or maybe three for Central America, and different cultures.

Indeed, on the English Wikipedia, "America" redirects to the United States of America, reflecting the country-centric approach to this term in the English language.

In English, if you live (or are a citizen of) Brazil, you are a Brazilian or a South American; if you live in Mexico, you are a Mexican or Central American; if you live in Canada, you are Canadian or a North American; but none of them would normally be considered "American" by itself. [Canadians may even take offence to being called "American"…]

There have been many, many continuous discussions about this topic on Wikipedia, and ultimately all you can say is "this is how it is". This announcement uses "the Americas" because it is written in English, and so those in the USA, Canada, and other English-speaking regions will understand that this cache serves Central and South America as well.

Team America (a tad NSFW) provides a good example of what "America" means in English.
7 months ago
I can't truly believe and/or support that.
It may be stablished like that, but that's not what I've learnt.
I was taught to know and consider America as a single continent and the former North and South American "continents" as subdivisions or subcontinents.
I was also taught that Mexico is in North America.
Why do U.S. people exclude Mexico from the North of the continent?
7 months ago
It looks like they don't… at least for U.S. political purposes. Perhaps other Americans could chime in, but since NAFTA includes Mexico, it is considered part of North America. (Though a lot of people still want to build a wall…)

But many British people, including myself, do consider Mexico or parts of Mexico to be in Central America, probably because the UN also counts it as part of Central America for statistical purposes, so it's on various maps. According to them, this is because Mexico is or was demographically more similar to the rest of Central America than the rest of North America, even though it may be culturally dissimilar.

Basically there are differing definitions of Central America. As there is for North America, depending on whether you're talking about the continent (which includes Central America) or some other kind of region (see also Northen America).
7 months ago
This is really bad.
Mexico may be part of Latin America, but not of the Center of the continent.
I might accept your point since you didn't attacked me like other people, but I can't definatelly consider America is a country and the actual continent is two or three separate landmasses.
I'm Mexican, but also American since the country is found in America, northern region.

Going a bit out of topic here:
Trump is SO wrong. Most people who cross the border are not even mexicans. And most criminal people in the U.S. are not even from Latin America.
The most ironic thing is, that the U.S. is composed by people from the whole world.
7 months ago
It is all a big mess. Personally I would like to see fewer borders - it would make going to furry conventions a lot easier! Sadly, my own country is going the other way. But as you say, we got a bit off-topic. :-)

If it makes you feel any better, Mexico is served images from Inkbunny's Arizona cache (which is clearly in North America), not São Paulo…
6 months, 2 weeks ago
As an aside: Were you taught that Europe and Asia are one continent also? So your continent list is America, Eurasia, Africa, Australia, Antarctica? Just five? I'm just curious.
6 months, 2 weeks ago
Europe and Asia are separate due to some differences they have (or something like that).
Australia and Antarctica aren't continents. Australia is a country in the Oceania continent, with New Zealand.
So there are still 5 continents:
America, Europe, Asia, Africa and Oceania.
7 months ago
Runs up and hugs the pink bunny and the green dragon.
6 months, 4 weeks ago
Joke's on you guys; America's not even going to be around by 2019 XD
6 months, 3 weeks ago
It's the U.S. branch of a European company - their parent, OCOM, is also based in the Netherlands… so if worst comes to worst, maybe we can get them to give us a pro-rata credit. Though to be honest they'd probably just declare bankruptcy and be done with it.

I imagine we'd have bigger concerns, considering ~59% of our traffic comes from the U.S.
On the plus side, if the country melts down, we'll have little need for an image cache in the area!
6 months, 3 weeks ago
Lol, typical GreenReaper x3
6 months, 4 weeks ago
I love how transparent you guys are about server upgrades and what does what and why. It's easy to understand to and I like that.
6 months, 3 weeks ago
baguette !
6 months, 2 weeks ago
They also call some 1U servers 'pizza boxes', which I guess contain a slightly different type of bread…

As for the server, it hit 97.5Mbps out this week - so far, it's turned out to be a diamond in the rough! ;-)
3 months, 2 weeks ago
Does this imply that the site will close in 2019?
3 months, 1 week ago
No. It means this particular cache server in the Americas has been leased for two years. After that point we will decide whether to renew it or replace it - most likely the latter, as the deals tend to improve over time, and we expect our bandwidth needs to rise.

It means nothing about whether Inkbunny will continue beyond that point. In fact we recently leased a new main server until April 2020… but that doesn't mean the site will close in 2020 either.
2 months, 4 weeks ago
hmmm.....just a suggestion and this will be software intensive...how bought some sort of kinda tipsy...sourcing files and processing out to network of computers willing to give out space and processing?  2 weeks i'll have a free terabyte and top of line computer that ill be using to watch videos and draw....?
2 months, 3 weeks ago
It's not really useful from a CPU aspect as we already have enough CPU power on the main server, and will soon have more. Similarly storage is not that expensive. What has been beneficial in a few cases is storage + bandwidth in certain locations of the world for caches, but given our 24/7 operation it only works with a long-term commitment to a datacenter server (residential connections can't be used per their terms), which few fans have - we currently only have one sponsored cache, with Bad Dragon.

In almost all cases, it makes more sense for us to use money to get precisely what we need for the time that we need it than to rely on shared hardware. We try to do so efficiently, and at this point we think we've got most of the world covered adequately. We could splurge on all-SSD everything but it wouldn't be cost-effective and we really do want artists to get that money instead. :-D
New Comment:
Move reply box to top
Log in or create an account to comment.