Inkbunny has historically served content, processed data, and produced web pages from just one server. This kept administration low, but divides one machine's resources between these tasks. It also means you had to fetch files from the Netherlands – a bit of a slog if you live on the U.S./Canadian west coast, let alone Australia, Brazil or Japan.
This week we launched two content caches, both to bring the files closer to you, and to relieve the main server from the burden of serving an average of 35MBit/sec (peaking at 50MBit in the evenings).
Both caches check their RAM and disks for a copy of the requested file, and give it to you directly if they have it; if they don't, they fetch it from another server and store it; but in other respects they differ.
The USA cache, a VPS provided by toy maker Bad Dragon, is a pure web cache - starting from nothing, and storing the most recently-used files up to its capacity. (Edit: Our new cache in AU does the same.) Our secondary server in the Netherlands contains a backup of Inkbunny's image library, updated nightly, and as files are accessed; it uses this library as its cache, rather than having a separate set of files.
The difference between the two is like the difference between a large local library and a national library, such as the Library of Congress - it might take a little longer to get it from the latter, but it's almost certain to have it; the local library will be a bit faster if it does have it, but slower if it has to order it.
Donors were already paying for an on-site backup, so why not use it to the full? As a bonus, we now have a fresher backup with redundant storage, updated over the course of the day. (Private and hidden files are not accessible via the cache because they require authentication; they continue to be backed up daily.)
Why not store a full backup in the USA? For a start, Inkbunny is 900GB and growing; the cache has 50GB of SSD and 50GB of hard disk to play with. This is enough to store around 35,000 original files and 1.35 million screen- and thumb-sized derivatives - all non-private files accessed within three days of regular use. We're using SSD for everything but original files, where random access is of less concern.
We also prefer to keep your private files and personal data under staff control. As much as we appreciate Bad Dragon's generosity, they're a separate organization in a separate jurisdiction. (That's also why JS and CSS comes from our main server.)
This cache has a positive impact for users basically everywhere to the west of Europe - most of North and South America should be seeing significant improvements, and access times from Japan and Australia have been cut by half. The U.S. East Coast, however is directed to the Netherlands because it isn't any faster to go to Arizona. Perhaps a spot for a future cache? In any case, traffic is fairly evenly divided, although it peaks at different times - midnight server-time in Europe, and six hours later for the USA.
This feature was a team effort – Salmy handled the heavy lifting code-wise, including integration of the GeoLite2 location database, while I specified the cache architecture, determined the traffic division, and tweaked the nginx configurations. (And thanks to Dari and varka on the Bad Dragon side!)
I'm very pleased with the performance of the system so far, and it should reduce or eliminate several potential bottlenecks in the future - most notably, bandwidth.