Load balancing your load balancers sounds a bit ridiculous but yes, that’s what we have to do to scale.
For about three and a half years, we’ve had a single pair of load balancers. One active, one in “hot standby” mode to take over if the active one goes offline. It’s worked really well, but they were reaching capacity. I’ve pulled all sorts of tricks out of my hat to reduce excess load, but the time finally arrived where new hardware was the only option left.
So, we just added two more load balancers to the equation. But these new ones are twice as powerful as the old ones, and instead of active/passive pairing, they’re both active. This means we have a total of three active load balancers, and these are load balanced with DNS round robin, with DNS monitoring/failover on top of that to quickly and automatically remove one from the pool in case it has a problem. The old ones act as a front for our entire service with a bunch of virtual services running on them, and hence remain paired (keeping them in sync would otherwise be a nightmare). The new ones are just for additional tracking capacity and nothing else though, so this setup is great for us.
Speaking of tracking capacity, this just quintupled it to ~25,000 pageviews/second. Just in time too, because last year, online activity on “Black Friday” and (cringe) “Cyber Monday” sent a ton of extra traffic our way and knocked us offline for a bit. This year, we’re ready, so bring it!
Something else we just changed today is that the static resources for our web site are no longer hosted on “static.getclicky.com”. A problem common with many trackers is that their domains are on blacklists of ad-blockers and other privacy software. These days, almost weekly we get an email from someone saying our site looks funny. This is because their browser, or their OS, or their ISP, or who knows, is blocking static.getclicky.com from loading anything, so the stylesheet doesn’t get loaded (nor any of the images, javascript, etc).
But we’ve had a secondary domain, “staticstuff.net”, that we’ve used as a generic CDN domain for our white label service for a while now. I had to change a few things around and get SSL running on it, but now that that’s done, all assets for our web site (except the tracking code itself) will load from this domain instead. It points to the same servers, but it should bypass 99% of these blacklists because they are almost always based on domain name.
