Over 300k req/min

Last week I pushed a major update to a codebase I had been working on the past few years, and saw some amazing results.

This is going to be a short entry, as the codebase actually had an unrelated issue that I am working through, but soon I should have some more detailed information about the actual stats.

From what I remember off the top of my head, there were peaks of 370k req/min, lows of 100k req/min on 8 standard 1x servers hosted on Heroku, plus the database server on Linode, it was essentially hosted on servers costing less than $250/mo.

The last article I wrote here: https://www.dougcodes.com/go-lang/best-priactices-of-writing-software-to-scale lays out exactly how I did it. The way I write Go code in a sense has become very “boring” since I am essentially making key-pair hashes to store information in-memory, and use a ticker in the background that runs every minute to either digest or ingest that data.

Of course though on the first request of a newly found key, the request has to go directly to the database but I will be doing updates that will instead hit a Redis server first, and then MySQL is pumped right into Redis so on subsequent releases I won’t be needing to ever hit the MySQL database for existing live endpoints of the service I’m creating, they will always get the data directly from Redis.

During these peak times of traffic, I saw the server load go up to as much as 5.0 for the dyno load, and about 45 MB of RAM being used. 1 MB of swap was being used which was suspicious, but I’ll look more into that later.

The worst thing about it, which isn’t really TOO bad when you think about it, is in a 24 hour period of running this where the low was at 100k req/min, there were only 300 errors and the majority were disconnects on the client-side or timeouts, so it’s entirely possible it wasn’t due to the actual application at all.

Again, I should have a follow-up article soon with some more details, but next stop is 1 million req/min on the same server setup!

Last Updated: 2015-07-14 18:51:55 +0000 UTC

What are your thoughts on this?


RSS feed

Follow Doug On Social