I want to announce a big backend server upgrade we just completed.

Since the move to Switzerland, after hitting the CPU limits for the first time on the new server, we began digging in to the CPU usage and tracking it and matching it to siteload times.

/u/d3rr made the amazing discovery that the main hosting process for saidit is NOT a multi-threaded process. Meaning the main sitecode process was basically running on one core. So while other processes would use our 4 cores that we upgraded to, which helped a lot, the main process would not use more than 1 core. And sometimes this would cause the site to take 5+ seconds to load if these requests got stacked up, which was happening more and more as saidit got more traffic. This system is a flaw with the reddit open source code that was not really explained in their documentation, because they obviously switched that process out for another one in their production server. It seems they left it in a sort of testing mode, without telling anyone on the github.

Figuring this out is basically what we've been doing the last 2 weeks.

After identifying the problem, d3rr just today figured out a way to multi-thread this process so now it actually uses all 4 cores. Thank you d3rr. Now those 5+ second random hangs are gone and the site is smooth and fast. This is a huge upgrade for saidit.

That's all the updates for today! Thanks everyone.