Michael Biddle

As many of you have no doubt noticed by now, we are running vBulletin 3.8.4 here on vbcom. In fact, we've been running the code here since around 11AM Pacific yesterday. But, this version is not available to the general public yet. Here's a brief account for what happened, and why it is not ready yet.

Originally, we planned to have the release out two weeks ago. However, because everyone with sufficient access was busy with working on 4.0 alpha, we did not had the resource to release it then. As such, we had planned for it to be review our resources for release last Friday. When we looked at it, we figured that it is best to release it on Monday, so if anything goes wrong, we can be around to fix the release as soon as we notice the problems, instead of leave things to go crazy all weekend.

Monday, yesterday, armed with Kier's documentations on how to do a release, we tried to do the release at around 10AM. We started to package the updated code, and upgrade scripts, only to realize that the upgrade script is not running as its intended to. This lead us to discover that in the midst of 4.0 preparation and regretful departure of Kier, Scott, and Mike, we've forgotten about updating the software version. As such, all changes we've made to that point was made for 3.8.3 release, not 3.8.4, so non of the template changes would kick in, and the any of the updated options/phrases we've added will not deploy proper. We sorted out the problems, did tests on our local environments, and pushed the code here -- it is always better to break our site than thousands of customers' site ;)

Even though the upgrade went through properly, we figured that it is best to let the code sit a bit, and see if any problems creep up on us before releasing. And it did. At around noon, we were told that there is an issue with a fix we've put in. Apparently, the diff placed the fix in a wrong place, causing all drop down menus to break. We corrected that and pushed the code to the live server.

Shortly after that, at around 1PM Pacific time, we start to get hit with DOS attack... No, it wasn't a swarm of happy clients hitting F5 in member's area looking for their download. It started as a SYN-flood like attack, IB's net ops deployed some scripts to deflate it, and prevent it creep up on us again. The attackers then went and changed their tactics and targeted our databases. This went back and forth until around 3PM or so. By then, it was probably too late in the day to be doing a release.

We also received words that there may be some potential problem which could be used to cause forum to stop working for some installations. So we took the time to investigate in this issue. Long story short, if your forum allows very very very long posts -- test data for reproducing the problem was 1.4MB or something in size -- the vbchop() function we use to make thread preview may eat up all the memory php is allowed to use. Going back and forth with the person who reported this problem to us for a bit, we were able to come up with a workable solution to this problem.

After all these chaos, it was too late to do a release, so we are going to do a take 2 today after our daily meeting -- probably around 10AM again. Hopeful thinking is telling me that we've done enough stress test on this thing, and it shouldn't have any problems this time around... but hey, you never know what might explode in your hand these days.

PS: As noted in the blog title, I forget to blog; so don't be too surprised if I forget to reply to your comments, too.
Attached Thumbnails



More...