failing like never before

Some of you may have noticed, that about three weeks ago, all but one of the blogs that I was hosting on my server, crashed, and stayed down for about two days. When I tried to access any page, I got a 500 error, and it was traced back to a Typo application error. This was really weird, since prior to the strange epidemic of 500 errors, I hadn't touched my server in quite a few days. However, one of the blogs on my server was working absolutely fine.

The application that was still functioning had been on my server the longest time. I thought that perhaps the newer version of Typo was unstable and that the blog that was still running was using a more stable code base, but no; all my blogs were running the exact same version of typo.

I went through some basic checks to see what was going on. The log files weren't particularly helpful in telling me what had happened, and mostly just showed that the applications had all failed around the same time, for no discernible reason. I use fast CGI, so I tried switching to just regular CGI, but to no avail. I checked the permissions of files, and even set the permissions for the whole applications to 755. I double checked source files and their timestamps, and found that it had been quite some time since any of them had been altered. I also checked to make sure that rails hadn't gone and updated itself to a less stable version, but it seemed as though nothing on my server had changed. My applications had just died, for absolutely no discernible reason.

It seemed that the easiest thing to do, was to just wipe the application out, and put rails back in. So I took one of the blogs that had hardly been used, and wiped it clean out, but left the database intact. I downloaded a fresh copy of the latest stable copy of Typo, tared it, made the necessary configurations, and found to my total surprise that it gave me the exact same 500 error! After much fiddling and triple-checking to make sure I hadn't messed up the simple installation process, I was still getting the same 500 error. I wiped the application out and tried yet again, getting the same results. The only thing that could seem to be wrong, was the database, so I wiped the database clean, and did a completely fresh install. Still the same results. I had never been this confused in my life, and that includes my Chem 20a final.

But remember, I still had one functioning blog left on my server. So I decided to copy the source code from that 500-error-imune application, and put it on a sub-domain and connect to a new database. And it worked. I have no idea why that particular instance of Typo was immune to the "500 epidemic" but it was, and I was too happy to ask questions.

Three weeks later, I still have no idea what the hell happened. If someone could enlighten me, I would be very happy.