The newest forum disasters

General discussion!

Moderators: Kisai, Chris Crosby

Forum rules
Keenspot is not responsible for content posted by members of the forum.

The newest forum disasters

Postby Gav on Thu Feb 27, 2003 12:18 am

Hey all,

So it's more than likely you've already been reading my updates on the forum problems. If not, I've attached my update log at the end of this message.

But to be brief, here's what happened and why it shouldn't happen again. There's some kind of horrible bug with either our filesystem (ReiserFS) or our operating system (Linux 4.5.19 and/or 4.5.20) or some combonation of the two. The plain truth, as I've learned it over the years, is that Linux just sucks. Now don't get me wrong, it doesn't suck anywhere near as much as anything written by Microsoft, but the developers have rushed its advancement too quickly, likely as a result of their competition with Windows (That's right, I blame Windows for Linux's faults). As far as I've seen, there is simply <b>no</b> stable version of Linux. Every version of Linux in existance has bugs. They don't bother to find and fix all the bugs before introducing new one. So each new version introduces as many new bugs as it claims to fix from previous versions.

So what's the solution? <b>FreeBSD!</b> A systadmin friend of mine suggested this long ago, but it's hard to simply abandon a whole operating system and the learning curve that comes with it. But we've been doing it recently and I truely believe that BSD is God's operating system. Why is it so good? Because they don't release a version as "stable" until the development version is <b>two years</b> old. There's no rush to introduce new features and there are none of these horrible disasterous bugs that destroy every file being written to during an unplanned power cycle. As I read on a website somewhere: "BSD is for people who like Unix. Linux is for people who hate Windows."

Case in point: Since we started migrating new servers to FreeBSD many months ago, the new BSD servers have not crashed <b>once</b> due to software (we had one major problem with the Keenspace server, but that was due to a failing hard drive).

So, OS plug over. Point being, we set up a new dedicated server for <b>each</b> forum (one for Keenspot and one for Keenspace). These servers are running (you guessed it) FreeBSD.

In addition, Nate is setting up a daily backup system today. So even if something equally horrendous should happen again, we'll only lose one day's worth of posts at the most.

In total, about 15,000 posts were lost--pretty much all of February. The <b>text</b> is still there, but the information that links them into threads is gone. As stated below, we're working on a reconstruction program in case any of you volunteers <b>really</b> want to try to reconstruct any old topics. It would involve sorting through those 15,000 posts, so it's not a light task, and I don't know that we'll even actually get this program written.

I apologize for these problems. The sad truth is that computers are unpredictable. We learn something new every day, but they keep giving us new horrible things to learn every day as well. I'm also sorry the forums had to be down for so long, but I hope you can all see it was done in the interests of providing the best possible forum safety for the future.

Here's the update log as I wrote it for the past week, for those that missed it:

Forums Down

The forums are seriously hosed. A filesystem bug in either Linux or ReiserFS seriously corrupted them. It is not known at this time when they will be back, or if they are even recoverable. I'm doing everything I can possibly think of to repair the damage, but it's all touch and go at this point. We apologize for the inconvenience.

Progress updates:

(2/22/2003 - 1:40 PM PST) Looks like it might be possible to restore most of the database, but at the moment, it appears that a great deal of posts may be destroyed (like the past month). I'm continuing to see what can and can't be recovered.

(2/22/2003 - 5:00 PM PST) Well, the last month's worth of posts are just plain gone. I've done everything I can think of to get them back. The database just got seriously corrupted. This seems to be some kind of bug with the filesystem that occurs when a server reboots. Even though I've been able to repair the damage to the database so that it works again, I'm reluctant to reactivate the forums for fear that the same problem could occur. I will have to discuss with Nate tomorrow about our plan. We will probably move the forums to a different, dedicated server with an operating system and file system without this critical bug. I'll talk to Nate and we'll make a decision tomorrow.

(2/24/2003 - 2:00 PM PST) Okay, we've decided that now is a good opportunity to transfer what's left of the forums to new servers, so that each is running on its own dedicated server. I'm doing the database dump right now. Please note that we are not looking for submissions of posts you may have archived. Having HTML of lost posts doesn't help us get information back into the database, unfortunately.

(2/25/2003 - 10:30 AM PST) Hrm. Maybe there is a spot of hope. I think that some of the post information is actually in there if I can figure out a way to reconstruct the data in the topics table. If that's the case, I may be able to retrieve some of the missing posts from the past month. Stay tooned.

(2/25/2003 - 12:32 PM PST) Grr. I may have been too optimistic. Let me explain some details of what I've found. First of all, there are three tables in the database listing data for the posts. There is the topics table, the post info table and the post text table. The topics table is missing data for everything since about January. I thought I could recreate the topics from the data in the post info table, which has data for posts from 165 topics that are missing from the topics table. Unfortunately, the post text table (which lists the actual text of all the posts listed in the post info table) contains the text for 16151 posts since the last listed topic, whereas the post info table contains data on only 257 posts since the last listed topic. That means that while the text of about 16,000 posts has not been lost, there's no information on putting them back together with their topics in the right order for more than 257 of them. I can reconstruct those 167 topics, but more than likely, they would only contain a spattering of random posts from the middles of those topics. It's infuriating to me that the text of each post is all there but I have no way of putting it all together, save a monstrous monthlong jigsaw puzzling session.

(2/26/2003 - 5:00 PM PST) Okay, I'll have the forums back up tonight. I couldn't save any topics like I thought I might. The few topics I was able to save only had scattered posts from the middle of the topics, and thus the topics had no subjects and were totally confusing anyway. It just wasn't worth it. I've enlisted the aid of one of our volunteers to put together a "post reconstruction program" for any of you that might be insane and dedicated enough to want to sort through the text and try to piece together old posts. It's probably academic at this point. But that could take some time and I've now worked out all the bugs involved in moving the forum to a new dedicated FreeBSD server and should be able to turn the forums back on, sans one month of posts, before midnight tonight.


Darren "Gav" Bleuel
(Nukees, an atomic comic)
User avatar
Keenspot Whipping Boy
Posts: 4024
Joined: Tue Mar 07, 2000 11:59 pm
Location: Berkeley, CA Disposition: pissy

Postby Dennis J. Squidbunny on Thu Feb 27, 2003 12:50 am

I love you.
User avatar
Dennis J. Squidbunny
Keenspot Juggernaut
Posts: 3781
Joined: Thu Dec 20, 2001 12:00 am
Location: Orrrstraaaaliaaar.

Postby Sam on Thu Feb 27, 2003 12:52 am


We shall survive!

... sigh. My images are always cursed to freeze out. Guess I'll just keep swapping them!
Last edited by Sam on Thu Feb 27, 2003 10:06 am, edited 4 times in total.
You found one of my posts with a signature in it. It must be ancient.
User avatar
Keenspot Deity
Posts: 12656
Joined: Tue Jan 14, 2003 5:45 pm
Location: a1/1

Postby Marcos on Thu Feb 27, 2003 12:54 am

Thank you very much for keeping us updated on what was going on! It's much nicer to know that the problem is being worked on and what progress is being made than to simply see a "forum is down" notice.
User avatar
Keenspot Juggernaut
Posts: 5105
Joined: Sun Sep 15, 2002 2:53 am
Location: Spokane, Washington, USA

Re: The newest forum disasters

Postby Trillan on Thu Feb 27, 2003 1:20 am

I came to the same conclusion a while ago. Sorry it caused you grief...
Grand Poobah Keenspotter
Posts: 593
Joined: Thu Jun 07, 2001 11:00 pm

Postby Guest on Thu Feb 27, 2003 2:41 am

I love you man. I really do, in a non-homosexual way.
Walk softly...with a big gun.
User avatar
Keenspot Despot
Posts: 2086
Joined: Mon Sep 09, 2002 6:56 am

Postby BBlalock on Thu Feb 27, 2003 2:53 am

Thanks for getting the forums back up!

Quick question, why was 4.5.x chosen over 4.4.x?

In the Linux versioning system 4.5.x is a testing release, and 4.4.x is "stable". So it seems to me that 4.4.x might have been a better choice.

I'm not meaning to imply anything by this question, as what I know about running a forum on any platform could fit in a very small thimble and leave plenty of room for the Hulk's thumb. This is just (mostly idle) curiousity.

The important part is that the forums are back up, and I wholeheartedly thank you for that.

And thank you for implementing a post reconstruction program, send all the Schlocky posts my direction, I'll be glad to help sort them out.
<script language="JavaScript">
document.write('<a href=><img src="' + Math.floor(Math.random()*21) + '.gif width=486 height=60 border=0></a>');
User avatar
Keenspot Despot
Posts: 2449
Joined: Tue Jul 24, 2001 11:00 pm
Location: Stop the world; I want to get off.

Postby bryann on Thu Feb 27, 2003 3:19 am

personally I'd blaim ResierFS before the rest of the kernel - the 2.5.x series is just as stable as 2.4.x or even 2.2.x generally speaking.....

great work guys
The definitive CRFH!!! fanart archive :
My CafePress Store
User avatar
Grand Poobah Keenspotter
Posts: 322
Joined: Sat May 26, 2001 11:00 pm
Location: Australia

Postby Kaoru on Thu Feb 27, 2003 4:07 am

I concur. Linux is great. ReiserFS, on the other hand, sucks. Why in the world were you using it? The only thing that truly distinguishes it is the fact that it can do journalling. But if you want journalling, you should have been using the ext3 file system, which impliments it far better and is an extenstion of the standard Linux file system (ext2).
That's not normal: normal is coming home to find your mother nailed to a cross screaming verses from the book of Mormon, your dad performing a dark ritual with your brother as the sacrifice, and your sister getting railed by a goat.

Then things start to get weird.
User avatar
Keenspot Despot
Posts: 1836
Joined: Tue Mar 27, 2001 12:00 am


Postby Jamie on Thu Feb 27, 2003 5:07 am

Well, I'm sure the idea of what to blame will continue, but I want to Thank Darren for all his extremely hard work and dedication to this other other Keenspot issues. The same goes for Nate! We love ya both!

You know, in that don't touch me way. ;)

Take care,

Jamie, KCI in the Order of the Knights of Jubal
Probi Immotiqve Este: Be virtuous and cool. PIQE.
Clan of the Cats
Melpomene: The Chaos Orb: An online graphic novel by Jamie Robertson and Clint Hollingsworth
Sebo: A weekly webcomic about a girl and her cat.

"I don't believe in atheism." - Hawkeye Pierce M*A*S*H*
"Some times the Magic works. Some times it doesn't." - Cheif Dan George Little Big Man
User avatar
Keenspot Juggernaut
Posts: 3554
Joined: Thu Mar 09, 2000 12:00 am
Location: NC

Postby vorn_the_unspeakable on Thu Feb 27, 2003 5:16 am

I know there's at least one person (yours truly) who lost important writings to the crash -- Would it be possible to run a regex on the busted posts for "Vorn\n\n?-+\n\n?Chapter \d\d" ("Vorn", one or two newlines,at least one minus, one or two newlines, "Chapter ", two digits) and return those posts that match it? There should be six or seven in the broken posts, about 95 in the entire board. And, yes, I'd like to volunteer to help rebuild...

User avatar
Keenspot Despot
Posts: 1540
Joined: Mon Feb 19, 2001 12:00 am
Location: Stratford, CT

Postby ToxicFrog on Thu Feb 27, 2003 5:25 am

All hail the Gav, resurrector of forums!

On a side note, I would be willing to help with the thread restoration for CRFH. The rumours that you wanted HTML cache trawls have had one beneficial side effect: we have a better idea of how the threads should look once reassembled.
FLEET: For Honour, Courage, and Rampant AIs
User avatar
Grand Poobah Keenspotter
Posts: 610
Joined: Wed Nov 20, 2002 5:36 am

Postby wrightc on Thu Feb 27, 2003 6:17 am

I've never had a problem with ReiserFS myself. Of course, right now my Linux box is packed away so I can say with complete honesty that I'm not currently having a problem with ReiserFS at all. :)

BSD will do fine, though. Groovy.
Christopher B. Wright (
The Internet's Most Dangerous Cartoonist
Help Desk:
Keenspot Despot
Posts: 1956
Joined: Sun Mar 12, 2000 12:00 am
Location: Troy, NY USA

Postby blondeGeek? on Thu Feb 27, 2003 6:53 am

Gav, Nate...


Now, go get some sleep without having nightmares about a corrupted forum. :wink:
Geek Girls Rock!
User avatar
Grand Poobah Keenspotter
Posts: 507
Joined: Tue Jul 02, 2002 4:30 am
Location: Geekdom, USA

Postby MadKat on Thu Feb 27, 2003 7:11 am

Whatta Guy!

And whoo... I just love all this alternative OS stuff. Mind you, I'm sitting here on a Win NT machine, but I admire those of you who can deal with the alternatives.... I do have Redhat Linux partitioned on here...just haven't had time to figure out what it's good for yet...

User avatar
Keenspotter Supreme
Posts: 61
Joined: Sat Nov 30, 2002 7:26 am
Location: Gorham, Maine, USA

Postby Ishtar on Thu Feb 27, 2003 7:20 am

yey!!!!!!!!!! the forums are back!! thankies, we all wub jo0, ^^.
User avatar
Keenspot Despot
Posts: 1020
Joined: Mon Dec 16, 2002 1:53 pm
Location: in my sparkly town of darkness

Postby McBean on Thu Feb 27, 2003 7:29 am

all I know is that you triumph in the gace of adversity. semper pie, fellas
Matt C. "mcbean"
User avatar
Grand Poobah Keenspotter
Posts: 569
Joined: Sat Jan 18, 2003 4:19 pm
Location: Virginia

Postby Major Tom on Thu Feb 27, 2003 7:32 am

thanks for all the hard work.
User avatar
Major Tom
Keenspot Deity
Posts: 18114
Joined: Thu Jun 21, 2001 11:00 pm
Location: Ground control

Postby Maritza Campos on Thu Feb 27, 2003 7:42 am


Many thanks to Gav for minimizing damage and getting our forums back. You rule.

As for the lost posts, I think it's better if we try to restore the important ones on our own. Gav has enough work already, and I have always said no one ever should rely on the boards for their only copy of important stuff such as fanfics.

And Gav... chesterforums? :D

User avatar
Maritza Campos
Keenspot Juggernaut
Posts: 6251
Joined: Fri Mar 10, 2000 12:00 am
Location: Just behind you

Postby Bemral on Thu Feb 27, 2003 7:49 am

Thanks Gav,
You're helping to save my (in)sanity at work.
Working hard at doing nothing
User avatar
Grand Poobah Keenspotter
Posts: 765
Joined: Wed Jul 17, 2002 8:56 am
Location: On the Edge of the Mundane

Postby IceSniper on Thu Feb 27, 2003 7:56 am

You guys rule.
User avatar
Keenspot Juggernaut
Posts: 4413
Joined: Fri Feb 22, 2002 12:00 am
Location: Under your bed.

Postby Tim Tylor on Thu Feb 27, 2003 8:54 am

Hey, thanks from me too. :D

Some day I must try to find out what a ReiserFS is. :oops:
User avatar
Tim Tylor
Keenspot Despot
Posts: 1119
Joined: Sat May 12, 2001 11:00 pm
Location: St Austell, United Kingdom


Postby Ucchan on Thu Feb 27, 2003 9:32 am

Yeah Gav, thanks so much for resurrecting the forums! ^_^

I agree, Maritza, there was another recent thread I was going to get involved in, but we can just start afresh. No real need to rescue February's threads when Gav saved everything else.

Thanks again,
"But, Asha, I was all ready to set up my shrine to the U-sisters, dual Mistresses of Kawaii." ~Freemage, fellow Keen Forumer
User avatar
Keenspot Despot
Posts: 2041
Joined: Wed Oct 04, 2000 11:00 pm
Location: Toronto, Ontario, Canada

Postby Gav on Thu Feb 27, 2003 10:20 am

Maritza Campos wrote:And Gav... chesterforums? :D

We name all our servers after Keenspot characters. The forums used to be on "kingluca." I figured that "chester" was the best name for the new dedicated Keenspot Forum server. :wink:

As for why we use ReiserFS and Linux 4.5.x, I can't really answer that. Nate's the one involved with installing such things. I do know that the motivation behind Linux versions has always been exactly what I've detailed above: NO VERSIONS OF LINUX WORK. This is a fundamental flaw of Linux. Every version of Linux has some fatal bug somewhere. Most average desktop users never see these bugs, but when you have as much traffic and varied applications running as we do, you run into them all.

When they do happen, usually the result is something horrific. Even if it's just a system crash--the average user is used to their computer just suddenly freezing for no obvious reason whatsoever, but when it happens to us, heads start to roll. Pagers go off at four in the morning.

So we spend days doing a websearch to find the problems and what we find is always the same. On some Linux message board somewhere, someone will have described the exact same problem and the accompanying advice is always, "Oh, that's a bug in Linux x.x.x. It was fixed in Linux x.x.x+y. You need to upgrade."

The problem, as I've pounded into the ground by now, is that for every x.x.x, there's a new set of bugs for which the only advice is to add another "y" on. I would say it's safe to say that we have gone through every "stable" version of Linux from the 2.2's to 2.4.20.

If they would just leave it alone for a few years until all the bugs were fixed and then released one stable version (like FreeBSD), it would be a decent operating system.

No Linux version is "stable."

Darren "Gav" Bleuel
(Nukees, an atomic comic)
User avatar
Keenspot Whipping Boy
Posts: 4024
Joined: Tue Mar 07, 2000 11:59 pm
Location: Berkeley, CA Disposition: pissy

Postby wrightc on Thu Feb 27, 2003 10:29 am

Didn't Linus Torvalds move into your neck of the woods? I can't remember. If he did, you should drop by and explain that to him. :)
Christopher B. Wright (
The Internet's Most Dangerous Cartoonist
Help Desk:
Keenspot Despot
Posts: 1956
Joined: Sun Mar 12, 2000 12:00 am
Location: Troy, NY USA


Return to Keenspot Central

Who is online

Users browsing this forum: No registered users and 0 guests