PDA

View Full Version : Board malfunction



a1call
2011-Jan-04, 06:21 PM
Strange, my last post did not raise the thread or attribute the last post to me in the main page of the Q&A!

megrfl
2011-Jan-04, 06:39 PM
Yes, something is amiss. Bautforum's website most often wont load and if it does, it is missing icons. Also, if I move around within the site it takes forever to load or I get booted with an error.

megrfl
2011-Jan-04, 06:41 PM
I got about 3 database errors with the last post and it is not showing up as being posted.

CJSF
2011-Jan-04, 07:21 PM
The database seems to be screwed up. Some posts that show as "new" on the main forum page don't appear when the forum is entered, and some posts don't show on the main page, but appear when you enter that forum.

CJSF

Jim
2011-Jan-04, 08:08 PM
I've been getting database errors, too, every time I post a comment. Most of them still go through, but someone has avoided an infraction in one case.

I'll just double up later.

kleindoofy
2011-Jan-04, 09:01 PM
Judging by the error description, this sounds like it could be a (reading?) error in one single table of the data base, i.e. the table that records the sums etc. of the vectors which point between the individual posts and the threads.

vBulletin may offer a repair function, but considering the size of the database, it could take weeks to run.

On the other hand, the forum has been acting up in the past couple of weeks, so it might just be a glitch in the synchronism caused by a table being read too slowly. Err, if that's possible.

Edit: reloading the page (F5) resolves the immediate database error.

danscope
2011-Jan-04, 09:16 PM
Me, too . Data base errors , slow response . Quelle tragique . Time and patience .

Dan

Strange
2011-Jan-04, 10:36 PM
The search function isn't working properly either.

Obviousman
2011-Jan-04, 10:54 PM
Happening for me, too.

I made two update posts in the Chapel Bell thread. Each time it resulted in a DB error, but when I went to the main page it was listed as a 'latest post'. Clicking on that brings up the thread and the posts were there... but if you simply go to the board, it does not list that thread as being updated (i.e. it is not at the top of the thread list).

Additionally, when I tried to use the search and advanced search functions to get the thread back (I mis-spelt a name) the result was again a DB error.

Lastly, using the 'todays posts' function, I only get two posts listed - none of which are the posts I made.

Hope this helps in some way.

Cheers!

slang
2011-Jan-04, 11:14 PM
There were some bigger problems earlier today, including a database down error, or something like it. Maybe some re-indexing is required.

astromark
2011-Jan-05, 04:09 AM
" Hullo Huston, We have a problem...." just posted and it says I did and its gone, gone, gone. really gone... now I'm gone also... I wonder if this will be gone...
No., here it is.

astromark
2011-Jan-05, 04:11 AM
but I can not edit... Oops yes I can... but it said not.

JAXAi
2011-Jan-05, 04:44 AM
Doing an analysis of the server currently their Apache server software is out of date (so is their VB software; the most recent update is 4.0.8 and its free to update for licensed sites). However without doing the upgrades (which would take about 30 minutes of site downtime if you are fast about it) repairing the tables in Admincp could help (Admincp>Maintenance>Repair/Optimize Tables).

The search function is completely unusable at this point, it only comes back as DB errors.

Spoons
2011-Jan-05, 07:32 AM
Ah, I just fired off a message to the Admins and a few Mods, then I saw this thread. Sorry guys, if you've already received a pile of messages about that.

gzhpcu
2011-Jan-05, 08:19 AM
What I see is "Replies=0" on recent threads with plenty of postings...

Hornblower
2011-Jan-05, 11:14 AM
This is a test of the Last Post function.

Hornblower
2011-Jan-05, 11:16 AM
What I see is "Replies=0" on recent threads with plenty of postings...


This is a test of the Last Post function.Addendum: Aforementioned problems unchanged so far. Still unable to use search functions.

ggremlin
2011-Jan-05, 11:18 AM
It looks like the Indexing is fried.

Jens
2011-Jan-05, 02:08 PM
Maybe some re-indexing is required.

I'm sorry David, I can't do that.

djellison
2011-Jan-05, 03:56 PM
the most recent update is 4.0.8 and its free to update for licensed sites).

The time to do that upgrade isn't free, and the risk of causing more trouble to the forum is non zero.

NEOWatcher
2011-Jan-05, 04:04 PM
Originally Posted by JAXAi
the most recent update is 4.0.8 and its free to update for licensed sites).The time to do that upgrade isn't free, and the risk of causing more trouble to the forum is non zero.
Plus; do we even know it's a Database problem itself, or from an outside cause? If so do we know what that problem is, and if the update solves it?

Anyway; this whole thing has gone wonky. Unread posts showing as read posts and vice versa; Icons not really reflecting what's happening; post counts all messed up; thread updates not showing consistently; frequent DB error messages and site unavailabilities...

Welcome to the new server? :think:

Swift
2011-Jan-05, 04:40 PM
Fraser has been notified of the problem. We'll try to keep you updated as we learn more.

JAXAi
2011-Jan-05, 04:45 PM
Wishing he could post an image...

"Everybody calm down, its all under control"

In Japanese we say "Gambare" or がんばって!Gambare Fraser!

Gillianren
2011-Jan-05, 06:35 PM
The time to do that upgrade isn't free, and the risk of causing more trouble to the forum is non zero.

I mean, we were all so thrilled the last time we had an update around here.

a1call
2011-Jan-05, 06:52 PM
It might come down to having to restore a server backup that hosts create automatically. We would lose some recent posts, but at least the board would function again.

JAXAi
2011-Jan-05, 07:15 PM
It might come down to having to restore a server backup that hosts create automatically. We would lose some recent posts, but at least the board would function again.

That would be the absolute last resort. Anyone worth their salt with either Apache or MySQL should be able to fix what is wrong without resorting to a restore with possible data loss. This is coming from someone who owns a website development company in Japan and is currently managing (personally) 14 MySQL databases (mostly Drupal builds, but a few VB builds as well). Its not rocket science (pun intended)

kleindoofy
2011-Jan-05, 08:12 PM
... do we even know it's a Database problem ...
There's not really much else it could be, unless it's the connection between the database and the php-code.

The database isn't one huge file. It's a set of tables and the elements we see on the screen are divided up among them. The posts are in one, PMs in another, user data in another, thread headers in another etc., etc.

The php-code reads the different tables and patches them together to what we see on the screen. For instance, the posts aren't sorted by threads, or sorted at all, they're just in one large table, where each post has a unique number.

Overly simplified, when we click on a thread, a piece of the php code is activated which reads a table of the database which holds the corresponding post numbers for that thread. These are then read from the post table in the database in the order recorded in the thread table. The information about the individual members is then pasted in from other tables, etc. etc. Each step along the way is coded in the php software. The database itself is passive.

One possibility is that there's a hiccup in the table which holds the thread history summaries, i.e. just numbers (statistics), not content. The database error reports could be a sign that it's not being renewed correctly. The posts and the threads themselves seem to be ok.

Another possibility is that the php code which reads the database has acquired a bug and is not reading things it should be reading, but that's not really very probable. If that were the case, the board would appear much more screwy than it does now.

As someone wrote above, a complete re-indexing may be necessary, unless the indices are fine and it's just a corrupted line in the php code.

Re-indexing can be done at run-time, but it's a slow thing. If it ran with high-priority, the board would be blocked, almost like during a DDOS.

slang
2011-Jan-05, 08:45 PM
I mean, we were all so thrilled the last time we had an update around here.

In fairness, that was a major version number upgrade (which are inherently more risky, with more changes) in combination with a hosting change. Minor updates are less risky, but not guaranteed annoyance-free. I'm sure Fraser is perfectly capable to decide whether BAUT needs every minor update, and which ones can safely be skipped. Besides, we don't even know if the database crash (if that is what happened) had anything to do with the vB software.

ETA: replying to this thread seems to be useless right now, since page 2 of this thread can't be displayed.. Perhaps start a new thread, which can be merged later when things return to normal.

tommac
2011-Jan-05, 08:51 PM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.

tommac
2011-Jan-05, 08:52 PM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.


As I write this I get:
The following errors occurred with your submission
Database Error

astromark
2011-Jan-05, 09:08 PM
"Hale. Open the door Hale...." * Peep, " Huston, We have a problem." peep.

But it does seem to be fixed... so I offer a thankyou.

astromark
2011-Jan-05, 09:09 PM
"Hale. Open the door Hale...." * Peep, " Huston, We have a problem." peep.

But it does seem to be fixed... so I offer a thankyou.

but no... all is not well is it ?:evil:

astromark
2011-Jan-05, 09:10 PM
"Hale. Open the door Hale...." * Peep, " Huston, We have a problem." peep.

But it does seem to be fixed... so I offer a thankyou.

but no... all is not well is it ?:evil: Who dripped water on the gremlin ?

caveman1917
2011-Jan-05, 09:17 PM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.

Whilst i'm sure that is a good meaning suggestion, what if something goes wrong? Having your own server means you have a legally enforceable contract for the service, you can sue them if anything goes wrong. This is not the case when you just let a friend host it for you, no matter how well meaning the intention.

caveman1917
2011-Jan-05, 09:21 PM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.

Whilst i'm sure this is a well meant suggestion, what if anything goes wrong? There is no legally enforceable contract then, no one to sue when the entire thing vanishes in thin air. There are just too many risks involved i think, no matter how well the intention.

caveman1917
2011-Jan-05, 09:26 PM
It's gotten even weirder. I posted a response, and as usual got the "database error" thing, but i clicked quick reply again to see if it would object to me posting the exact same thing twice in under 15(?) mins, thus knowing the post did get through. It shows up in the forum view (last post by caveman1917...), but doesn't show up in the thread itself here. At least i don't see it. Anyone happen to see the post (as well as this one i presume)? Just asking so i know wether it's a cache issue on my side, or our database problem just got worse.

Moose
2011-Jan-05, 09:37 PM
Basically, when posting, you can expect a "database error" message, but the post went through. Edits will do the same thing, but they'll also be successful.

You can expect to have to hunt for new postings, the only real indication is the "last post" field on the main page. There are some navigation issues in multi-page posts, but otherwise, just hold tight for now.

GeorgeLeRoyTirebiter
2011-Jan-05, 11:18 PM
If it ran with high-priority, the board would be blocked, almost like during a DDOS.

Considering how difficult to use the board is in its current state, that doesn't sound like a bad thing at all.

kleindoofy
2011-Jan-06, 12:17 AM
This post will go to the second page of this thread, so nobody will be able to read it for a while, but I just noticed that the post count for the board as a whole reads 1,800,00.

Although I can't really imagine it, I wonder if this might be some sort of limit above which the post numbers are not recorded correctly in the post number variable of the post table in the database.

It seems strange that we're having these troubles and the board's postcount just happens to be a round number like 1,800,000.

kleindoofy
2011-Jan-06, 12:20 AM
I can't edit my previous post, so I'll splurge and write a new one.

After posting the above post, I checked the board's postcount and it was 1,799,967, not 1,800,00 as before.

Aliens?

HenrikOlsen
2011-Jan-06, 01:21 AM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.
You have no idea about the load this board puts on a server, do you?
What backups do you take, how often?


I actually strongly suspect that this is caused by us being put on a server that can't take the load, in particular one with an underspecced or badly configured database server and it's not certain if Fraser can do anything about that with the current hosting situation.

kleindoofy
2011-Jan-06, 01:38 AM
ETA: replying to this thread seems to be useless right now, since page 2 of this thread can't be displayed.. Perhaps start a new thread, which can be merged later when things return to normal.
Ther's a trick to get around that.

Start a new post and read the quoted posts below the post composition window. Then just discard the post by hitting a button for something other than "submit," unless of course you have something to say.

kleindoofy
2011-Jan-06, 01:56 AM
Well, so much for the 1,800,000 post theory. After mysteriously losing 73 posts, the board worked its way back up and just went back to and past the 18,000,000 mark.


... I actually strongly suspect that this is caused by us being put on a server that can't take the load, in particular one with an underspecced or badly configured database server and it's not certain if Fraser can do anything about that with the current hosting situation.
Being "in the cloud" sounds kinda cool, but maybe it isn't when it comes down to it.


[re: tommac] ... You have no idea about the load this board puts on a server, do you? What backups do you take, how often?
Apart from other strong objections, having a member doing the hosting with access to everything and the finger on the button? No thank you.

kleindoofy
2011-Jan-06, 02:09 AM
... and the board just lost 20 posts again. :think:


... or badly configured database server ...
The problem is that the database error is selective and consistent in its selectivity.

It only seems to be screwing up the post number statistics. All the display errors fit in with that, including the page problems, etc.

If the database server was screwing up in general, the error wouldn't be so exacting.

kleindoofy
2011-Jan-06, 02:10 AM
With a copy of the vBulletin php code one might be able to reverse engineer what table in the database is causing this.

kleindoofy
2011-Jan-06, 03:07 AM
Is it possible that a table in the database got marked "read only" during the last backup?

That would explain things.

Obviousman
2011-Jan-06, 04:19 AM
http://www.abc.nl/blog/wp-content/uploads/2008/01/dontpanic.jpg

Obviousman
2011-Jan-06, 05:12 AM
Hmm - I made a post to this thread, got the normal DB error message, saw that it turns up in 'latest posts' but doesn't appear in the thread!

a1call
2011-Jan-06, 06:47 PM
IMHO the most likely cause is one of the PHP files losing privileges/permissions on the mysql database. However the correct response would be to restore a routine-backup usually made daily by hosts which justifies paying for the hosting service. I would be surprised if the baut host does not do that. This would have to be initiated by contacting the host's tech support.

kleindoofy
2011-Jan-08, 04:44 PM
Ok, things are back. At least new things.

Let's see what happens to a post on a (momentarily) invisible page ...

[edit] ok, a new post on a page that wasn't showing brings back the hidden pages, i.e. it rewrites the post number index.

Well, that will save reindexing the whole board. Very good.

Ok, so, please tell us: what was wrong?

Moose
2011-Jan-08, 04:58 PM
There was a problem with the device that goes ding when there's stuff. It went ding. Turns out there was stuff.

Seriously, an exact what-the-heck-happened will have to wait until Fraser can have a look at the logs. I can guess, but it wouldn't be anything more than a somewhat educated guess without getting my hands pretty dirty. The less technical answer is that our fearless admin found a "fix everything" button, and pressed it.

Buttercup
2011-Jan-08, 05:03 PM
I'm going to blame a passing flock of vampire bats. :mad:

(...an "inside joke" from another forum; never mind...)

ToSeek
2011-Jan-08, 05:33 PM
I found a "Repair Tables" function on the admin page and ran it several times. This is part of the report I got after running it the second time:


searchcore_text
Check
Warning: Table is marked as crashed and last repair failed
searchcore_text
Check
Warning: Size of indexfile is: 664851456 Should be: 1024
searchcore_text
Check
Warning: Size of datafile is: 1131270048 Should be: 1131156716
searchcore_text
Check
Error: Record-count is not ok; is 1620641 Should be: 0
searchcore_text
Check
Warning: Found 1624583 key parts. Should be: 0
searchcore_text
Check
Error: Corrupt
searchcore_text


All those went away after i ran the repair the third time.

Buttercup
2011-Jan-08, 05:44 PM
...yeah yeah.

You just don't want to admit MY theory is the correct one. :hand:

ToSeek
2011-Jan-08, 10:59 PM
Apologies for the hangup this afternoon. I told the forum to send out a mass email to tell everyone it was back online, and doing that apparently took everything the system had.

kleindoofy
2011-Jan-08, 11:02 PM
Just to get back to a clean start and to avoid a possible overflow source, I'd suggest clearing the board's cache. That can work wonders, depending on the source of error.

slang
2011-Jan-08, 11:04 PM
This is part of the report I got after running it the second time:

That "marked as crashed" jives with what seemed to be happening earlier that day.. extreme slowdown and then DB errors instead of BAUT loading. Thanks ToSeek.

a1call
2011-Jan-09, 01:01 AM
The issue is resolved.
Thank you ToSeek.

astromark
2011-Jan-09, 02:55 AM
Unsurpassed brilliance is rewarded with proclamations of pleasure... Great work 'TOSEEK'... above and beyond the call of duty.
Exceeding expectations.
and otherwise pleased as hell 'MY' favourite site is back...

Nowhere Man
2011-Jan-09, 03:05 AM
Things are almost back to normal. F'rinstance, the main page says that the latest post in the banned users thread was made on 06-January-2011 11:06 AM. The Forum Rules page says that the latest post was made on 03-January-2011 04:11 PM. No, refreshing the page does not change it. The thread says that the latest post was made on 06-January-2011 11:06 AM.

Other than that, which will probably take some analysis by Frasier, I'm very happy to have the board back up. Thanks, ToSeek! It didn't affect me much at all (twitch).

Edit to add: This post did show up as the latest entry on both the main page and the Feedback page.

Fred

ToSeek
2011-Jan-09, 03:10 AM
There still appears to be a problem with attachments, which is mostly an issue in the Astrophotography forum. There are relevant table errors, but running the repair function doesn't seem to help.

kleindoofy
2011-Jan-09, 03:16 AM
Things are almost back to normal. F'rinstance, the main page says that the latest post in the banned users thread was made on 06-January-2011 11:06 AM. The Forum Rules page says that the latest post was made on 03-January-2011 04:11 PM. No, refreshing the page does not change it. ...
That we be resolved the next time a post is made in that thread.

The new post number and the index of previous numbers, including the missing numbers, will be passed to the table used for building the on page latest post statistics, the thread pages, etc.

The table involved in that data transfer seems to be what was going wrong. It's interesting to note that the main page doesn't seem to read that table. It must have it's own.

kleindoofy
2011-Jan-09, 03:18 AM
There still appears to be a problem with attachments, which is mostly an issue in the Astrophotography forum. There are relevant table errors, but running the repair function doesn't seem to help.
Try editing one of the posts involved, e.g. just add a letter, and re-submit the post. Maybe that will resolve the table issue.

Maybe not.

Maybe the post has to be new.

Or, add a post to the thread and see what happens.

Swift
2011-Jan-09, 03:28 AM
Try editing one of the posts involved, e.g. just add a letter, and re-submit the post. Maybe that will resolve the table issue.

Maybe not.

Maybe the post has to be new.

Or, add a post to the thread and see what happens.
Editing didn't change anything. The problem seems to be both for old posts and threads (pre-trouble) and for a thread just created today.

Swift
2011-Jan-09, 03:33 AM
I still can not get to Page 12 of the Gold mining on the moon thread (http://www.bautforum.com/showthread.php?p=1838384&page=11#post1838384). Just threw in a test post and I can't get to it, even though it lists as the newest post in that thread.

kleindoofy
2011-Jan-09, 03:34 AM
Editing didn't change anything. The problem seems to be both for old posts and threads (pre-trouble) ...
Then the post number parameter that places a vector between the attachment and the post must be passed more permanently at post creation time.

The table for the page statistics seems to be temporary, i.e. it is rebuilt on the fly.


... and for a thread just created today.
Ouch. That should have been resolved.

The Astrophotography forum doesn't have too many posts, so maybe it would be worth it just to repost them later.

I love the pictures the people post there.

Ara Pacis
2011-Jan-09, 03:35 AM
I still can not get to Page 12 of the Gold mining on the moon thread (http://www.bautforum.com/showthread.php?p=1838384&page=11#post1838384). Just threw in a test post and I can't get to it, even though it lists as the newest post in that thread.

Me too, posted a quoted response in the same thread and it's not visible... unless I try the "go advanced" trick to view them.

Swift
2011-Jan-09, 03:37 AM
Me too, posted a quoted response in the same thread and it's not visible... unless I try the "go advanced" trick to view them.
The weird thing is that before the Fix I had this problem all over the place; now that is the only thread that still seems to have the problem

kleindoofy
2011-Jan-09, 03:37 AM
Sounds like the table fix was only temporary and the same problem is creeping up on us again.

There must be some kind of overflow somewhere, or something that is causing the certain table of the DB to choke.

kleindoofy
2011-Jan-09, 04:07 AM
Just for the people reading this who don't know what all this table etc. talk is about, here's an example taken from pbpBB, which runs basically the same as vBulletin.

This is the MySql-data base internal description of the table for the posts:


CREATE TABLE IF NOT EXISTS `phpbb_posts` (
`post_id` mediumint(8) unsigned NOT NULL auto_increment,
`topic_id` mediumint(8) unsigned NOT NULL default '0',
`forum_id` mediumint(8) unsigned NOT NULL default '0',
`poster_id` mediumint(8) unsigned NOT NULL default '0',
`icon_id` mediumint(8) unsigned NOT NULL default '0',
`poster_ip` varchar(40) collate utf8_bin NOT NULL default '',
`post_time` int(11) unsigned NOT NULL default '0',
`post_approved` tinyint(1) unsigned NOT NULL default '1',
`post_reported` tinyint(1) unsigned NOT NULL default '0',
`enable_bbcode` tinyint(1) unsigned NOT NULL default '1',
`enable_smilies` tinyint(1) unsigned NOT NULL default '1',
`enable_magic_url` tinyint(1) unsigned NOT NULL default '1',
`enable_sig` tinyint(1) unsigned NOT NULL default '1',
`post_username` varchar(255) collate utf8_bin NOT NULL default '',
`post_subject` varchar(100) character set utf8 collate utf8_unicode_ci NOT NULL default '',
`post_text` mediumtext collate utf8_bin NOT NULL,
`post_checksum` varchar(32) collate utf8_bin NOT NULL default '',
`post_attachment` tinyint(1) unsigned NOT NULL default '0',
`bbcode_bitfield` varchar(255) collate utf8_bin NOT NULL default '',
`bbcode_uid` varchar(8) collate utf8_bin NOT NULL default '',
`post_postcount` tinyint(1) unsigned NOT NULL default '1',
`post_edit_time` int(11) unsigned NOT NULL default '0',
`post_edit_reason` varchar(255) collate utf8_bin NOT NULL default '',
`post_edit_user` mediumint(8) unsigned NOT NULL default '0',
`post_edit_count` smallint(4) unsigned NOT NULL default '0',
`post_edit_locked` tinyint(1) unsigned NOT NULL default '0',
PRIMARY KEY (`post_id`),
KEY `forum_id` (`forum_id`),
KEY `topic_id` (`topic_id`),
KEY `poster_ip` (`poster_ip`),
KEY `poster_id` (`poster_id`),
KEY `post_approved` (`post_approved`),
KEY `tid_post_time` (`topic_id`,`post_time`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=9489 ;

It lists all the possible information connected to a single post, including the post number (`post_id`), the thread number (`topic_id`), which forum it belongs to (`forum_id`), etc. The field names are mostly self-explanatory.

The header of the table for the posts looks like this and describes the order in which the information is entered into a data line (= one post) of the table.


INSERT INTO `phpbb_posts` (`post_id`, `topic_id`, `forum_id`, `poster_id`, `icon_id`, `poster_ip`, `post_time`, `post_approved`, `post_reported`, `enable_bbcode`, `enable_smilies`, `enable_magic_url`, `enable_sig`, `post_username`, `post_subject`, `post_text`, `post_checksum`, `post_attachment`, `bbcode_bitfield`, `bbcode_uid`, `post_postcount`, `post_edit_time`, `post_edit_reason`, `post_edit_user`, `post_edit_count`, `post_edit_locked`) VALUES

The data for the posts themselves looks like this:


(1, 1, 23, 2, 0, 'xx.xxx.xxx.xx', 1206139214, 1, 0, 1, 1, 1, 1, '', 'Welcome to the forum!', 0x4e75...', 0, 'fw==', '3d0ce0r3', 1, 0, '', 0, 0, 0),

Following the header, one can see that this is the first post made on the forum, in the first thread, in the 23rd forum, by the second member to sign up, with no post icon, etc...

No, it's not magic, is it, just data.

The information for the forums, the attachments, PMs, the users, etc. is all stored in separate tables like the one shown above and are connected to one another via the IDs. The Vbulletin php code reads the information out of the tables as it needs it.

The pages we see on the screen don't exist anywhere in storage. They are created on the fly every time a user clicks on a forum or post name, the data being taken directly from the database table as shown above. Yes, every single time (except for cached information).

Van Rijn
2011-Jan-09, 04:15 AM
I still can not get to Page 12 of the Gold mining on the moon thread (http://www.bautforum.com/showthread.php?p=1838384&page=11#post1838384). Just threw in a test post and I can't get to it, even though it lists as the newest post in that thread.

You can't see your post in that thread? I'm seeing it:


This is a test post.

Gillianren
2011-Jan-09, 05:27 AM
Just for the people reading this who don't know what all this table etc. talk is about, here's an example taken from pbpBB, which runs basically the same as vBulletin.

Speaking as someone for whom quite a lot of this thread is "blah blah problem blah blah database blah blah," I can tell you that I still don't have any idea what all that code really means. However, I have faith in the fact that there are people who do, and that's really what matters. As long as the board gets and stays fixed, I don't have to understand what went wrong.

forrest noble
2011-Jan-09, 06:06 AM
In the ATM section the same problems remain when I logged on today. Although I did see one or two new postings in one thread, I could not make any of my own there. I can see the history of the unrecorded postings in my own and other threads when I go "advanced" that have not posted on the board. So essentially there still are no question-and-answer sessions or new postings for the most part in this forum as far as I can tell.

This thread seems to be O.K. somehow, so at least we can talk about it :)

forrest noble
2011-Jan-09, 06:37 AM
You can't see your post in that thread? I'm seeing it:

That's interesting. It could be that different viewers may see a different listing? When viewing my own ATM I can't see any new postings on the board for my thread but seven new ones can be seen below the "go advanced" as well as many repeat postings, most of them have been my own when first trying to post to the board when the problem seemingly first started.

forrest noble
2011-Jan-09, 07:26 AM
I still can not get to Page 12 of the Gold mining on the moon thread (http://www.bautforum.com/showthread.php?p=1838384&page=11#post1838384). Just threw in a test post and I can't get to it, even though it lists as the newest post in that thread.

It seems the problem is in the "Go Advanced" function. I think I figured out a "work around" procedure. I think I fixed the "gold mining on the moon" thread. At least it looks good to me now. I also fixed my own ATM thread. What needs to be done is that for all non-listed postings you need to do a new posting on the thread without using the "Go Advanced" function. When you do this all past hidden postings will then be listed once you go back or refresh the thread. This seems to work, so that if the "Go Advanced" function is used it needs to be followed by another posting that does not use the "Go Advanced" function, for it to be listed on the board. You can then see all postings. You can now see all postings for the "gold mining" thread. This "work around" procedure seemingly will work for all threads. This procedure has worked so far for all threads that I've tried it on that are missing postings. Of course there still may be other types of malfunctions that I am unaware of but this procedure does the trick for prior un-posted replies whether yours or others.

captain swoop
2011-Jan-09, 12:21 PM
try logging off and back on again.

That's what they keep telling me when our coll logging system goes wonky.

speedfreek
2011-Jan-09, 12:54 PM
Apologies for the hangup this afternoon. I told the forum to send out a mass email to tell everyone it was back online, and doing that apparently took everything the system had.

Just a note to say that I received no such email, and I have checked my junk mail folder, and have also checked that my registered email address is correct. I don't know if this is relevant to the situation, or not.

Solfe
2011-Jan-09, 01:58 PM
I didn't get an email either, but welcome back Bautforum! I have been sweating, shaking and irritable the whole time you were gone. :)

ABR.
2011-Jan-09, 05:26 PM
Thanks, ToSeek! While BAUT was down, I actually had to talk to my wife and play games with the kids. My six-year old is nine now! Who knew?

Swift
2011-Jan-10, 02:47 AM
This is the first time in three or four hours I could get on the board without a timeout error from my browser.

djellison
2011-Jan-10, 02:48 AM
Can someone please reiterate my volunteering to host this board on my server with Fraiser? I PMed him but he did not respond.

That would be the worst idea in the history of everything, ever. You would not be doing BAUT or it's members a favour.

kleindoofy
2011-Jan-10, 03:00 AM
This is the first time in three or four hours I could get on the board without a timeout error from my browser.
Judging by the posting time codes, it seem to have affected everybody.

Seeing that a few people posted that they had not received the mass email, maybe the board is still sending them out, but just in blocks, of a few hours a piece. Who knows?

I do ask myself if sending out >75,000 emails, mostly to people who only ever came here once, is worth it. But that's not mine to decide.

a1call
2011-Jan-10, 03:19 AM
One attachment in OTB that would not open for me the other day, opens just fine now.

NEOWatcher
2011-Jan-10, 01:08 PM
Seeing that a few people posted that they had not received the mass email, maybe the board is still sending them out, but just in blocks, of a few hours a piece. Who knows?
Possible, but I had that problem last month (http://www.bautforum.com/showthread.php/110468-Forgot-password-function) and still haven't recieved this one. I would think >22 hours is good enough for 75k emails (<1/sec)

I did work with a mod, and no resolution was found. We chalked it up to a cookie problem on my side or a temporary glitch.

a1call
2011-Jan-10, 01:45 PM
Apologies for the hangup this afternoon. I told the forum to send out a mass email to tell everyone it was back online, and doing that apparently took everything the system had.
Deja vu :)

http://www.youtube.com/watch?v=RN5aQSN7qo8&feature=youtube_gdata_player

Warren Platts
2011-Jan-10, 07:12 PM
I think maybe the possibility that bautforum.com is under attack should be considered. It's like there's some kind of trojan horse that keeps undermining every fresh repair. There's an awful phishing scheme going around facebook at about the same time. It got my account. Changed the password, that seems to have solved my personal problem, but the fact that the Gold Mining on The Moon thread seemed especially screwed up is quite a coincidence, or not--it's probably nothing...

kleindoofy
2011-Jan-10, 09:13 PM
The original problem (new posts not showing) is starting up again, at least partially.

Time to move the database to somewhere else.

Swift
2011-Jan-10, 09:49 PM
I noticed at least two threads (Tron: Legacy in Media and the French Fry conditment thread in OTB) that I can't get to the new page of (other than with the Go Advanced hack).

Substantia Innominata
2011-Jan-11, 12:01 AM
You're encountering problems "only" with specific thread pages? I couldn't even log in for quite a time and until just now?! It was still possible to browse the forum - while not logged in - and, as far as I tried, to read any desired thread. Yet, whenever I attempted to log in --> got the 404. :think: Now it works again, I was just wondering why, as it seems, all (..?) the other users were still (and at least) able to log in? :confused:

Ara Pacis
2011-Jan-11, 12:03 AM
New problem threads. I posted in the "Is Today Black Monday?" thread and it looked fine, my posts and all the other posts were visible. I saw a new post by Henrik Olsen just now and when I went to look, my reply, plus the two later replies, plus the two previous replies were all invisible, unless you used the "Go Advanced" workaround.

caveman1917
2011-Jan-11, 12:08 AM
Now it works again, I was just wondering why, as it seems, all (..?) the other users were still (and at least) able to log in? :confused:

But would it not be expected that only those users able to log in would leave posts in that time period, those unable to log in would by definition be unable to leave behind evidence of their failure to log in, so i'm just a bit surprised you take the existence of posts by some users as evidence that all of them were able to log in except you. I for once had a lot of trouble logging in the past couple of days, and when i did i got mostly thrown out again when switching to a different thread or something.

slang
2011-Jan-11, 12:09 AM
Time to move the database to somewhere else.

There's an offer on the table. :) IIRC vB offers a way to rebuild the indexes, as I recall Fraser had to use that on one of the previous moves. That might fix the display of forums and posts (IE what Swift mentioned). Quite time-consuming though.

kleindoofy
2011-Jan-11, 12:22 AM
... those unable to log in would by definition be unable to leave behind evidence of their failure to log in, ...
In fact, either BAUT has dropped in popularity by ca. 70% within the last 48 hours, or a good number of regular posters are having trouble logging in.


There's an offer on the table. ...
Yeah, right. Do the math.

But please note that I said "move the database," not "move BAUT."

MySQL databases are usually hosted on separate database servers anyway. Maybe that server is going south.

caveman1917
2011-Jan-11, 12:49 AM
In fact, either BAUT has dropped in popularity by ca. 70% within the last 48 hours, or a good number of regular posters are having trouble logging in.

Or a good number of regular posters have simply stopped trying with the recent malfunctions and are just waiting for the mass email before trying again, which seems to have its own problems in getting to everyone's inbox.

inflector
2011-Jan-11, 02:36 AM
Sounds like the table fix was only temporary and the same problem is creeping up on us again.

There must be some kind of overflow somewhere, or something that is causing the certain table of the DB to choke.

If the same pattern of problems is showing up again that potentially means the corruption is going to be harder to pin down and fix. In general, this is not a good sign in a database.

In a previous life, I wrote a lot of low-level database indexing code and SQL processing code in C++ for commercial database engines as well as plenty of application code using various databases, Oracle, MySQL, etc.

There are three types of potential corruption:

Disk corruption - Problems with the data in the tables because of problems with the disk. This will depend on whether or not there is a RAID system running the database files. If you have the right kind of RAID, this won't be a problem. If not, there could be physical corruptions on the disk. If they are in the indexes only this isn't such a big deal. If they are in the actual tables, you will lose some data and may need to go back to a backup, or you'll have continual problems.

Database corruption - This is corruption which the MySQL utilities ought to be able to detect. Usually the database has some redundancy built in so that it is possible to traverse tables and reconstruct indexes if an index is corrupted. Someone should run the appropriate check table table commands directly against the SQL database using phpMyAdmin or whatever is appropriate for the vBulletin setup you have. This will detect corruption in the database tables themselves and is probably repairable.

Logical corruption - This is problems in the structure of the links in the entries that are responsible for vBulletin's structure. Things like posts that are missing, or that link to the wrong thread, etc.. Hopefully, vBulletin's own check and repair facilities can handle logical corruptions. This could be caused by bugs in the php code for vBulletin.

I think I recall Fraser saying that the system was being moved to the Amazon cloud. If that's the case, then the problems could be coming from the Amazon code that is running the cloud providing the database services.

It may be that ToSeek or Fraser will have to restore back to a physical backup from before the corruption itself. This will be a big pain for them. I hope not.

This kind of problem is a major hassle to fix as it is generally best done with an extra server and lots of testing time. A database as large as BAUT's must be is not quick to check or repair. Over the cloud?

I'd be rethinking the whole decision at this point, if this is pinned down to a cloud issue.

Good luck everyone.

kleindoofy
2011-Jan-11, 03:28 AM
@inflector

Yup.

I'd doubt logical corruption. The vBulletin code is relatively stable and widely used.

Tele-diagnosis without any actual access holds about as much water as a sieve, but since an admin without server access could do a perliminary fix with vBulletin ACP means, I'd also rule out disk corruption. I doubt vBulletin could directly access RAID mirror data. The errors seem to be logical, not blurred.

This sounds like server inherent database corruption and/or cloud trouble. Hence my thought of cleaning and moving the database.

inflector
2011-Jan-11, 03:48 AM
The errors seem to be logical, not blurred.

This sounds like server inherent database corruption and/or cloud trouble. Hence my thought of cleaning and moving the database.

I haven't seen the errors myself so I'll have to take your word that they look like database errors.

Database errors could easily be coming from cloud trouble, something with the way they have the database servers on Amazon set up. It is mighty suspicious that these problems have all been coming so soon after the switch to Amazon's cloud. But unfortunately, I think there was a vBulletin upgrade too. So there are a few variables changed at the same time which makes it harder to assign blame.

Not the sort of problem you like to see.

Moving the database to isolate it makes great sense, even if only temporarily at larger cost than you'd like. I have no idea what sort of server setup is needed for BAUT but it must be pretty substantial.

I started and ran a phpBB board for a few years with about 1/20th the traffic of BAUT and it taxed a dedicated machine pretty heavily. PHP is easy to write but really inefficient since it ends up sending so many individual queries. It's very taxing on the database servers.

Ara Pacis
2011-Jan-11, 05:22 AM
I didn't log in, at least not recently. I assume a cookie is keeping me active. Any chance that there's a problem with cookies? Is it cookies that affect what a person is allowed to view (e.g. moderator forums) or is expected to view (new posts since last visit) or is it all serverside, or does a new release have different cookie data?

Grashtel
2011-Jan-11, 11:11 AM
It seems that at least part of the problem is that the board isn't always updating the count for the number of replies correctly (the number shown on the forum main view, the post count number does seem to be updating correctly though). For the thread "Space Habs and Gamma Ray Bursts (http://www.bautforum.com/showthread.php/110445-Space-Habs-and-Gamma-Ray-Bursts)" prior to my post it was displaying as only having had 30 replies despite there being 32 messages in the thread (the number of replies only counts actual replies not the original message), though after I had posted it showed the number of replies correctly at 32.

Warren Platts
2011-Jan-11, 12:59 PM
In fact, either BAUT has dropped in popularity by ca. 70% within the last 48 hours, or a good number of regular posters are having trouble logging in.

This would be consistent with a denial of service attack....

Warren Platts
2011-Jan-11, 01:05 PM
I didn't log in, at least not recently. I assume a cookie is keeping me active. Any chance that there's a problem with cookies? Is it cookies that affect what a person is allowed to view (e.g. moderator forums) or is expected to view (new posts since last visit) or is it all serverside, or does a new release have different cookie data?

I must say I that I thought it was odd that I was apparently still logged on this morning. I don't recall hitting the "Always Stay Logged In" button last night (I usually never do). Usually the system logs me off automatically after a period of nonactivity....

Substantia Innominata
2011-Jan-11, 02:17 PM
But would it not be expected that only those users able to log in would leave posts in that time period, those unable to log in would by definition be unable to leave behind evidence of their failure to log in, so i'm just a bit surprised you take the existence of posts by some users as evidence that all of them were able to log in except you...

Well, I couldn't tell anyway whether all of them were.. hence the question mark. Also I wasn't referring to potential posts being placed in the corresponding time period -- instead, my only clue was the Currently Active Users list, down at the bottom of the index page. ;) As we learn now: Not quite the cocksure indicator either, as to what is really going on, but I just noticed that there still were pretty much as many users listed (allegedly as 'currently online', i.e. regularly logged in) as always.. yet with only (?) my username missing. This is what puzzled me. So when I tried to log in, and failed, I could then instantly reload the BAUT index page, although logically retained guest status -- and no (not even residual-) sign whatsoever of my username in the down-bottom listing.


I must say I that I thought it was odd that I was apparently still logged on this morning. I don't recall hitting the "Always Stay Logged In" button last night (I usually never do). Usually the system logs me off automatically after a period of nonactivity....

Interesting, that sounds just like exactly the opposite trouble. I don't use auto-log-in either, but that shouldn't make a difference, since this clearly was not a cookie-problem, nor was it local in any other sense. Still, just to be sure I deleted mine -- to no avail.

As to that, though, everything works smoothly by now.

kleindoofy
2011-Jan-11, 02:45 PM
... Is it cookies that affect what a person is allowed to view (e.g. moderator forums) ...
No. As in the database example I posted above for posts, there is a database table for registered users. Each user-ID has a long row of variables which reflect various bit of information about that user, including the name, avatar, password, viewing and posting rights, bannings, member status, groups, etc.

Every single time a user makes a mouse click, php code is read which goes through a myriad of conditional queries, e.g. something along the lines of (in plain English) "if the moderator parameter for this user-ID in the user-ID database is marked 'mod,' then allow display factor 'mod,' if not, disallow display factor 'mod.'" Etc., etc. etc. Every imaginable feature of Bulletin is checked every time. (At least that's how it works in phpBB, and I'm pretty sure both systems are similar in that respect.)

This is part of what inflector was referring to above. Tons of php code that roars through the server's cpu, bogging it down. In fact, if I'm not mistaken, the majority of cpu traffic caused by the forum results from these queries, 80% of which are always answered "no" and will never be answered with "yes." But they have to be asked anyway.

This shouldn't be confused with external traffic (bandwidth). It all happens in the server itself.


... or is expected to view (new posts since last visit) or is it all serverside ...
I'm pretty sure that's all sever side. If not, any script kiddy could make himself an admin by hacking his cookie.

NEOWatcher
2011-Jan-11, 02:59 PM
I'm pretty sure that's all sever side.
I'm also fairly sure something's been severed. http://www.cosgan.de/images/smilie/frech/s045.GIF

Buttercup
2011-Jan-11, 09:39 PM
This surely has already been answered/discussed, but I don't read every thread.

For hours today I couldn't access this board. Kept getting an error. Every computer application I use and all other regular sites have been working continually and flawlessly.

Apparently my inability to access BAUT was a database error within it?

jlhredshift
2011-Jan-11, 09:44 PM
This surely has already been answered/discussed, but I don't read every thread.

For hours today I couldn't access this board. Kept getting an error. Every computer application I use and all other regular sites have been working continually and flawlessly.

Apparently my inability to access BAUT was a database error within it?

Me too, I just now have become un-severed.

inflector
2011-Jan-11, 09:58 PM
Same here. Couldn't get into BAUT for hours and gave up trying until I got an email that this thread had been replied to.

kleindoofy
2011-Jan-11, 11:28 PM
Strange, without logging in I take a glance or two at the board at various times during the day today. No problem whatsoever.

That sounds like a regional, IP, or port problem. I have no idea how "the cloud" is hooked up to the rest of the world, but disturbances seem to be selective.

Considering what has been happening, perhaps we should rename "the cloud" "The Fog." :wall:

Trebuchet
2011-Jan-11, 11:35 PM
I've also been unable to get into BAUT. Whenever that that happens, I've been checking Universe Today and can't get into that either. Not surprising, since I'm pretty sure they're on the same server. No problems with any other sites.

Gillianren
2011-Jan-11, 11:58 PM
I haven't been checking regularly lately, as my computer's in the shop, but it's been down every time I have checked the last few days.

Ara Pacis
2011-Jan-12, 05:53 AM
It's working for me now, but it was down earlier and gave me a VB database error, but I didn't keep checking to see when I was able to see it again.

Jens
2011-Jan-12, 06:06 AM
I haven't been checking regularly lately, as my computer's in the shop...

It's pretty noteworthy that you manage to post without your computer. Telekinesis?

HenrikOlsen
2011-Jan-12, 09:20 AM
Library?

Tensor
2011-Jan-12, 04:17 PM
Interestingly, I could not read the forum using my computer, however, I could read the forum, using my phone. However, for quite a while I couldn't sign in on either of them.

Gillianren
2011-Jan-12, 05:41 PM
It's pretty noteworthy that you manage to post without your computer. Telekinesis?


Library?

You're both wrong! I am able to borrow Graham's occasionally if he isn't home. If he is, I can only use it for things he deems important, and I must admit that checking here isn't on that list.

Jim
2011-Jan-12, 08:06 PM
You're both wrong! I am able to borrow Graham's occasionally if he isn't home. If he is, I can only use it for things he deems important, and I must admit that checking here isn't on that list.

Would you like for a few of us to come over to your house and have a quiet talk with Graham about his completely unreasonable stance?

Gillianren
2011-Jan-12, 08:42 PM
If it's not out of your way.

Tobin Dax
2011-Jan-13, 12:54 AM
Would you like for a few of us to come over to your house and have a quiet talk with Graham about his completely unreasonable stance?

Jim's a bit closer than I am, but the trip could be made. This is something that should be seen to.

ABR.
2011-Jan-13, 03:38 AM
Sounds like you're forming a Brute Squad!

Gillianren
2011-Jan-13, 03:47 AM
I can get pins declaring them as such, too! My ren faire boss casts 'em out of pewter.

ABR.
2011-Jan-13, 04:05 AM
Crud. You are on line. I swear I didn't actually make that egregious grammatical error...it was a board malfunction! Yeah, that's it.

Luckmeister
2011-Jan-14, 06:54 AM
40 hours ago, I was laughing at how people seemed to be going into panic mode when our beloved BAUT malfunctioned...... then my power went out for a few seconds. When it came back on, a voltage spike killed my computer motherboard and power supply...... I stopped laughing. I quickly discovered how addicted I really am to BAUT and the rest of my computer usage.

I just got a replacement computer going and all my multimedia stuff installed. I was writing this post and had just typed the words "then my power went out" and...... you guessed it, my power went out again for about 3 seconds, came back on for about 3 seconds, then went out again. I unplugged the power cord as quick as I could hoping I wasn't right back to square one. This time no fatal spike zapped the system.

This happened with another computer 4 years ago and that time it took out my hard drive as well. It was like I had just experienced a lobotomy. I learned the importance of backing up my drive. Both times, the system was powered through a consumer-grade surge protector which failed to protect.

Mike

Tobin Dax
2011-Jan-15, 03:34 AM
Sounds like you're forming a Brute Squad!

I am the Brute Squad!

(Late reply, I know, but it's been a busy week.)

ABR.
2011-Jan-15, 04:04 AM
I am the Brute Squad!

(Late reply, I know, but it's been a busy week.)

Thank you, Tobin Dax! I knew I could count on you for not leaving me out to dry. And for the record, I may have to journey northward this year to visit a certain Ren Faire -- I want a Brute Squad pin!

Tobin Dax
2011-Jan-15, 04:33 AM
I want one, too. I've been trying to convince myself not to take another trip back home to the NW until next Christmas, but this is one more reason head back that way.

Trebuchet
2011-Jan-15, 05:05 AM
Cool! We can have a BAUT reunion at the Faire! I went last year; I hope it's not quite as warm next time.

Nowhere Man
2011-Jan-15, 02:01 PM
Luckmeister, UPSes are your friends. They will even out power glitches like that, and with any luck will sacrifice themselves to save your computer.

Getting back to board troubles, the ATM thread Stephan's Quintet (http://www.bautforum.com/showthread.php/110135-Stephan-s-Quintet) seems to be hosed. The board reports that it has 26 posts, but open it and 30 appear (one page full). I tried to add a test post to see if I could kick it into behaving (fulling intending to delete it) but can't even see it, or the post before it, because they're on page 2. (The test post is gone now; I bumped my page size to 40 so I could see it and kill it.)

Fred

Gillianren
2011-Jan-15, 06:26 PM
Cool! We can have a BAUT reunion at the Faire! I went last year; I hope it's not quite as warm next time.

You got to go home. However, since one of our neighbours retired, we'll be getting her old spot this year. It's just across the Row, but it's on grass, which will be much nicer. If it can manage not to break maybe 85 degrees, that would be nicer still.

Hornblower
2011-Jan-20, 12:55 AM
I think we are getting bugs again. According to the Last Post column on the main page and the Science and Tech page, there is a new post from Swift about the Kamchatka volcanoes, but when I open the thread I find nothing since last week.

kleindoofy
2011-Jan-20, 01:03 AM
... there is a new post from Swift about the Kamchatka volcanoes ...
As the "Go Advanced" work around shows, there are actually two.


... According to the Last Post column on the main page and the Science and Tech page ...
Interestingly enough, the last time the Last Post column on the main page was always renewed while the Last Post columns in the individual forums were not.

What's different this time?

Tensor
2011-Jan-20, 03:40 PM
I think the forum is getting errors again. The main page is showing that tusenfem posted in the Banned Posters thread at 11:06 GMT this morning. If you click on the last post icon (or go into the thread manually) there is no post by tusenfem. Unfortunately, since that thread is closed, the workaround won't work for that particular thread.

CJSF
2011-Jan-20, 05:42 PM
I've been bouncing around the fora this morning and early afternoon and was noticing that new posts seemed to have stopped. Then I looked at the current users area on the main page and saw only three members (and 21 guests) were on! Is THAT a database error, or is something else happening?

CJSF

Fazor
2011-Jan-20, 06:14 PM
I've been bouncing around the fora this morning and early afternoon and was noticing that new posts seemed to have stopped. Then I looked at the current users area on the main page and saw only three members (and 21 guests) were on! Is THAT a database error, or is something else happening?

CJSF

BAUT was unaccessable for me around that time. My guess is the same was true for almost everyone else. When I checked a few minutes ago, it was only showing about 9 of us (and some guests) online, so probably still going on. *shrug*

Trebuchet
2011-Jan-21, 12:36 AM
Yup, it was down for me this morning (PST) as well.

ToSeek
2011-Jan-24, 07:57 PM
Fraser took the board down and did a bunch of work on it - see his post in the announcements forum.

Solfe
2011-Jan-25, 05:12 AM
I have this odd problem - if I search for a phrase or word, I see the posts in question. I am used to seeing the title of the thread. Did I change a setting on my profile or is that one of the issues?

I also seem to have a problem editing posts. Every time I try, I get the delete this post screen. If I back up and refresh, it works correctly. If this doesn't stop I will have to start proofreading and using proper spelling/grammar every time I post. :)

astromark
2011-Jan-25, 08:56 AM
As the Late Douglas Adams said... DON'T PANIC !... and promptly while attempting to kick the cat slipped on the polished floor and busted a arm... the cat flicked its tail and smiled.... The computer had gone down.
The lesson learned is this. I keep a book of some interest to me near my computer. All is not lost. Yet.