Update on the boards- Technical and Non Technical versions included

[becka]

Alex - Do you have logging turned on?

 

[Fizzgig]

More drives is usually better than faster drives, statistically.

May I ask how often you burn through drives, that you need to go 0+1 on a small array?

Thank you for the explanation! I'm still wondering about your architecture on a pure curiosity basis (I do middleware engineering standards for a big bank). I deal with WLS, WAS, and Apache, running on everything concievable.

You guys have a wonderfully user-friendly set-up here, and I appreciate the time you put in making it so! Thank you!

Brandie

 

[Fizzgig]

Can we come up with a non-technical explanation of RAID? I'll start:

1) The big crash we had in 1999 can't happen again, because if there is a drive failure, the data is can be recovered from a mirror of that drive.

....

The data is written onto multiple drives; so if one drive fails, the data is immediately available somewhere else. No downtime. No data loss, unless something -really- bad happens to the drives.

Brandie

 

[Rafiki Rafiki Rafiki]

we have 8 gigs of ram in the database server, there no paging and those numbers look good
Galahad- thats the db server, there's also 2 webservers and a load balancer

Stop talking like this. You're getting me so excited.

 

[Carl]

Gosh, that DOES explain a lot, and it should provide a very clear, up-to-date, indication to folks who want the old search back precisely why it is infeasible.

Are you folks are looking into creating a new, indexed relation table to serve as a JOIN target for that massive IN clause? (That's not a recommendation: Database design is not my forte. I'm asking solely for personal edification in that regard, i.e., what would people who do know databases do if faced with the problem you're facing.)

And during that time one of the server worker threads was blocked, right? Get enough of them, and the whole application would freeze up, which is what actually happened.

AVS Forums does indeed put thread length limits in place. Quite frankly, I prefer them regardless of the fact that they also provide performance benefits. YMMV.

I agree. How many times do we need to see "Steve Irwin, Crocodile Hunter Dies" come to page one? Or the one about Disers trying to conceive?

 

[C.Ann]

Non technical summary

1 thing was broke we fixed it
2 things are still broke we are working on fixing them

Is that better?

-----------------------------

Now you're speaking in a lanuguage I can understand..

Thanks for the update..

 

[tiggger1]

Hubby is a computer nut and stuff like this is what he does and I showed his this thread. He asked how much RAM are you using? He said board performance has more to do with ram than drive space? He is going to read the thread better when he gets to work.

 

[JR6ooo4]

Also, the bus speed must be proportionaly escalated with the HD RPM improvement to be able to adequately appreciate the RAM speed/quantity values.

If you still suspect bugs then send in the snake!

I just made all that up... except the part about the snake.
I know nothing about servers, except how to tip them.
Mikeeee

 

[Fizzgig]

Hubby is a computer nut and stuff like this is what he does and I showed his this thread. He asked how much RAM are you using? He said board performance has more to do with ram than drive space? He is going to read the thread better when he gets to work.

Did he say drive speed or drive space?

Either way, when the request to pull up a thread with 125,000 posts is done, I'll lay money that the available memory is filled. Your users try too many of those, and no "normal" computer or database can handle that sudden load.

Mikeee: That poor dead snake. I once had a ticket on the wall where one of my techs had to pull someone's lipstick out of the printer. "Yeah, I think we found the problem, ma'am...."

Brandie

 

[becka]

Either way, when the request to pull up a thread with 125,000 posts is done, I'll lay money that the available memory is filled. Your users try too many of those, and no "normal" computer or database can handle that sudden load.

With some tuning and certain database designs you may be able to handle it. The problem is you don't want to tune all your queries and design your database just to handle those large threads and queries because the vast majority of your db calls are for small results. You typically will tune for the average request. Tuning for those large threads may increase there performance but hurt the performance of the smaller thread calls and ulitmately hurt the overall performance of the db.

 

[WebmasterAlex]

I am reading this and will post some answers later but posting long answers from my iphone while at jury duty is not fun

 

[agnes!]

I agree. How many times do we need to see "Steve Irwin, Crocodile Hunter Dies" come to page one? Or the one about Disers trying to conceive?

People are still interested in those topics, so they post to them. Isn't that how threads end up on Page One? Maybe you or I might not "need" to see that thread, but somebody else sure does, or the thread would just drift off to wherever old threads go to die.

agnes!

 

[Fitswimmer]

Non technical summary

1 thing was broke we fixed it
2 things are still broke we are working on fixing them

Is that better?

Now that's an explanation that I can understand!!

 

[Fizzgig]

With some tuning and certain database designs you may be able to handle it. The problem is you don't want to tune all your queries and design your database just to handle those large threads and queries because the vast majority of your db calls are for small results. You typically will tune for the average request. Tuning for those large threads may increase their performance but hurt the performance of the smaller thread calls and ultimately hurt the overall performance of the db.

Right--that's what I meant by "normal."

Most of the traffic here is small density, low impact (well, except for any posts from GRUMPY PIRATE ). We've already seen the tuning for that is causing WM_Alex's hair to curl... But if he tuned to handling LARGE, mama-jamma threads, would it be efficient for the business that sponsors this site? You're talking development work on the database/website or big rig hardware.

And while I'm a big proponent of PHP and mySQL... I like my big database software, too. I agree, don't use a hammer for everything--not everything is a nail... but... Alex, how are the other big vBulletin websites managing to balance out extremely chatty people with lots of small graphics with a lot more small density traffic? Is it possible to change the db with pointers to increment/decrement from the front and back of a thread? Or is that already in place? Does the software have to read the whole thread into memory in order to go to post XYZ?

Thanks for helping me out with my curiosity problem. Good luck in jury duty with your crazy iPhone. DH just got the Verizon equivalent and that sucker is -attached- to him.

Brandie

 

[Fizzgig]

Oh, and Alex, I'll trade you my dentist visit for your jury duty... PLEASE? I'm scared of needles.

*grin*

 

[lovetoscrap]

I didn't attempt the Riddle of the Day because it involved math. But it is looking easier than trying to figure this out!

I will stick with the it broke and we fixed it theory. (pretty pink dunce cap)

But I have to agree that it seems that if the chat threads are a problem then they could be limited. You are providing a service to us by allowing us to have them. And it would be fun coming up with new names and posting before the end comes!

 

[SplshMtn99]

DH is the systems admin for MA National Guard. So for chuckles, I sent him this thread to see if he understand Alex. I knew he would. This is the reply I received. I don't understand so I just copied & pasted. lol

Yes, I understand. Ram should be added.
In addition, suggest using the 2 15 K SAS drives in Raid 0+1 configuration for the OS and using Raid 5 configuration on the remaining drives for the MYSQL Database.

He's never spoken like that me. Good thing. I would just stare blankly at him.

 

[momof2inPA]

Can't you just roll-over a thread and re-title it automatically after 1000 posts; eg, it becomes PMS-2, instead of PMS?

 

[Fizzgig]

If the RAM isn't paging, adding more memory won't solve the problem, Alex? What you have isn't getting strained.

Alex, are you seeing any spikes of memory usage that rapidly go away, or consistent low usage and locked processes?

Locked processes + low memory usage could mean an app error like the SQL statement issue Alex brought up on the front page.

Locked processes + high memory usage (which is not what was described) could be possibly -helped- by more RAM or by refined code. <--Notice I didn't say fixing your code there.

If the vBulletin code grabs a lot of data at once via database (SQL) statements that pull all the posts on a 125K post thread, that would fill up RAM and maybe cause paging (if there's enough to overflow). If the machine lets go of the data properly, no problem. If the machine does not let go of all of the data, just most of it, then forgets to let go of the rest, that becomes an issue fairly quickly on a high-traffic website. The amount of memory used keeps increasing to the point the machine falls down and goes boom. (Like that technical explanation there?? )

Oh, and while I have made reference to the clique threads here, my DISMeets threads are pretty stinkin' huge, too. There's 3+ DISMeets threads for every week (I'm counting just DCL threads on one Magic and two Wonders a week, but you could add fridge swaps, PhotoPass shares, coupon trains, freebie threads, etc).

 

[WebmasterAlex]

Gosh, that DOES explain a lot, and it should provide a very clear, up-to-date, indication to folks who want the old search back precisely why it is infeasible.

Are you folks are looking into creating a new, indexed relation table to serve as a JOIN target for that massive IN clause? (That's not a recommendation: Database design is not my forte. I'm asking solely for personal edification in that regard, i.e., what would people who do know databases do if faced with the problem you're facing.)

And during that time one of the server worker threads was blocked, right? Get enough of them, and the whole application would freeze up, which is what actually happened.

AVS Forums does indeed put thread length limits in place. Quite frankly, I prefer them regardless of the fact that they also provide performance benefits. YMMV.

There's not much you can put in place to help a query like that, it's why search is turned off!
As for thread limits the people that post on those threads like them and they are part of the fabric of this community. Pete feels very strongly and I agree that we would only put limits in place as a last resort, we will exhaust our technical options first.

Unless I'm mistaken, thread limits would LOCK threads after a certain number of posts, not clean threads out. I think cleaning threads out is a bad thing, since important information, that people might want to refer back to, would be lost.

Correct we would NEVER get rid of them only roll them over into new threads

Alex - Do you have logging turned on?

Good lord no! If I did it would be on another set of drives

People are still interested in those topics, so they post to them. Isn't that how threads end up on Page One? Maybe you or I might not "need" to see that thread, but somebody else sure does, or the thread would just drift off to wherever old threads go to die.

agnes!

Exactly

If the RAM isn't paging, adding more memory won't solve the problem, Alex? What you have isn't getting strained.

Alex, are you seeing any spikes of memory usage that rapidly go away, or consistent low usage and locked processes?

Locked processes + low memory usage could mean an app error like the SQL statement issue Alex brought up on the front page.

Locked processes + high memory usage (which is not what was described) could be possibly -helped- by more RAM or by refined code. <--Notice I didn't say fixing your code there.

If the vBulletin code grabs a lot of data at once via database (SQL) statements that pull all the posts on a 125K post thread, that would fill up RAM and maybe cause paging (if there's enough to overflow). If the machine lets go of the data properly, no problem. If the machine does not let go of all of the data, just most of it, then forgets to let go of the rest, that becomes an issue fairly quickly on a high-traffic website. The amount of memory used keeps increasing to the point the machine falls down and goes boom. (Like that technical explanation there?? )

Oh, and while I have made reference to the clique threads here, my DISMeets threads are pretty stinkin' huge, too. There's 3+ DISMeets threads for every week (I'm counting just DCL threads on one Magic and two Wonders a week, but you could add fridge swaps, PhotoPass shares, coupon trains, freebie threads, etc).

The memory is fine, the reason to add memory if your drives are strained is because paging can sometimes look like a drive problem That's not the case here.

There's really a lot more to this than even what I have said here. Standard vbulletin uses MyIsam tables which perform a table lock during an insert. One thing that would help a lot is to convert some tables to Innodb. The next version of Vbulletin is rumored to have Postgresql support so maybe we will go that route. We may look at having some modifications done. I HATE to do that because every upgrade then becomes so much harder.
We may add another DB server at some point but the problem there is the replication reduces efficiency, from what I have seen another DB server only gets you about a 30% increase.
We have the 33rd largest discussion board on the internet according to big-boards.com and the 12th largest Vbulletin board and we are bound to have some growing pains once in a while. I know at one point today we had well over 3000 people on line though something is working