Help with some ideas?

[Hooz]

I was just curious what you guys think are the most important types of benchmarks that we should run when reviewing a motherboard or system.

We will have the opportunity in the very near future to play with some chipsets/boards/CPUs that noone has ever seen before... How should we approach it? What are you really concerned about?

Obviously once you review a few boards based on the same processor and/or chipset, the numbers themselves become moot, but how should we deal with a whole new PLATFORM?

[EM]

I vote for synthetics because I have those. I am not about to purchase Photoshop just to compare my system to a newer platform.

My $.02

Ed

[spaceley]

ERMMMM..... I think you may have forgotten to put the...
"LET SPACELEY TEST IT !..." category in the poll. NO NO its ok ! I know that its a simple oversight. We all know that if you wanna find out how something really works you gotta let SPACELEY SPROCKETS take it for a test flight. :D
great site but it keeps me from my work, so cut it out!;)

[Jason_Dickey]

I think real application benchmarks are probably the best, but many of the real applications are not CPU/Memory bound anymore.

I won't speak of 3DSM, but I know Photoshop is usually more bound by disk speed than Memory/CPU performance.

Example: I have often edited a whoppingly big blueprint in Photoshop, and the scratch files Photoshop generates while editing the thing have easily been over 4GB in size. So, benchmarking in Photoshop with things that actually take some time to process (like 2-3 min. to open the thing), are not stressing the CPU/Memory, but rather the disk implementation. If you work with a file small enough to fit in RAM only, the speed in Photoshop is near instant anyway, and differences between systems are likely to be insignificant. (Who cares if it takes 1.2 sec vs. 1.5 sec to rotate an image?)

As a more practical suggestion, I would like to see a combo of rendering-type benchmarks, as well as some synthetic benchmarks.

Am I safe to assume that most people won't be gaming on these systems anyway?

[Hooz]

You are safe in assuming that people will not be buying these machines to play Quake. Unless of course they just have money to throw around ;)

[Adam Brackney]

Are you guys trying to cut my gaming ass out of this:)

[Skute]

do all of them:)

--Scott

[Defiler]

This is a rough question, mostly because you've bundled motherboards and systems together. Motherboards typically perform within one or two percent of each other, so features and stability are more important. Especially stability, and overclocked stability.. Now, if they perform 10% slower than the other motherboards in the category, something is wrong, and we'd like to know about it.
For systems, I'd say application benchmarks.

[BuyALambo]

I'm all about the synthetic benches. It seems that everybody runs those. Who's to say that people will not play Quake on high end dual boxes?.....

There are people out there that build/collect computers as a hobby. Heck I know the biggest stress tht my box has been under lately is MIRC, Outlook Express, and Internet Explorer at the same time!

I know, I should go take my medication now :(

[grindles]

Well, I would like to see memory benchmarks, as I have never had good memory subsystems in any of my machines.

I would also like you to test some of the common add-ons I might stick in. Could you try them out with some stuff like win-tv cards... I know, I know, who is gonna put a TV card in a server? Well I am hoping to put a server motherboard in my all round PC, as I am sure many will do... SBlives seem to dislike all duals so far, how do the new ones do? DVD decoder cards (not always necissary).

Also, details of what the BIOS can actually do... there are so many undocumented features in so many boards... some of which are good features - like apparently the tyan thunder 2500 can adjust the PIII multiplier!

Also, a real test of what CPUs boards can actually take, like single celeron 2s, and such things, as all these questions come up in the forum, as I am sure things like "will durons work in a 760MP board, I mean REALLY work?" will be asked in a few months.

Functionality of onboard components is also important, like- if it has built in video, can this be run at the same time as an AGP card? (could be useful for people with two screens). How good is the built on scsi? Does the IDE RAID work? This is assuming these board(s) have those features.

Oh, and last, but most certainly the most important, how easy is it to crash... try having a TV card, a DVD playing, maybe run seti in the background, and surf the web, all at once, for a month or two, without restarting. Stuff like that is where crashes come from.

[BlackWolf]

What about the numbers of RC5 packets they can crunch!

P.S. How do I change the SMP Newbie bit to something else?

[Black_Baron]

Maybe I'm old fashioned, but I just want to know things like stability, compatability, ease of use, stuff like that. I'd also like to have some sort of idea of what its like to bea actually sitting in front of the system wailing away at those webpages I so carelessly neglect (you don't kill a geforce by playing games).
For me, computers are about making things easier for me, how easy is it for me to be installing office whilst outlook and IE are running, with the possibility for MSSQL, coldfusion, IIS, and a Q3 server is running. Granted, that's not the ideal server config, but that's how I work, and I haven't seen anything that'll tell me how well it performs (my dual 933s did 2175 GFLOPS, but that didn't give me any clue that the thing would bitch while I rip a CD and listen to winamp).

Hidden 'features' like this are what I'd like to know about, I don't really care how well it frags, I care how well it runs on a day-to-day basis.

[Mouse_Launcher]

I'm with Skute on this one - do the lot, as different people want different things. Maybe a focus on stability and compatibility, but with the rest thrown in as well for us benchmark junkies.

[krylon]

I am most concerned about stability of a machine. You need to put the thing through as much stress in the least amount of time and keep at it. You need to see how long you can keep from rebooting the machine and how the system's performance degrades over time. That is what makes a real machine.

[EM]

Originally posted by BlackWolf
P.S. How do I change the SMP Newbie bit to something else?

Edit your user profile, under custom text or something like that.

Ed

[Kampf]

Stability is one of the most vital statistics we measure about any product, whether it's a motherboard, video card, optical drive or magnetic storage.

From the time i got my VP6, till i finished the review was nearly a month. I would have liked to test it longer in many cases, but the board was doing an incredible job and i was itching to get the review out. Besides, it just felt done.

Synthetic benchmarks are preffered. They're usually more specific to a subsystem and can be more easily influenced by changing a component. Using application benchmarks for total system performance is also good, but it had to be taken that way as well.

I think i'm going to keep doing what i've been doing.

[aryan]

RC5 crushing for float performance.

q3a highest quality@1152 or 3dmark2001 for dx8 performance

and some winbench or something for int/hd performance



life is simple....

[Donald2B]

I agree with just about everything people have said thus far. But for me and what I do, stability is the most important thing. Without a stable machine I can't get my work done, which in turn means that I don't get paid. So if I am using 3D Studio Max 4, Lightwave 6, or Maya 3 I need to have the most stable machine I can get. Nothing is worse than having a animation that takes several days to render. I don't want to sit around and babysit my machine because I never know when the thing might crash, overheat or whatever. Don't get me wrong, performance is really nice, but stablity is the key for me. :D

My 2 cents worth.

[Edwardo]

Kampf-

I really liked that last review you posted. It had everything a good review should have. I liked it how you put the Q3 and MDK2 performance on there. I know I play a lot of games on my machine. Seriously though....gaming is mostly an afterthought with these high dollar systems.


I think everyone has hit the nail on the head with what they desire out of these new boards you are reviewing. STABILITY.

I am assuming one of them is an AMD solution. In lieu of the aformetioned point I think it is crucial that we have some serious stability testing. I mean fricking days of torture tests. AMD is new in this arena of dual systems and I think it is essential to create a review in which someone will feel very confident whether or not that solution is right for them.

I also put a vote in for application benchmarks(3dStudio). I think a lot of people in here are planning on building rendering machines. I think the server/networking benches are less important because if I were needing some sort of mission critical server...I would probably read reviews on vendor full systems(Dell, Compaq, etc...)

[TonySquared]

What I really like to see in benchmarks is clear comparisons of Apples to Apples with top CPU chips. A VP6 board benchmark with 800EBs is no good any more because I wouldn't buy anything less than 1GHz PIII chips.

The main thing I want to see are chipset/memory comparisons. I'm having a hard time figuring out if DDR motherboards are getting faster or not.

[LinkerX]

Honestly I do all types of shit on my home computer ranging from video editing to 3d modeling and the only way to really show what a computer can do is to have a variety of tests. That means you need to have everything from gaming to synthetic etc. If your going to take the time to review a system then you better take the time to test all areas of performance. Also you have to figure since you are a site that is primarily about 2cpu systems then you better be using the fastest chips out in your reviews because I doubt many people that are interested in your site don

[awmsc000]

Well, here is a suggestion.

After reading a number of responses, it is clear that many of us are VERY concerned with stability and compatibility. Fine.

My suggestion: Let's try to QUANTIFY stability...
occasionally, we hear that the average amount of crashes for such and such a test run is, say, 6.

Then we hear that such and such a board only crashed "2 times", and that "this is extraordinary."

How about...

Stability score: 2/6, or some such easily identifiable marks.

Also, perhaps we make sure that all testing is done with a single standard for memory type?

I remember at one time taking two motherboards from the same run, and two (AMD 486) chips from the same run, and finding that when chip A was on board A, the board wouldn't even post. But when chip A was on board B, it would. Chip B on Board A also worked. It just pointed out that to really get to the meat of the matter, you had to eliminate variables. So how about deciding on a SINGLE brand/style/speed of memory to use for these tests, and a SINGLE brand/style/speed of SCSI HBA, etc. Just a suggestion.

[JackieO]

Wow, this has to be the most informative thread I have read in ages. I feel the pain of having to approach new platforms and how to test them. I know this is the testing set I devised.

3DSM (two tests, one wireframe one high polygon, both software rendered)
Sysmark 2000 (with the rundown of individual apps)/2001, CC winstone, business winstone
CLI SMP Bench
PVMPOV (Rendering with SMP)
Spec viewperf
Cachemem
Flask encoding (SMP)
Sciencemark
Quake3 (we needed one game!)
SETI times with one instance and two instances running (for SMP) with a standard LC WU.

The problem is there is really NO quantifiable benchmark that will accurately show off the pure power of say, Dual Palamino's or Fosters... know what I mean? The first problem is lack of benchs that are even multi-threaded, the second is finding ones that are stressfull enough on the system to show the advantages over say, Dual PIII's...

Of course stability is the most important thing. But writing an article saying "it was stable" doesn't cut it. We need to show benchmarks :)

I like the idea of a stability score though, but stability is subjective. I am very harsh on even one lockup, some people don't mind a few... see what I mean?

[y-dawg]

first off, i dont have a dualie sys yet (almost did by accident. bout a abit mobo, thought it was uni, but it was smp, but only supported up to 500 mhz celly and i had just bought a brand spanking new 566...oh well) but im hoping to get a new 760mp with athlon-c/4 and maybe ddr.
btw, this WOULD be for gaming (and not quake, but slightly better quality games :p). but i do a lil of everything, including video editing, and 3d stuff.
so of course i would like to know how this is gonna do on games. seeing 3dmark scores is nice. i dont really put much value in whether i can get 190 or 200 fps in Q3, theres better new games out there. ppl need to move on. seeing sandra scores is nice. im not sure about sysmark, and if there will even be a new videomark...
for now specifically, id like to sorta have a feel for what areas of machines will have the greatest bottleneck, and what areas are fine. DDR is a big one. i loved the idea at first. but it seems, its not really doing much for performance gains right now. maybe there is overhead, and in the near future this will change...
id also like to see the comparison of single cput systems with their counterpart dualie systems. as well as what gives the best price to performance ratio. for example, would more ram be better than a second raid hd on system X? what about system Y?
thanks for wanting to know what readers want. and i cant wait to get my first smp machine.

[Hooz]

I have been keeping an eye on this thread all day and a lot of you have had some really good suggestions. Suggestions that we will take into serious consideration... But there are a few things that we run up against as harware reviewers that make things a little tougher than you might think.

While I agree that standardization is paramount when comparing systems and components, there are a few problems. We are still a relatively small site (but getting bigger :)) and the guys who work on the site are spread far and wide. It is almost impossible to make sure that everyone has identical components to use in their testing. Now if we were a large site with an actual "lab", this would be no problem at all (Anand, Tom, Sharky, etc) because the central location would facilitate the use of the exact same components (and a whole slew of them) in every review.

JackieO made an excellent point too... There are little or NO benchmarks that really show the benefits of even running dual processors. <b>We</b> know they're there, but let's face it... If all we were doing was running Office apps, why would we need a single PIII 600+ let alone a dual? I don't put much stock in Sysmark, Winstone, et al because quite frankly, they are worthless for anything except showing what a single processor can do (which is great for a site that covers single CPU systems, but not for us).

What are we left with? Synthetic benchmarks which are nice for overall system generalizations, specific applications which show the power of dual processors (3DSMax, AutoCAD 2i, etc), and personal opinions about stability... Not much at all really.

Some people have given us some crap over the last two (yes, <b>two</b>, almost :)) years about our "subjective" reviews, but lets face it... Subjective is about all we've got in this market. We as a staff have to work with this technology every day in our real lives, so we have opinions about what we think makes a good product based on what we see in every day use. My personal machines run for weeks/months without a reboot, but the stress I put them under may be VERY different than you... We just need to find a happy medium.

Rest assured though... We will <b>NEVER</b> recommend a product that we wouldn't use ourselves, but as always, <i>your mileage may vary</i>.

[JackieO]

Amen to that Hooz... You guys have pioneered a tough market. Reviewing dduallies. I mean, it is SO hard to quantify the gains of a duallie system. The one benchmark I swear by is CLI bench and it is relatively unknown by most people who read the reviews.

You guys may want to take a real look at FlasK 0.6 and PVMPOV rendering. One supports SMP natively, the other requires PVM to be installed on Linux, but really goes a long way in showing the FPU and memory performance of a system.

Oh, and Sysmark does blow and suck, but some of the tests it uses actually are multi-threaded, so it isn't all bad. For example, my PIII 1.0Ghz dually does better than my single Athlon 1400 in Sysmark 2000 and 2001. Not a lot mind you, since not ALL the bcnhmarks it uses are SMP capable, but it isn't all bad.

So just keep doing what you're doing. You have a base of readers that know that you know what you're doing anfd respect your opinion - and that is what it all really biols down to, an opinion....

[Scott Van Note]

Doesn't matter which benchmarks.
The motherboard reviews at 2cpu.com are so narrow shallow trivial that benchmarks won't help much.

Just continue on.
List the manufactuere/name of the motherboard and just say whether you like or not.
Why spoil the tradition?

*I* don't expect better anymore.

[Jason_Dickey]

I wouldn't trust CLI Bench scores very much. The disk section of it seems especially poor.

Actually, I haven't found one [free] synthetic disk benchmark that can truely measure the benefits of hardware RAID.

But that's not what we're talking about.


I would like to see (or at least what I would look at):
Sandra CPU/Memory screens
RC5/SETI scores
Some sort of Rendering benchmark comparison
A general subjective evaluation including stability
and some evidence that onboard components (IDE, SCSI, Sound, Video, NIC, etc.) don't suck - basically up to industry standards

I think the last point hasn't been raised enough yet - we don't want to get a good, stable board that has very lousy IDE performance or an onboard NIC that saps the CPU excessively.

[Black_Baron]

*fumes* I'll be ignoring that parochial(sic) post as I'm currently ill-equiped to control my rage.

The thing that gets me the most and really bothers me is most often, when a board is reviewed, all tests are done without the components being mounted in a case. Granted, its easier for the review to troubleshoot problems if the board isn't insside a case (it also makes for easier viewing of "the magic smoke" if the board isn't in a case, but that's asthetics, we're talking function). I'd just like to see a few pics of the board inside a fairly common *or* standard case, take the addtronics 7896A for example, it has a removable motherboard tray, I'd like to see this thing on a tray inside a case before or after the tests are run (it'd also be nice to know that the board does function inside a 'sealed' box). This way, readers can actually see for themselves the problems that reviewers speculate on. DVD266-R reviews say that the floppy connector position could interfere with its use in cases such as the one previously mentioned. instead of telling us, why not show us? Right now I'm looking at that case and praying that when I do buy one, my motherboard and all the associated cooling hardware will fit in it (right now it doesn't fit in this cheapo $50 case). If I could see even an air-cooled VP6 in there, it'd be nice get an idea about spacial relations.
I can look at parts in the store, and I can come home and look at my parts. I can go back and measure parts in the store, I can come home and measure my parts. I *can't* definatively say "it will work." I built a dual processor server for a company once, I didn't order cables until the day after I had played with the parts A LOT.

As for having a "lab" to test parts, I have to say I'd never trust a lab review (after all I can get 786k on that speedpath in "the lab", so that means I can get it here right?). They always seem to boast that "x" can do "y" for "z" but when the part comes out of the lab, it don't work that way, in which case you may as well throw it all out the door and start guessing as to what's going on (sound familiar?). I did have one idea though.
I'd say its a fair assumption that at least 90% of the people that come here have a least one SMP system running. And since there are only a limited number of computer parts available, one can then assume that there would be several of us with *similar* setups. Why can't we do testing of sorts? Now I know this is really a dicey proposal to be getting into, but, this way, it'd be possible to test a lot of stuff without a lot of overhead.

*shrugs*

oh, and how many ppl can code C++? I'm talking advanced knowledge. If you know how, why don't *you* (or a group of yous) write your own benchmark?

I can vaguely see 2Cpu.com becoming more than just an SMP review/news page. It *could* be a veritable SMP resource page. But that takes a lot of work and a lot of dedicated individuals...

[Jim_]

Originally posted by Scott Van Note
Doesn't matter which benchmarks.
The motherboard reviews at 2cpu.com are so narrow shallow trivial that benchmarks won't help much.

Just continue on.
List the manufactuere/name of the motherboard and just say whether you like or not.
Why spoil the tradition?

*I* don't expect better anymore.

Oh well excuse us for doing such a horrendous job. I'm going to lose tremendous sleep knowing one person isn't satisifed with the hard work we do here. I love how people will sit on the sidelines and criticize how people evaluate hardware. You know what? If you think we're doing such an inferior job, feel free to register a freaking domain and do it yourself.

If you had ANY idea how much time I put into the DVD266-R review to get 10 pages you'd lose your mind. People think we have an easy job considering we do get to keep some of the hardware we evaluate. The fact of the matter is the time I had to put in on that is worth much more then the cost of the board.

We have real lives, real jobs. You know, the 8-5 sort of thing. As of today I've received 0 dollars for my contributions to this community we have here. If you want 26 page reviews where more then 1/2 is regurgitated from the press kit that came with the hardware... visit Anandtech.

Oh I love being appreciated.... sigh.

[JackieO]

Originally posted by Jason_Dickey
I wouldn't trust CLI Bench scores very much. The disk section of it seems especially poor.

Actually, I haven't found one [free] synthetic disk benchmark that can truely measure the benefits of hardware RAID.

But that's not what we're talking about.


I would like to see (or at least what I would look at):
Sandra CPU/Memory screens
RC5/SETI scores
Some sort of Rendering benchmark comparison
A general subjective evaluation including stability
and some evidence that onboard components (IDE, SCSI, Sound, Video, NIC, etc.) don't suck - basically up to industry standards

I think the last point hasn't been raised enough yet - we don't want to get a good, stable board that has very lousy IDE performance or an onboard NIC that saps the CPU excessively.

I agree with everything you say except one point I want to ask about - besides the CLI disk bench sucking, what don't you like about the other tests CLI runs? The only reason I ask is because I think some of the tests it runs (in particular the memory throughput test) are very good....

[jano]

There's never been a "benchmark" for this, as obviously it's incredibly time consuming and probably pricey. But what about "hardware compatability or compliance" tests?

The kind where you try several various combinations (different ram, different vid cards, different sound cards, different cpu's), etc etc. But not just one or two different.. I means several of even the same kind!

[JackieO]

Originally posted by Jim


Oh well excuse us for doing such a horrendous job. I'm going to lose tremendous sleep knowing one person isn't satisifed with the hard work we do here. I love how people will sit on the sidelines and criticize how people evaluate hardware. You know what? If you think we're doing such an inferior job, feel free to register a freaking domain and do it yourself.

If you had ANY idea how much time I put into the DVD266-R review to get 10 pages you'd lose your mind. People think we have an easy job considering we do get to keep some of the hardware we evaluate. The fact of the matter is the time I had to put in on that is worth much more then the cost of the board.

We have real lives, real jobs. You know, the 8-5 sort of thing. As of today I've received 0 dollars for my contributions to this community we have here. If you want 26 page reviews where more then 1/2 is regurgitated from the press kit that came with the hardware... visit Anandtech.

Oh I love being appreciated.... sigh.

A-*******-MEN!

[freesailer]

All I can say is keep up the good work.

Considering the time and money it takes to do a "Good Review" I think you all have done a very remarkable job. As to the test configurations all using the same or nearly the same components I can see a problem with that in that parts that are supposed to be identical are not always so. Be it boards, memory or hard disk subsystem.

Synthetic benchmarks only give you general idea as to what a system can do at that time and place. But it is a start.

There needs to be more testing of sub systems such as AGP or IDE. The Serverworks chipsets come to mind here. I know I have just uped the time and cost by ten + fold for a review. This is the kind of info that can make or break a decision for me as to how I spend my money on a system.

My 2 cents worth is: Make it a mix of all with a view to system staiblity and general performance.

[cvidler]

One thing which has only been touched on in the thread (so far) is - benchmarking procedures depend heavily on what you want to benchmark!

If you're testing how fast a P!!! 1GHz is compared with a Athlon 1GHz you'd want to use a synthetic benchmark - that stresses only the CPU, so components like the hard disk, nic, video card, etc are not important, and components like RAM and the mobo (essential for comparing between Intel and AMD etc) something like SiSoft Sandra gives a good 'raw' result, and distributed.net RC5 is great as a more 'real world' like test (while still being an almost 100% CPU test - make sure you disable disk buffering). This is the best way to compare individual components.

If you want to compare entire systems - thats when you go for 3DMark or WinBench, as these application level benchmarks test all the components of the system and gives you a result on how they work together (having a 1GHz processor and 4MB of RAM will still give you a slow ass system).

So for my AUS$0.02 (currently worth about US$0.01) give us benchmarks relevant to what's being tested - CPUs, Mobos, RAM etc. syntetic is the way to go - entire systems - application level benchies.

[acc3d]

Hooz-

Just so you know, it is possible to test Photoshop 6 without EVER paging to disk!! All that needs to be done is turn the number of history states to 1 under preferences. As far as valididy, I say Photoshop is one of the best real world processor/memory tests out there. It is also the most widely used graphics program on the planet.

I put my vote in for benchmarking Photoshop with PS5Bench:

http://www.geocities.com/Paris/Cafe/4363/

It includes basic, intermediate, and advanced tests that will surely stress any system. We all know that Photoshop is has multithreaded extensions--perfect for testing that new duallie.


Also, I recommend benchmarking with some multithreaded 3D animation/rendering packages such as Lightwave3D, Maya, or 3D Studio Max. These rendering pipelines will show FPU performance. Lightwave in particular ships with benchmark scenes, and you can check out some dual and quad results here:

http://www.blanos.com/Benchmark/

Well, there's my $.02. Actually, it sounded more like $.04...

[Black_Baron]

I seem to get the impression that the only way to test a product and eliminate interferrence by completely isolating it from other influential componenets.

i.e. testing the stability of Via chipsets by eliminating board components from the mix - testing *just* the chipset, BGA package only...

But, that's not possible and I'm probably only trying to push the number of posts I've written up, but I felt I had something relevant to say...

[Kampf]

Well, as much as we'd love to do more comprehensive hardware tests, we really can't.

My first review for this site was a KVM, the Linksys. Why did i review it? I had it, and two systems, and a lot of people have KVMs.

Second review was of the DFI P2XBL-D. A board i bought to run two P3s in.

Third review was the Tyan Tiger 100 rev F. Another kick-ass board from Tyan that loved my cB0 600s. Again, i bought it for my personal system.

Fourth review, LiteOn FS020. Bought a new case cause, well, it's cool.

Fifth review, Coolermaster ATC-200. A friend started selling these and i thought it would be cool to try it out. I no longer have it, as it wasn't mine.

Sixth review, ABIT VP6. Slots are fine, but sockets are where it's at. Two 700s and some good ram make for 2GHz of fun. I bought EVERYTHING.

Seventh review, Supermicro SC-750. Needed another case, bought it.

So....Six of seven i bought, and the seventh was just for a review. All components, save for the Fastrak66 in the Tiger100 review, which was Hooz', were purchased by myself. All RAM, HDDs, Video, CDROM, etc.....

We are getting more stuff all the time, Tyan has once again blessed us with a bevy of new motherboards. Corsair has answered the call and supplied some RAM. Things are looking up, and busy. I still have nothing to do :D

So, as a hobby, i buy computer hardware, write my opinion of it, and these guys let me put it up. Most people dig it.

You're welcome.

[Black_Baron]

I should probably mention that I totally appreciate what reviewers do and would try myself if I found myself in possession of something worth reviewing, but I don't think I have anything new or special so I'll just wait. Maybe someday you guys'll find out what my watercooling rig looks like (near as I can figure 78F below 32F)...

[cvidler]

back to what I was saying in my previous post...

I like synthetic benchies for individual components - so I can chose the best components.

I also like Application/games benchies for complete systems, so I can tell if the best components (results from above) do indeed work best together. Because I have found that sometimes the best (fastest, biggest, etc.) don't always like working together (eg. IBM DTLA HDDs don't like SAMSUNG HDDs on a HPT366 controller [bp6] - although they work happily on a HPT370 controller [vp6]).

Basically what I'm getting at is - it is very difficult (read - time/money consuming) to get benchies that will satisfy everyone - but following the few guidelines I mentioned in my previous post - we should be able to give everyone benchmarks they can use (even if it is not the ones they want) for the particular component/system being tested. This should al work fine as long as the test are run in a controlled manner - same software/os/drivers, same mobo/hdd (as appropriate) - so results can be cross-compared with others (both past and future benchmarks).

This is where I disagree with SiSoft Sandra and 3DMark - the CPU (and other) benchmarks change substantially between versions making it impossible to accurately compare results from older/newer hardware. You can compare your Voodoo 2 with a GeForce3 because the results for the Voodoo 2 were (most likely) done on 3DMark 99 - the GF3 on 3DMark 2001 - the numbers don't correlate in any way. How much faster is you new Athlon compared to a PIII - how realy knows AMD release benchies that really suck on Intel and vice versa, you try another program but again it may be optimised for one processor more than another resulting in unfair results (P4 all over again).

Hope this has given someone an idea to create a universal benchmark that (in the case of CPUs) does not contain any unfair advantages (non-optimised code?) and has enough for-throught to be valid for not only todays ~1Ghz processors but yesterdays 500Mhz - 5Mhz CPUs and tomorrows 10GHz monsters...

----
If you can't tell - benchmarking gets me fired up :rolleyes:

[c_ramseyer]

I am just about to get into this whole SMP thing and I say (drum Roll please) Do all that you can. This is not 2cpuserver.com or 2cpugamer.com.

I personally am looking tword the future. Windows XP will use dual processors and games will use dual processors. The rest of the world will have to catch up, I am going to do it now so when the next guy comes to me asking for a sh*t fast computer I can be ready.

As for benchmarks, I would like to see at least 1 gaming benchmark becouse that is what I do on my computers. The hosting of F22 Lightening 3 does not even stress my system that I have now and I do not play any games that stress what I have now. But, I am assuming that your NEW platform is going to be dual Athlon 4's and I still want some test that show how many times it crashes in a weeks time. I hate doing something and it crash...Amd has left a bad taste in my mouth from my first system...K6_2 450 and a Tyan 1598...CRASH AND BURN!

[Wowzers]

Hmmm,

I have noticed in a few comments, that Flask was being considered as a benchmark. I would love to see it as one, but I heard that the current version (.60 as I recall) isn't really doing a good job of SMP. I can't remember where (Tom's Hardware??) but I seem to recall that when operating in SMP it would only run each CPU up to 50% and no higher. This ment that one CPU could actually perform as well as 2 in a dual. :(

If they fix up the software, it would be a great test. Did anyone else hear something about that? For all I know, it has already been fixed.

[Black_Baron]

I've noticed that when I'm running seti or PRime95 on my VP6 either program will take the cpu load to 50%, which is 100% of 1 cpu. I don't know if this is quite what you're talking about, but its what I've been experiencing.

[JackieO]

Originally posted by Wowzers
Hmmm,

I have noticed in a few comments, that Flask was being considered as a benchmark. I would love to see it as one, but I heard that the current version (.60 as I recall) isn't really doing a good job of SMP. I can't remember where (Tom's Hardware??) but I seem to recall that when operating in SMP it would only run each CPU up to 50% and no higher. This ment that one CPU could actually perform as well as 2 in a dual. :(

If they fix up the software, it would be a great test. Did anyone else hear something about that? For all I know, it has already been fixed.

Alright, let me set the whole FlasK/SETI thing straight.

FlasK 0.6 DOES support SMP. I don't know what Tom was thinking (well, one of his lackey wrote that) but I have a screenshot...hold on.

http://fullon3d.com/files/FLASKSMP.jpg

Sorry modem users. Anyway, that shows only FlasK running with just about 100% CPU utilization. That is with two CPU's of course.

As for SETI - no it does not support SMP with one instance running - BUT, you will notice if you start a SECOND client that has been installed in a completly new directory the CPU utilization will jump to 100% (using both chips) and your SETI times will increase only about10-25% instead of 100%. So for example if you do one unit in FIVE hours then doing them one at a time two of them will take TEN hours right? With a dually if it takes you FIVE hours for one, then running them two at a time it should take about SIX hours... So you save FOUR hours or so.

See what I am saying? So to benchmakr SETI on an SMP system what you do is run one unit, measure the time, then run two at a time and measure the time it takes to do both. Of course you need to use a standardized unit, but it is a great benchmark.

[Adam Brackney]

JackieO, what's this you say, Tom's Hardware screwing up when it comes to SMP, say it isn't so:)

[Hooz]

I have considered using Flask as a benchmark before. I'll look into it a little more to see if it would be a decent benchmarking application.

I haven't played much with the latest Flask, because I actually <b>use</b> it (and the latest was junk IMO). I stuck with the older, slower, more reliable version.

[JackieO]

Really? I found 0.6 to be much better that 0.54 or any of the older ones... Oh well...

[LinkerX]

Well as we all know the more software that you load on a system the more unstable it becomes. So why not have a standard of applications running on all test systems like im sure many of us use aim, icq, etc.... so why not make a standard list of those products and then load those on every computer that is tested.

[Black_Baron]

Yeah, that's what I'd *most* like to know, how the system behaves if you use it as a personal workstation. I noticed the PC I've been using since my VP6 system ate my Geforce ultra (then the reseller I'd sent it to for an RMA disappeared) crashes almost three times a day, and it was a rock stable NT 4.0 server system that ran Q3 server flawlessly. It doesn't really mean anything to me if you let the linux kernel compile for 3 months straight and it never skips a beat. I don't do heavy compiling work on my system (lack the oragnizational capabilities to lay out C++ projects) all the time. I do run ICQ and notice little things like when I use personal sound scemes that I don't get any sound, or that when I fire up my 8250i and grind out a cd winamp sounds like someone listening to phonos in the car going down a bumpy road (styx did sound pretty cool though). Or, it'd be nice to know little things like that...whether I can trust it enough to leave outlook express open while I play quake3 with winamp in the background...

I like q3, but q3 & darvoset don't mix...

[spaceley]

BRAVISIMO JACKIE 0..!
FlaskMpeg is a fantastic tool for system validation. Unlike Spec Viewperf or 3dMark et.al there is scalable measurement without the do or die realities of say Spec or 3dmark. In those performance tools if your display system is not optimized or not dialed in they crash. Fine if you like an all or nothing result. With Flask the results are determined by fps and there are many areas of the system settings and ones hardware choices that can and do reflect in the fps results in Flask. There are some problems with using Flask as a quasi-standard tool if you are in the reviewing biz.
Tom is an ass this we all can agree but he exposed the area that can be a source of repudiations when comparing Flask results. The fact that the source is open and there are several optimized variations for Flask leave open the opportunity to enhance for a specific result. Tommy boy used this to prop up the AMD Flask results so he could slam the p4. Well there are now some great AMD Flask versions which optimize performance for the AMD based system.
Conversely there are many p3 versions as well. Most do not improve on Avery's work. Some do add extra features and provide fixes for inconsistencies. But the fact remains that Flask is an ever changing force and its results rely heavily on the users ability to tweakup the system. This I suspect is not very helpful for journalists doing reviews.
Also , I am not so sure that Flask can sustain much more visability and still remain what it is today. It lives within the grey area surrounding mpeg4 and dvd ripping. Once this silly SAG strike potential dissipates we will see a concerted effort on the part of "Hollywood" to find a fix for the DVD ripping situation. Some big wigs have lost a fortune over this and they want it to stop. We see what happened to MP3.com...The value of the stock went from an opening value of $28 ps to a closing day high of $108. Now little more than 18mos later its somewhere under $5 ps. Point being, when Hollywood bitch slaps they do it well. This all points to a probable end to the beauty that is Flask.
One last diatribe.... The reason I suspect most people that use or have used Flask feel it does not use SMP very well are the same users who reduce the source from its native 740x480 to somewhere around the aprox.400x300ish range. But if we look closely at Jackie O's settings we can see upscaling. I can tell you that a good way to push some P3's is to take a lowly 240x176 media, and bump, well super bump, it up to 720x480. Now thats how you make intel sweat!
I wish Flask would remain amongst ourselves, I see how commerce mutates these things which we deem perfect.
This you can quote me on...
I think that 2cpu.com would be true to its pioneering spirit by using Flask as one of its benchmarks. By choosing the standard "AVERY" verified release this site would once again catapult itself to the top of review sites as being FEARLESS but fair and not JUST ANOTHER HELMET POLISHING FLUFFER for the Corpo. king pins....(read Tom, Anand, et.al.)
hehehehe
SPACELEY OUT!

[JackieO]

Well said. I agree with everything you mentioned. I like FlasK a lot. Originally it was just going to be for benchmarking, but it has pulled me into it's 'cult following'. The only problem I see with using it to benchmark is that there is no standardized file to test and settings to use. So you need to test against other systems as a sort of pole for your FPS.

For example, in my next review I'll be using it and I am getting fsp around 60 with the files I am encoding - obviously that is higher than Tom's 7-20 FPS with his file.. the systems I am using are faster but not a ton, so obviously there is a setting difference. Of course I will need to mention that or I'll have a million little Tom fans bitching me out :D

Anyway, it does go a long way towards determining overall system speed... you guys might want to give it a whirl just to see how you like it as a test.

[webox]

Hard Drive Raid is probably the most important. Isn't the storage the biggest bottleneck with any system?
We are having trouble getting consistant readings in Sandra on this with our D6VAA & 694D's for instance.
One persons opinion. (<:

[Jim_]

We are having trouble getting consistant readings in Sandra on this with our D6VAA & 694D's for instance.

I think Sisoft Sandra is the worst freaking disk benchmark known to man. It sickens me. You get a different score everytime and a good portion of the time the score doesn't reflect how the controller 'feels' when using it.

I'm contemplating dropping it from my benchmark list and just using HDTach and ClibenchSMP. I think they're MUCH better I/O benchmarks.

[JackieO]

Originally posted by Jim


I think Sisoft Sandra is the worst freaking disk benchmark known to man. It sickens me. You get a different score everytime and a good portion of the time the score doesn't reflect how the controller 'feels' when using it.

I'm contemplating dropping it from my benchmark list and just using HDTach and ClibenchSMP. I think they're MUCH better I/O benchmarks.

If you have the time. Intel's IOMeter rocks, but it is a pain to learn and use.

[Black_Baron]

it'd be *really* nice if the benchmarks used could be had by the little guys and the beginers, perhaps eventually with a "how-to bench" article so that everyone can get to comparing the results...

[Wowzers]

Hmmm, Interesting.

I agree with Adam Brackney, Say it Ain't So!

:eek:

[JackieO]

Originally posted by Black_Baron
it'd be *really* nice if the benchmarks used could be had by the little guys and the beginers, perhaps eventually with a "how-to bench" article so that everyone can get to comparing the results...

Most of them are easy to download. Check the benchs I use in my next article:

3D Studio MAX 4
CLI MkIII SMP Bench 0.7.10
FlasK 0.6 Beta
PVMPOV 3.1E
SETI Cli 3.03
Sysmark 2000 / 2001
Content Creation & Business Winstone 2001
ScienceMark V1
Cachemem
SPEC Viewperf 6.1.1
Quake3 Arena demo001
Evolva Demo

Seven of those 14 are downloadable, one more just about everyone has... So it isn't totally unrealistic to get some of them.

[Black_Baron]

I'm sorry I should have worded that differently, I mean that it'd be nice if reviewers included links to the bench programs they use so the little guys can download them...I see it so often that there's a really good bench program that looks like it'll do the job well, but I can't for the life of me find it, and it takes about a lifetime to get info back from the reviewer.

3D Studio MAX 4
CLI MkIII SMP Bench 0.7.10
FlasK 0.6 Beta
PVMPOV 3.1E
SETI Cli 3.03
Sysmark 2000 / 2001
Content Creation & Business Winstone 2001
ScienceMark V1
Cachemem
SPEC Viewperf 6.1.1
Quake3 Arena demo001
Evolva Demo

notice that *most* of those can only be downloaded from you favorite warez server...

[DiabloRex]

a written conclusion is always a good thing. So keep that in mind when you have all the other numbers from several benchmarks programs.

[JackieO]

Originally posted by Black_Baron
I'm sorry I should have worded that differently, I mean that it'd be nice if reviewers included links to the bench programs they use so the little guys can download them...I see it so often that there's a really good bench program that looks like it'll do the job well, but I can't for the life of me find it, and it takes about a lifetime to get info back from the reviewer.

3D Studio MAX 4
CLI MkIII SMP Bench 0.7.10
FlasK 0.6 Beta
PVMPOV 3.1E
SETI Cli 3.03
Sysmark 2000 / 2001
Content Creation & Business Winstone 2001
ScienceMark V1
Cachemem
SPEC Viewperf 6.1.1
Quake3 Arena demo001
Evolva Demo

notice that *most* of those can only be downloaded from you favorite warez server...

7 out of fourteen free isn't bad :)

And if anyone wants to know where to get one, EMAIL ME.

[Black_Baron]

does 3dsmax still need a dongle?

[Rocketman]

Anandtech had an interesting benchmark in their dual P4 review.....
It was a snapshot of 30mins of change to their forum databases. This snapshot was then replayed on the test systems with identical I/O configs. The systems could then by ranked on the time it took them to implement the changes, obviously faster being better. It threw up some nice results.
Also the linux compilation test were good too. Rest of the review and conclusions were pretty humdrum though.

Any chance of seing these benches, or similar, in future reviews?

thanks

Rocketman

[JackieO]

Originally posted by Rocketman
Anandtech had an interesting benchmark in their dual P4 review.....
It was a snapshot of 30mins of change to their forum databases. This snapshot was then replayed on the test systems with identical I/O configs. The systems could then by ranked on the time it took them to implement the changes, obviously faster being better. It threw up some nice results.
Also the linux compilation test were good too. Rest of the review and conclusions were pretty humdrum though.

Any chance of seing these benches, or similar, in future reviews?

thanks

Rocketman

Problem with that is I don't think these forums have 60,000 users with over 600 logged in at any given time :)

I see what you're saying. and it was a cool idea for him to do that...Just keep posting and maybe we'll see it here too!

[Black_Baron]

I'd just like to take the time to point out the futility of the linux compilation bench. How many times an hour can you install linux? How many times do you want to? I can tell you one thing, I've intalled linux twice and any other windows based OS about fifty billion times, I can install NT in under 30mins. But that doesn't make the computer any better. It just says that I know what I want. Kernel compilation tests are virtually useless as they really don't show anything (although I can't rule out how they hellped in the 1.13GHz affair). Sure, it shows stability under those High-load situations, but what of the rest of the time when we aren't compiling code for 72 hours straight? what about times like this (see attached picture)?

[Rocketman]

You could always extend the period from 30mins -> 1 day, to provide a larger number of changes for the databases to implement. The problem is I'm not sure how good VBulletin would be for this benchark type compared to the custom stuff Anand uses. Borrowing Anand's database to standardise a benchmark would be difficult at 3Gb a go.

Black_Baron - For me the point is not how fast I can install Linux, it can just be used as a good multithreaded synthetic benchmark, with added emphasis on stability, as long as you standardise the kernels you are compiling (apart from chipset/processor differences obviously). Also I do a bit of programming/compiling and so the linux compilation tests would show some relevance to my purchase.


Rocketman

[Black_Baron]

ok, ok, so it has some merits. What really bothers me is that the more we probe into this issue the more disparity we encounter. We all do some programming to some level (I think the last program I wrote was a couple of months ago) and we all install at least one type of OS. Benchmarks really are only replicable on the machine in the specific configuration they're tested as we all know. The only real issue that we can compare is the ease of use, and as with our CPU utilization habbits, there's disparity there as well. No two people like the same setup. I like SMP systems because they have the power to be able to multi task beyond the wazzoo, but my dad (windows NT systems administrator) like Uniprocessor systems because they're easier to maintain.

Perhaps the best way to review a product is to simply take pictures of the thing in various configurations, as well as come up with a vary basic config using a standard type of baseline parts to test it then post up times of how long it takes to do various routine tasks, like install windows, or something like that.

I dunno...6 of one, half a dozen of another...

[Proteus]

I've always maintained that no benchmark is realistic unless its represents common usage. Sure..run your standard benchmark. HOWEVER..in the background make sure your ICQ/MSN/Yahoo, Antivirus, firewall, MP3 player, CD burning app, outlook, etc are also running. I guarantee you'll see some major differences in single and SMP performance. What most people don't realize is that while dualies don't necessarily improve single app performance, they dramatically improve responsiveness when multiple apps are open.

[Proteus]

Seti actually supports 8 processors quite well thank you...at least in text mode. I use it to burn in our Datacenter boxes at work. The GUI mode currently supports only 2 way SMP...but again, does it fairly efficiently. If you need more details, let me know.
-Dan

[Black_Baron]

like this?

[Black_Baron]

http://members.home.net/ldwells4140/tbb/untitled.jpg

[PsycoBob[Q2]]

Proteus: what version of SETI@home are you using? Is it the windows 3.03 version?

As for reviewing.... Doing good so far, but I'd like to see more detailed information when I see 'placement might be a problem.' Is it possible to start a database of various boards (perhaps with the coopertaion of other groups?) as simple stick-figure bluepritns, showing where the connectors are in relation to possible problems (sockets and large capacitors comes to mind, along with DIMM slots and HDD bays)


Goofy question: any idea why noone's made an AGP board with the components on the OPPOSITE side from the damned PCI slots, so putting enough cooling to keep things from smoking doesn't block things?

[Black_Baron]

When the PCI bus was originally created, the cards were built in opposition to those of the ISA bus so that a *moron* wouldn't try and put a PCI card in the ISA slot and vice-versa. AGP cards MUST follow the same orientation of the PCI cards as long as the motherboard and case manufacturers want to maintain backwards-compatibility with older designs that *don't* have the opposed position AGP slot.

It is not possible to "flip" the compnents side of the AGP card and *not* have the small wires on the adjacent PCI card in contact with the AGP card and the wires on the AGP card contacting the PCI card. Normally these wires aren't very stiff, but that's when they're more than 1/8" or so long. At that length they're very stiff and will easily scratch the traces on the adjacent card, making electrical contact and frying both boards. (try it by populating a PCI slot on an unpowered AGP equiped motherboard. Then take an ISA card and hold it over the AGP slot connector. You'll see what I'm talking about.) The only feasible way to have the components on the side of the agp card opposite the current side, is to move the agp slot closer to the CPU/Memory/IO connectors, decreasing the area available up there. This isn't a lot of spacce we're talking here, but think about some of the space restrictions you'd run ito with whatever you're sticking on there. Also the millimeters of distance you'd be moving the AGP connector wouldn't be enough that you wouldn't be able to install such an AGP motherboard in a system designed to be non-agp, it'd be just enought to obscure the mounting screw hole that you'd need to secure the card. This would be enough to warrant an entirely different set of cases needed to accomodate and properly mount a card and secure it. That could potentially be an expensive few millimeters for case manufacturers wanting to support the new standard.

I don't want to make a lightyear out of a millimeter here, but that small distance could have the potenetial to cause quite a stir. Or a relatively simple clamp could be fashioned to do the job, but inevitably someone would come along and design a case specifically for the new standard and take the market...however the other cases would be more versitile.

now that I've thought about it, the flipped agp card wouldn't fit in a case at all...unless you locate it at the opposite end of the board where the ISA slots used to reside. Otherwise you'd have to come up with an entirely different case. Forget what I said about small modifications, that would be quite a bit of involved engineering going on there to get everything to work again.

SImply put, its not possible due to the original design of the ISA bus. THe PCI bus was designed based on the ISA bus, but it could have been fixed there, it wasn't so what now follows is that there is no design leverage over the AGP slot placement and card component orientation. it *HAD* to be that way.

[MichaelMR2]

What I think psycobob is refering to is not moving the WHOLE card around, Rather just moving the devices the need large amounts of cooling(and/or those parts that stand too tall to allow the first PCI slot to be populated)

Basically I'm talking Moving the GPU and the RAM to the other side of the board which I doubt would be much of an engineering problem...

However this brings up another major problem in that their ain't much room on the other side either!!
I run a fan stack which, even if it were low profile would not have clearence for the P3 Cooler I have strapped to my Card.




Anyway, back to Benchie stuffedge.

I believe Quake 3 badly needs to be forgot about in our situation, considering it goes backwards with SMP enabled..what we badly NEED is a game that runs like a beudi in a dual system(I don't know of any but there must be SOME; maybe RTCW?)

A assortment of IO benchmarks would be sweet.

I find the most usefull thing reviews provide is knowledge of board/system quirks..for example,(Relating to the MPX) the lack of USB? the halving of bandwidth between the 2 chipsets should you install a 33mhz card in 66mhz slot etc

Using some sort of Div;-) encoder would be good considering the amount of ppl that wanted a dual so they could to this AND still use their system. (I use EasyDivX http://easydivx.does.it)

from then on maybe some proggies aimed at the 2 main groups of readers?
Those with Dual and looking at what else is new(looking at more advanced benchmarks like server and so on)
Those considering the benefits of moving to a dual platform(maybe some more games and as someone previously recommended some sort of responciveness test?)