Club G34 xD For All Your G34 Solutions & Needs :)

[Gold Leader]

Looks monstrous, And this is only with two CPU's xD I think it will be a tight fit for the top socket, but seeing how these Noctua's are just barely covering the first ram slot, I think yer in luck here.
When looking at the top right CPU socket I kinda looked at the top ram slot of the 3rd CPU socket, and there is some space between that and the case ceiling so, using that as a reference I think you will be fine.

Only them cables hanging on the ceiling you may have to redirect them alongside the top outline of the motherboard, so they won't block the heatsinks when placing them, it should be possible, even if it's a tight fit, as long it fits yer fine and good to go

[Peter Tong]

Hi Gold,

Do you run Windows Server 2008 R2 perchance? i spent the time installing it (there is a 120 day eval) and for the most part it performs like Windows 7 when you enable the various features... the only downside is that it doesn't seem to play nice with some Direct X apps. i can run them rendering with Open GL but not Direct X in certain cases. Unigine Valley for example... will not run in Direct X mode any longer (will run in OpenGL). Its this type of software incompatibility (or just not knowing whether something will run perfectly) that is a bit irritating. Productivity stuff seems to work however. Man... how i wish that Micro$oft made a patch that upped the physical cpu count for Win 7 to 4.

Adding that 8GB of ram in CPU slot 2 dropped performance slightly. about 2% in Cinebench 11.5, and 6% in CB R15. Not sure why. i experimented with what few ram timings are in the BIOS and nothing brings back the difference. i should receive another 8GB today to fill in CPU slot 1... so we'll see whether performance picks up again.

SSD performance has been a bit sporadic as well and very dependent on power saving mode. It is very dependent on the minimum cpu percentage setting. If i set the minimum to 5%... sequential performance picks up a good amount over 400MB/s sequential reads and writes... but 4k64thrds suffer. If i set minimum to 100% (basically performance setting), 4k64thrds picks up to its maximum but sequentials go down about 100MB/s. I found the best compromise to be 53%... for raw ssd performance. In ATTO Setting the cpu setting down like that makes the larger blocked sequential transfers less "ragged".

Have you experimented with this? i'm curious if this phenomenon is also present on the H8Ds...

i do love the Noctua's... very quiet... when i am punishing the machine with an extended Prime 95 run on all 24 cores they do hit around 60C. i will be picking up some A9 PWM fans to help this. Normal use in "performance" mode... temps sit at around 39-40C. In "power saving" mode around 30-32C....

i did just total up things... just over $2k spent up to now... filling things out completely (another 2 processors, and 2 noctua HSF, will cost another $400 or thereabouts)... for a total cost of around $2450 or so (not including the cost of the OS).

Its definitely a workhorse that is for sure.


PS: i should add one thing... and that is the fan clips on the Noctuas extend downwards enough such that i cannot use the top most PCI x16 slot. When i pull that HSF to install the 8GB of ram i'll take some pics of this.

PPS: Samsung RAPID mode really is amazing with a multi core machine as it spreads its writes across all cores.

Originally Posted by Gold Leader

Looks monstrous, And this is only with two CPU's xD I think it will be a tight fit for the top socket, but seeing how these Noctua's are just barely covering the first ram slot, I think yer in luck here.
When looking at the top right CPU socket I kinda looked at the top ram slot of the 3rd CPU socket, and there is some space between that and the case ceiling so, using that as a reference I think you will be fine.

Only them cables hanging on the ceiling you may have to redirect them alongside the top outline of the motherboard, so they won't block the heatsinks when placing them, it should be possible, even if it's a tight fit, as long it fits yer fine and good to go

[Gold Leader]

Well OS wise, I always use Windows 7 Professional UK + Service Pack 1 on all my Multicore Opterons.
Goes for the following three:

Firefox SMP 2006:
Dual 2-Core OpteronDP 285 E6 4x 2.6Ghz
4x 2048MB PC-3200 ECC Reg HP/Micron Tech Server Ram
ASUS K8N-DL
NVIDIA GeForce GTX 480 PCI-E 1280MB GDDR5 320Bit Rev.A2 5009 Engineering Sample
Auzentech X-Fi Prelude 7.1 PCI 64MB XRAM

Blue-Leader SMP 2009:
Dual 6-Core OpteronDP 2435 D0, 12x 2.6Ghz
4x 4096MB PC2-6400 ECC Reg Kingston Value Ram
Supermicro H8DAE-2
EVGA GeForce GTX 480 PCI-E 1536MB 384Bit GDDR5 Rev.A3 2110
Realtec ALC 883 7.1HD Audio

Valkyrie SMP 2014:
Dual 8-Core OpteronMP 6140 C0, 16x 2.6Ghz
4x 4096MB PC3-12800 ECC Reg HP/Micron
Supermicro H8DG6-F
Sapphire Radeon R9 290X PCI-E 4096 512Bit GDDR5 Rev.A13 3713
Auzentech X-Fi Forte 7.1 PCI-E 64MB XRAM

Although I do think that Multi-Core Apps would run better with Windows Server 2008 R2 over Windows 7 Professional + SP2, just guessing here but I do greatly agree with Win7 Pro should have 4CPU support over 2CPU support anyday, oh yes I greatly agree there

As the SSD performance goes I am not so sure, so far everything loaded way faster than any HDD could do, I can test this for you when I get back home, I will let you know when I am back, at the moment I am on my 3 Month South European Camping trip with my parents I am on a Campsite in Blanes, Costa Brava Spain atm

But I can run some tests for you to find out how the H8DAE-2 and the H8DG6-F do with their SSD's here, both do use a SATAII-300 connection btw.

[Peter Tong]

Well i seem to have fixed my disk performance issue with this SuperMicro setup.

After much testing... it turns out that the C1E support was completely screwing up my SSD performance. It would show up in AS SSD, but it was completely obvious in ATTO.

Configuration.. SuperMicro H8QGi-F with single Samsung 840 EVO 250GB SSD running in "Legacy" mode (ie not set up for RAID) on Highpoint RocketRAID 2720SGL.

With C1E on in the BIOS:



With C1E off in the BIOS:



If you are running SSD... turn this doggone setting off

I hope this helps someone! Very important with these SuperMicro motherboards at least. i still wish my RocketRaid 2720SGL supported AHCI... but that is a different story.

[Gold Leader]

Hey Nice thanks for the tip!

Actually your motherboard is the Quad CPU variant of the one I have xD

We both have the same Dual AMD AM5690 North Bridges, 1x SP5100 South Bridge and the same SATA controllers as well, the same Matrox G200e Onboard VGA hmm

Only diff mine has to yours is the LSI 2008i SAS2-600 RAID controller, there is a variant of your board type that does have this;
Which is the Supermicro H8QG6-F:
http://www.supermicro.nl/Aplus/mothe...x0/H8QG6-F.cfm

Since our motherboards being quite similar hence yours made for Quad CPU, mine made for Dual CPU, and your lacking the SAS2-600 controller for the rest they are identical as parts would go, so as functions go I think we can still relate with many Bios settings as well

So I do think my bios has this C1E setting, thanks for the tip man, this should increase server load times when I run my Primal Carnage server for League of Dilo's the League & Community I founded for Primal Carnage xD
Here is when I was the only server people could join, since all the public servers were down for about 3 weeks, 64 people joined within2 to 3 minutes

I am (LoD)Tylosauurs as you can see, which is marked in bolded yellow

32 to 40 players runs fine but beyond the 40 my connection could not keep up much it did get laggy rofl, even though people didn't want to leave as I was the only server running, we all laughed really, this was a very cool experience but it also proved how stable the system was, also when I had to do a level change and the system successfully migrated all 64 players from one map to the other within 40 seconds without crashing, when a public server does this from pingperfect.com it always crashes and has to be restarted, their fastest is a Dual Quad core Xeon 2.26Ghz based on the C602 chipset.

[Peter Tong]

Yup... same basic family it seems These boards have a bunch of "gotchas" like this i have found from testing. i don't use the onboard controller except for my optical drive since its only SATA2... basically though to get this up to optimal performance has required a bunch of testing. Performance wise it seems to like only 4 out of the 8 memory channels populated for each CPU. If 4 are populated then a BIOS tweak nets you that 2-6% gain in memory performance. If all 8 populated... that tweak no longer works. My Tyan board seems more consistent in this respect.

At any rate... pretty soon i'll be flashing this with the [H] overclocking BIOS, and we'll see how much extra performance is available. i'll be happy if i get a 10-12% overclock... to around 2.8 GHz. The 6180SEs supposedly don't have very much headroom though.

More on C1E from AMD:

http://developer.amd.com/community/b...wn-with-amd-p/

C1E is a power management state that allows the processor to reduce power beyond just the cores. With C1E, the processor power can be reduced by lowering the memory controller clock speed and halting the HyperTransport™ technology links. This new feature was extremely important for our 12-core processors with the increase in memory channel support as well as the increase in HyperTransport™ technology links within the design.

According to AMD's publicly available BIOS and Kernel Developer's Guide, section 2.4.3.3:

The C1 enhanced state (C1E) is a stop-grant state supported by the processor. The C1E state is characterized by the following properties:

All cores are in the halt (C1) state.
The ACPI-defined P_LVL3 register has been accessed.
The chipset has issued a STPCLK assertion message with the appropriate SMAF for C1E entry. Note that [The ACPI Power State Control Registers] F3x[84:80] specify the processor clocking and voltage behavior in response to the C1E SMAF.
The processor has issued a STOP_GRANT message to the chipset.
General requirements for C1E:

The ACPI-defined C2 and C3 states must not be declared to the operating system.
C1E should only be enabled when the platform is in ACPI power management mode.

[Peter Tong]

i thought i would share another strange behavior i notice with these dual cpu G34 boards... some benchmarks are definitely off... and memory performance is also down compared to my DDR2 with registered ECC Tyan system:

Here is a Geekbench 2 comparison between my dual Opteron 8439SE (running DDR2 reg ECC at 667) and the newer H8QGi-F setup with dual Opteron 6180SE... memory performance is slower even with DDR3-1333 (well its actually non ECC DDR3-1600 running at 1333). Check out the Stream results.

http://browser.primatelabs.com/geekb...456357/2498206

Most of my multicore results are as expected (both chips are running K10 cores, although 6180 is running them at 2.5 GHz). On some of the multicore scores... my dual 8439SE is faster (despite having 1/2 the cores).

i have noticed this also in Passmark... where two of my submissions scored cpu marks in the 11700 range, and another 8800.

Maybe someone can clue me in on what is going on here?

[MU_Engineer]

Originally Posted by Peter Tong

i thought i would share another strange behavior i notice with these dual cpu G34 boards... some benchmarks are definitely off... and memory performance is also down compared to my DDR2 with registered ECC Tyan system:

Here is a Geekbench 2 comparison between my dual Opteron 8439SE (running DDR2 reg ECC at 667) and the newer H8QGi-F setup with dual Opteron 6180SE... memory performance is slower even with DDR3-1333 (well its actually non ECC DDR3-1600 running at 1333). Check out the Stream results.

http://browser.primatelabs.com/geekb...456357/2498206

Most of my multicore results are as expected (both chips are running K10 cores, although 6180 is running them at 2.5 GHz). On some of the multicore scores... my dual 8439SE is faster (despite having 1/2 the cores).

i have noticed this also in Passmark... where two of my submissions scored cpu marks in the 11700 range, and another 8800.

Maybe someone can clue me in on what is going on here?

Sure, I can help. The memory benchmarks you are running are crap. They are single-threaded and coming nowhere neat actually saturating the memory controllers. You are seeing results of less than 2 GB/sec. Think about that- you have a platform with a theoretical memory bandwidth of over 40 times that much. You need to run a proper multithreaded memory benchmark. I'd expect figures in the 50-60 GB/sec range with a good program if you're using two 6100 Opterons with DDR3-1333. Also, I'd expect the 8438SEs to outscore the G34s on that crappy single-threaded benchmark since the cores are identical but an 8439SE core runs 300 MHz faster and is connected to an IMC/L3 cache that runs 400 MHz faster (2.20 GHz vs. 1.80 GHz.)

Some "multithreaded" benchmarks aren't that multithreaded. They're often optimized for around 8-12 threads so that Intel's latest 6/8-core LGA2011 i7s with HyperThreading look better than the 8-thread LGA1150 i7s which also have to look better than the non-HyperThreading i5s but don't get spanked by fairly cheap older off-lease multi-socket Nehalem/Westmere Xeons or G34 Opterons like yours. They also have to have enough poorly-threaded bottlenecks in the multithreaded benchmarks that an 8-core AMD FX can't spank HT-disabled four-thread i5s too badly. I sound cynical but remember that BAPCo, one of the used-to-be-giant benchmark vendors, was found to actually be a wholly owned subsidiary of Intel and use a lot of nasty tricks to make AMD chips look bad. I look very poorly on benchmarks that aren't a completely known quantity, such as compiled from known source with a known compiler with known flags. Generally that means I look at and trust Phoronix since they do the above and I also run some of my own benches. I dismiss nearly all of the rest unless I am running that particular program and care to see how it runs (very unlikely, I run Linux and apart from Phoronix and some rare Web server/DB benches on Anandtech and Tom's that I don't run in real life, nobody benches on Linux.)

Also, a pair of 8439SEs will outperform a pair of 6180SEs up until somewhere around 16 cores being used hard because the 8439SEs are clocked higher and have a faster L3 cache/IMC. Put something that really stresses all of the cores in the system and the 24 6180 SE cores will absolutely walk away handily from the 12 slightly faster 8439 SE cores.

[Gold Leader]

You should try running BarsWF x64 CUDA, I ran this with my Dual 6-Core AMD OpteronDP 2435 and it's GTX 295 Red Editions and the later GTX 680's, this little apps is very multi threaded friendly, I use it to compare CUDA Performance over different GeForce GPGPU's but also to compare my Opterons from my previous builds with my later ones
Here the thread I made for it which has all the info you need
http://forums.2cpu.com/showthread.php?t=99107

[Peter Tong]

Hi Gold,

result for you:



i ran it for to the full 100%... setup: SuperMicro H8QGi-f, 2 x Opteron 6180SE, and a EVGA GTX 760 non superclocked.

Me thinks you are trying to produce a plume of hot air in this room Coming from the back of this PC ))))

Will respond with more later.

Originally Posted by Gold Leader

You should try running BarsWF x64 CUDA, I ran this with my Dual 6-Core AMD OpteronDP 2435 and it's GTX 295 Red Editions and the later GTX 680's, this little apps is very multi threaded friendly, I use it to compare CUDA Performance over different GeForce GPGPU's but also to compare my Opterons from my previous builds with my later ones
Here the thread I made for it which has all the info you need
http://forums.2cpu.com/showthread.php?t=99107

[Peter Tong]

Hi Mu,

Thanks for your reply. i do like the points you make.

Do you know of any technical reason why the IMC on the 8439SE runs faster than those on the 6180SEs? Why would they have downgraded like that? Power reasons?

Unfortunately i haven't got around to experimenting with Linux so i am stuck with what is available on the Windows platforms

The 8439SE setup does seem a bit more snappy than the 6180SE setup... i'm hoping to find some "extra spicy" MC chips so i can recover some of that snappiness.

Originally Posted by MU_Engineer

Sure, I can help. The memory benchmarks you are running are crap. They are single-threaded and coming nowhere neat actually saturating the memory controllers. You are seeing results of less than 2 GB/sec. Think about that- you have a platform with a theoretical memory bandwidth of over 40 times that much. You need to run a proper multithreaded memory benchmark. I'd expect figures in the 50-60 GB/sec range with a good program if you're using two 6100 Opterons with DDR3-1333. Also, I'd expect the 8438SEs to outscore the G34s on that crappy single-threaded benchmark since the cores are identical but an 8439SE core runs 300 MHz faster and is connected to an IMC/L3 cache that runs 400 MHz faster (2.20 GHz vs. 1.80 GHz.)

Some "multithreaded" benchmarks aren't that multithreaded. They're often optimized for around 8-12 threads so that Intel's latest 6/8-core LGA2011 i7s with HyperThreading look better than the 8-thread LGA1150 i7s which also have to look better than the non-HyperThreading i5s but don't get spanked by fairly cheap older off-lease multi-socket Nehalem/Westmere Xeons or G34 Opterons like yours. They also have to have enough poorly-threaded bottlenecks in the multithreaded benchmarks that an 8-core AMD FX can't spank HT-disabled four-thread i5s too badly. I sound cynical but remember that BAPCo, one of the used-to-be-giant benchmark vendors, was found to actually be a wholly owned subsidiary of Intel and use a lot of nasty tricks to make AMD chips look bad. I look very poorly on benchmarks that aren't a completely known quantity, such as compiled from known source with a known compiler with known flags. Generally that means I look at and trust Phoronix since they do the above and I also run some of my own benches. I dismiss nearly all of the rest unless I am running that particular program and care to see how it runs (very unlikely, I run Linux and apart from Phoronix and some rare Web server/DB benches on Anandtech and Tom's that I don't run in real life, nobody benches on Linux.)

Also, a pair of 8439SEs will outperform a pair of 6180SEs up until somewhere around 16 cores being used hard because the 8439SEs are clocked higher and have a faster L3 cache/IMC. Put something that really stresses all of the cores in the system and the 24 6180 SE cores will absolutely walk away handily from the 12 slightly faster 8439 SE cores.

[Peter Tong]

interesting article... pertaining to the stream benchmark on these types of G34 machines.

http://www.cs.uchicago.edu/files/tr_...TR-2011-02.pdf

[Peter Tong]

Hi Mu,

i am not sure how familiar people here are with the Passmark benchmark... but its interesting in that they have a database containing a boatload of CPUs... also interesting is the fact that performance on the multi chip Opteron setups has been low there. For example - Dual Opteron 8439SE... 8300, and a Quad 8439SE... only 8800 cpu marks - that just was never believable to me as far as scaling. My dual 6180SE... only scored in the 8800 range despite having double the cores. Well being curious about this... i run this benchmark once and a while while going through BIOS changes... and i noticed on two of my runs... the score jumped up to 11700. i compared these results to my previous results and found that this was due to 3 of the benchmark tests (physics, primes, and encryption) typically testing way low on these multi Opteron rigs, the rest of the cpu tests being consistently what you might expect when you double up on cores, etc. However i could double up on cores and primes basically stayed the same... perhaps a few hundred extra. So yesterday i spent a few hours looking into this (making change rebooting, testing, etc) and found the solution to why the Opterons consistently would scale in an under par fashion with these tests. i made a 16 entry test matrix for the following BIOS settings and started to methodically go through them... testing performance with multiple CB 11.5 runs, and Passmark runs.

For Passmark, i found that the following settings fixed the problem. Most importantly... Node Interleaving must be set to Auto (enabled). The others can have a detrimental effect if not set properly but don't destroy the test results completely in terms of expected performance with double the cores.

Bank Interleaving needs to be Auto (ie enabled)
Node Interleaving needs to be Auto (ie enabled)
Channel Interleaving needs to be Auto (ie enabled)
Bank Swizzle needs to be Disabled.

These are also the most optimal settings for my rig so far. There is about a 7-8% difference in performance under Cinebench 11.5 when the BIOS settings are set incorrectly.

The downside... ram latency jumps from 67 to 113 or so. This would make sense since you are getting rid of NUMA and addressing everything without regards to node.

I was also experimenting with Stream (bandwidth benchmark) and when Node interleaving is disabled... the Opterons perform horribly. I'm talking only 8GB/s. Setting thread affinity gets the test running what i would expect... 46MB/s.

Anyways, i thought i might share this info just in case someone running these dual/quad Opteron rigs finds things aren't scaling on that test properly past 12 cores.

[MU_Engineer]

Originally Posted by Peter Tong

Hi Mu,

i am not sure how familiar people here are with the Passmark benchmark... but its interesting in that they have a database containing a boatload of CPUs... also interesting is the fact that performance on the multi chip Opteron setups has been low there. For example - Dual Opteron 8439SE... 8300, and a Quad 8439SE... only 8800 cpu marks - that just was never believable to me as far as scaling. My dual 6180SE... only scored in the 8800 range despite having double the cores. Well being curious about this... i run this benchmark once and a while while going through BIOS changes... and i noticed on two of my runs... the score jumped up to 11700. i compared these results to my previous results and found that this was due to 3 of the benchmark tests (physics, primes, and encryption) typically testing way low on these multi Opteron rigs, the rest of the cpu tests being consistently what you might expect when you double up on cores, etc. However i could double up on cores and primes basically stayed the same... perhaps a few hundred extra. So yesterday i spent a few hours looking into this (making change rebooting, testing, etc) and found the solution to why the Opterons consistently would scale in an under par fashion with these tests. i made a 16 entry test matrix for the following BIOS settings and started to methodically go through them... testing performance with multiple CB 11.5 runs, and Passmark runs.

For Passmark, i found that the following settings fixed the problem. Most importantly... Node Interleaving must be set to Auto (enabled). The others can have a detrimental effect if not set properly but don't destroy the test results completely in terms of expected performance with double the cores.

Bank Interleaving needs to be Auto (ie enabled)
Node Interleaving needs to be Auto (ie enabled)
Channel Interleaving needs to be Auto (ie enabled)
Bank Swizzle needs to be Disabled.

These are also the most optimal settings for my rig so far. There is about a 7-8% difference in performance under Cinebench 11.5 when the BIOS settings are set incorrectly.

The downside... ram latency jumps from 67 to 113 or so. This would make sense since you are getting rid of NUMA and addressing everything without regards to node.

I was also experimenting with Stream (bandwidth benchmark) and when Node interleaving is disabled... the Opterons perform horribly. I'm talking only 8GB/s. Setting thread affinity gets the test running what i would expect... 46MB/s.

Anyways, i thought i might share this info just in case someone running these dual/quad Opteron rigs finds things aren't scaling on that test properly past 12 cores.

I'm not particularly surprised as I am somewhat familiar with those benchmarks. They consumer benchmarks designed to run well on Intel's single-socket CPUs. They are either completely NUMA-unaware or are very poorly optimized for NUMA. The Windows OS they run on is also very poorly optimized for NUMA as well. A pair of 6180SEs has four NUMA nodes and WILL suffer quite a bit if the program and/or OS doesn't have some sort of thread affinity as there WILL be a large number of remote-node accesses with that many nodes. Your options you picked essentially turned NUMA off and simply striped all data equally across all nodes, making it really a 25% chance of a local hit (very fast), a 25% chance of a same-package hit (relatively fast) and a 50% chance of a remote hit (slow) vs. potentially nearly all remote hits.

FYI, Intel's 5500/6500/7500 and later Xeons which do have NUMA don't suffer as much from poor NUMA locality as G34 Opterons as they only have one node per socket instead of two. Having half the number of nodes reduces the chance of having high-latency remote hits which really sap performance. Fewer nodes does reduce total system bandwidth but the gains from lower latency in poorly-coded applications can easily outweigh the increased bandwidth.

However, if you have a decently NUMA-aware OS and a NUMA-aware application, it can run very well on the Opterons and scale very well. If I were you I would go install your favority Linux distribution and run the Phoronix Test Suite equivalents to those benchmarks you listed. I'll betcha your results will make the Opterons look a whole lot nicer since Linux can be very NUMA-friendly and lots of programs can link to libnuma and work likewise. The "numastat" command will tell you how well your programs are doing in that case.

[Peter Tong]

Yeah i definitely get your point. This is all very new to me - i've never had to deal with this issue on my Dual 8439SE. It seems it was new to the admin over on the Passmark forum as well.

Does your SuperMicro BIOS have an option to enable/disable HT Assist? My Tyan S2927E board does... but this OEM Gateway H8Q board with BIOS P06 does not. CPU-z is showing 10MB level 3... so i am assuming that HT-Assist is being turned on by default and is using that 2MB of cache.

i have no idea how large those interleaved stripes are (are they BIOS definable for tuning purposes?), but i would think that HT assist would hide a lot of the accesss latency by caching accesses away would it not? But it can only hide so much... i suppose depending on how the app is written/how the threads access memory.

i am going to have to install Ubuntu soon. Hoping to have my hands on 4 Extra Spicy MC chips in about two weeks (will see).

i really would like to be finished with the PC soon... sometimes i feel like i am spending money on it like a drunk sailor.

if i can get it running with 2 spicy chips dead reliable at 2.8 GHz... i'll be happy.

Originally Posted by MU_Engineer

I'm not particularly surprised as I am somewhat familiar with those benchmarks. They consumer benchmarks designed to run well on Intel's single-socket CPUs. They are either completely NUMA-unaware or are very poorly optimized for NUMA. The Windows OS they run on is also very poorly optimized for NUMA as well. A pair of 6180SEs has four NUMA nodes and WILL suffer quite a bit if the program and/or OS doesn't have some sort of thread affinity as there WILL be a large number of remote-node accesses with that many nodes. Your options you picked essentially turned NUMA off and simply striped all data equally across all nodes, making it really a 25% chance of a local hit (very fast), a 25% chance of a same-package hit (relatively fast) and a 50% chance of a remote hit (slow) vs. potentially nearly all remote hits.

FYI, Intel's 5500/6500/7500 and later Xeons which do have NUMA don't suffer as much from poor NUMA locality as G34 Opterons as they only have one node per socket instead of two. Having half the number of nodes reduces the chance of having high-latency remote hits which really sap performance. Fewer nodes does reduce total system bandwidth but the gains from lower latency in poorly-coded applications can easily outweigh the increased bandwidth.

However, if you have a decently NUMA-aware OS and a NUMA-aware application, it can run very well on the Opterons and scale very well. If I were you I would go install your favority Linux distribution and run the Phoronix Test Suite equivalents to those benchmarks you listed. I'll betcha your results will make the Opterons look a whole lot nicer since Linux can be very NUMA-friendly and lots of programs can link to libnuma and work likewise. The "numastat" command will tell you how well your programs are doing in that case.

[MU_Engineer]

Originally Posted by Peter Tong

Does your SuperMicro BIOS have an option to enable/disable HT Assist? My Tyan S2927E board does... but this OEM Gateway H8Q board with BIOS P06 does not. CPU-z is showing 10MB level 3... so i am assuming that HT-Assist is being turned on by default and is using that 2MB of cache.

HT Assist is able to be turned off on Socket F Istanbul chips but not able to be disabled on any C23/G34 chips.

i have no idea how large those interleaved stripes are (are they BIOS definable for tuning purposes?), but i would think that HT assist would hide a lot of the accesss latency by caching accesses away would it not? But it can only hide so much... i suppose depending on how the app is written/how the threads access memory.

Not sure what the stripe size is and I do know there's no way to adjust it in the BIOS.

i am going to have to install Ubuntu soon. Hoping to have my hands on 4 Extra Spicy MC chips in about two weeks (will see).

i really would like to be finished with the PC soon... sometimes i feel like i am spending money on it like a drunk sailor.

if i can get it running with 2 spicy chips dead reliable at 2.8 GHz... i'll be happy.

Good luck!

[Peter Tong]

Thought i would post up a video of what these dual G34s are capable of...

https://www.youtube.com/watch?v=QYX_...ature=youtu.be

This is after i fixed my BIOS settings... its amazing how much quieter the system is now... compared to when i was first getting things together on it. Can barely hear it now. Noctua heat sinks are working superbly... just under 50C running Prime95 torture test? And extremely quiet as you can hear from the video. Very satisfied.

[whooo?]

I always wonder how those benchmarks relate to the real world. After all your tinkering, you got Cinebench to basically go from 15 points to 16.

Now, what difference would that make if I was rendering a 10 minute SD 16:9 video to HD (that does take awhile), how many hours do I save with that one point of improvement?

[Gold Leader]

Originally Posted by Peter Tong

Thought i would post up a video of what these dual G34s are capable of...

https://www.youtube.com/watch?v=QYX_...ature=youtu.be

This is after i fixed my BIOS settings... its amazing how much quieter the system is now... compared to when i was first getting things together on it. Can barely hear it now. Noctua heat sinks are working superbly... just under 50C running Prime95 torture test? And extremely quiet as you can hear from the video. Very satisfied.

Very nice 50C ain't bad, my 6140's never got past the 34C, also thanks to my case's great airflow from them three 140Mil fans in the front and the one in the rear.

And I told ya that them Noctua's where the coolers to get
The same here bud, I also hardly heard my Dual 6140's Noctua's make a single sound even under full load

Noctua just knows how to make you the perfect air cooling solution for a very good deal

But as real world performance goes, BarsWFx64 CUDA tells me all I need to know

[Peter Tong]

Hi, i went from mid 15s to 16.98 so around 9.5% gain. So essentially through BIOS tuning i went from a 24 x 2.5 GHz processors to a 24 x 2.73 GHz processors for heavily multithreaded stuff. So figure on shaving about a minute off each cycle and multiply the # of cycles that you do during the lifetime of that PC and also subtract from your your electric bill by some fraction of that difference and most of all subtract the value of your irreplaceable time All for no downside except the cost of the initial tinkering

i should also add... in a rather strange sort of geekish way... the fun of trying to eek out extra free performance by knowing what your computer likes to run better

Originally Posted by whooo?

I always wonder how those benchmarks relate to the real world. After all your tinkering, you got Cinebench to basically go from 15 points to 16.

Now, what difference would that make if I was rendering a 10 minute SD 16:9 video to HD (that does take awhile), how many hours do I save with that one point of improvement?

[Peter Tong]

Hi Gold,

i went from this:

https://www.youtube.com/watch?v=-2mx...Uh2iQ4xV_qciBw

to this:

https://www.youtube.com/watch?v=QYX_...ature=youtu.be

PS: i didn't even have the rear exhaust fans hooked up in that first video!

It cost me about $300 to remove all that noise. (2xNoctua's = $120, Gelid fan controller = $50, 1 x used SuperMicro 1400w PSU - 90% efficient with quiet fans - $130)... but yes i'm definitely very satisfied now!

Originally Posted by Gold Leader

Very nice 50C ain't bad, my 6140's never got past the 34C, also thanks to my case's great airflow from them three 140Mil fans in the front and the one in the rear.

And I told ya that them Noctua's where the coolers to get
The same here bud, I also hardly heard my Dual 6140's Noctua's make a single sound even under full load

Noctua just knows how to make you the perfect air cooling solution for a very good deal

But as real world performance goes, BarsWFx64 CUDA tells me all I need to know

[SadTopher]

Having spent the last couple of months gradually clearing out a serious load of single socket / unused kit, to free up some space and funds, I decided to re-invest it in one big build so it just had to be (what else! ) a G34 quad rig.

To date I've collected the following parts...

...4 x AMD Opteron 6378 (a mixture of new & used tray cpu's, although prices were almost identical and surprisingly reasonable)
...Super Micro H8QGL-6F
...64GB (16x4) Crucial Ballistix Sport VLP DDR3-1600 1.5 / 1.35V
...4 x Cooler Master Hyper 212 EVO (to be fitted using the 'Musky Mod' extension brackets) and some additional fans for push-pull operation
...Jou Jye GTX-U322D-FD Super Tower case (supports HPTX but I'll mod it for SWTX mount points). Uses the same sub-frame as the Rosewill Blackhawk Ultra Super Tower & Xigmatek Elysium


I've pretty much covered my outlay so far but there are still a few items needed...

...PSU (maybe Seasonic if I can find the right deal)
...Graphics (was thinking of the new Nvidia 970)
...USB3 card (for case ports)
...Storage
...Additional / replacement case fans

A few pics of the above











Links to the case

http://www.jj-computer.com/artinfo.php?artnr=A 1930
http://www.jj-computer.com/media/cm_..._MidiTower.pdf

[Peter Tong]

My oh my... Aren't quad G34 setups a thing of beauty? Looking forward to your getting all this together.

[Gold Leader]

Well I changed the name of this topic to Supermicro's 2CPU & 4CPU G34 solutions
I got rid of the word new, since these motherboards are not that new anymore but there ya go, I hope one of the mods and edit this topic name to:

Supermicro's 2CPU & 4CPU G34 solutions

All good then!
Btw Lovely 4CPU G34 setup thar SadTropher!!!!! Wow I really look forward to it's progress!!!! Keep it coming!

Anyways I do plan on giving my H8DG6-F based build 4x4GB PC3-12800 ECC Reg HP Server ram as upgrade to a lovely amount of 32GB and with this I will have Quad Channel DDR3-1600 @ 1333 per CPU as well The Ram I seek is the same type of ram I am using now, although I just dunno where to look for it

Here the sets I am using now btw:

Top view:


Rear view:


I am hoping to find the same ram as I am using now, if any of you can help, that would be nice, I got these from Aryan btw

[Peter Tong]

Hi Gold,

There appears to be tons of it on Ebay

http://www.ebay.com/sch/i.html?_from...-CH9Q1&_sop=15

i've found lots of good ram deals on there.... these last 8GB (4 x 2GB - F3-12800CL9D-4GBNQ) that i added was had for $60. Pretty amazing. Just ran a few passes of memtest after installing it... and good to go.

Peter

[Gold Leader]

Hmm I need PC3-12800 ECC Reg HP Server ram as pictured and now PC3-10600
As I want to use the same ram when I upgrade to Dual 16 Core OpteronMP 6380 C2's you see, which take use of 1600Mhz DDR over 1333 Mhz

[SadTopher]

Figured that was why you were looking for PC3-12800R. Problem seems to be that, even on the bay, matching 4GB sticks are fairly rare...it's all 8GB and above, like these

http://www.ebay.co.uk/itm/Hynix-8GB-...item1c457fc2aa

http://www.ebay.co.uk/itm/Samsung-8G...item1c459d730d

PS: Pete, GL...Thanks for the support on my quad build to come, I'm sure to have a few questions along the way. Also meant to say that the board was from sona.de that GL had mentioned elsewhere and they were way cheaper than UK suppliers, although I did need to wait nearly a month between ordering and delivery.

[Peter Tong]

Hi,

I took the "easy" way out of just getting a SuperMicro case for these rather huge 4P motherboards... but i am really curious how much in the way of mods you'll have to do to fit that into a non SM case. It should be interesting.

i wonder how many folks in Europe have 4P home PCs

with regards to my own project, i will be gone on a trip for the next week or so but after that i'll have 4 61xx AMD Engineering Samples coming in that are proven capable of running at 3.0 GHz (the guy folded for over a year running them that way). i can't wait to see what the performance is.

Originally Posted by SadTopher

Figured that was why you were looking for PC3-12800R. Problem seems to be that, even on the bay, matching 4GB sticks are fairly rare...it's all 8GB and above, like these

http://www.ebay.co.uk/itm/Hynix-8GB-...item1c457fc2aa

http://www.ebay.co.uk/itm/Samsung-8G...item1c459d730d

PS: Pete, GL...Thanks for the support on my quad build to come, I'm sure to have a few questions along the way. Also meant to say that the board was from sona.de that GL had mentioned elsewhere and they were way cheaper than UK suppliers, although I did need to wait nearly a month between ordering and delivery.

[SadTopher]

It's really nothing more difficult than modding the positions of non-aligned standoff mount points which is fairly straightforward. I may also have to replace the roof fans for slimline variants but that's not as critical.

I did a fair bit of research before I picked the case, especially on [H] and Amdzone forums and as I said earlier the sub-frames of the Jou Jye, Rosewill and Xigmatek are identical (save for variations in drive bay orientation) and the latter two brands have already been successfully used for 4P SM builds. Here's a couple of pics of the Rosewill version (identical to my Jou Jye), one of the bare case internal layout...

http://i1343.photobucket.com/albums/...ps970f91ea.jpg

...and one with your SM board layout already installed, complete with Noctua heatsinks (but not their fans).

http://i1343.photobucket.com/albums/...ps7aae9724.jpg

One advantage of the socket layout on my board is that the top one is slightly further away from the edge of the board, so I have more room to play with.

As for Euro 4Pers, probably not that many. The market in price and availability is completely different to the States though we do try our best to keep up!

Like the sound of your impending move to 4 ES chips - that's hardcore.

[guyr]

I'm skeptical that we are much better off with higher clocked memory. Every time the clock rate increases, so does the latency. In real world terms, are we any better off with DDR3-1600 CL11 than we were with DDR2-667 CL5? For my dual C32 Asus build, I took a chance with Kingston DDR3-1600 CL9. Fortunately, it worked out of the gate, and I haven't had any problems in 2 years. But even CL-9 is not that great.

Maybe spin torque memory will finally get us into the 21st century? http://spectrum.ieee.org/semiconduct...hows-its-might