Subject: RE: Running CCM3 on PC Clusters
From: Ghan, Steven J (Steve.Ghan@pnl.gov)
Date: Tue Jan 08 2002 - 13:30:06 MST
Zong-Liang:
We run CCM3 on a PC cluster, and can get you some useful information. My
response follows each question.
-----Original Message-----
From: Zong-Liang Yang [mailto:liang@speer.geo.utexas.edu]
Sent: Monday, January 07, 2002 10:03 PM
To: ccm-users@ncar.ucar.edu
Subject: Running CCM3 on PC Clusters
Dear CCM Users,
I'd like to build a PC cluster for running CCM3
or CSM. I'd appreciate advice concerning the questions
as follows
1) To have a 50-year simulation @T42 resolution with
prescribed SSTs, what is a minimum number of nodes
that I need to have (e.g., 8 nodes, 16 nodes,...)
It depends on how patient you are. You can run a 50-year T42 L18 simulation
in roughly 120 days on a single 1 Ghz Athlon processor. Adding processors
increases the speed, but the increase is sublinear because of communication
overhead. The communication overhead depends on your communication hardware.
Fast ethernet provides cheap communication, but the simulation speedup
saturates at about a factor of four using 8 processors (depending on
processor speed). Myrinet provides improved scaling with processors, but
costs about $1300 per node, plus about $5k for a switch. Another
communication option is Scali-Dolphin, which costs about the same as Myrinet
(per node) and is about as fast. For a small cluster it will be cheaper than
Myrinet because it doesn't use a switch. We are presently testing
Scali-Dolphin on one of our clusters.
2) What are the recommended specifications (CPUS, hard disks ...)?
I've found the Athlon cpus to provide better performance/cost than pentium.
You'll want about 100 Gb disk on your head node, and maybe 20 Gb on the
slave nodes. The head node should have 512 mb RAM, and the slaves 256 mb
RAM. You'll also need the Portland Group Cluster Development software and of
course mpi. Don't forget UPS power.
3) How much do they cost?
Single cpu slave nodes will cost less than $1k. The head node will be
somewhat more. You'll also need a switch, which is about $2k for ethernet
and about $5k for Myrinet. Plus $1300 per node for Myrinet cards/cables. As
you can see, the communication can cost as much as the compute nodes. So you
might consider dual processor nodes if you go with myrinet. They'll cost you
$2-3k per node, but provide very fast communication between processors
within nodes. I can suggest a few cheap but reliable vendors. The Portland
Group Cluster Development kit costs about $5k. mpi comes loaded free on
clusters.
4) What does it take to manage a PC cluster? Does it need
a dedicated computer technician?
Not dedicated, but competant. Our lab technicians manage our clusters for
$600 per month.
5) Is it possible to run the full version of CSM
using a PC cluster?
That is coming with the release of the next generation of CSM.
6) Can you run a regional climate model (say, MM5) nested with
CCM3?
MM5 is normally run off line, with boundary conditions derived from CCM
history. We have codes for producing the boundary conditions. MM5 runs very
well on PC clusters.
Many thanks.
Liang
---------------------------------
Dr. Zong-Liang Yang
Dept. of Geological Sciences
The University of Texas at Austin
Austin, TX 78712-1101
phone: (512) 471-3824
fax: (512) 471-9425
Email: liang@mail.utexas.edu
URL: http://www.geo.utexas.edu/climate
This archive was generated by hypermail 2b27 : Tue Jan 22 2002 - 10:02:33 MST