BIONIC work units for global climate prediction are too huge, 500 hrs / workunit!

[HireMe.geek.nz]

BIONIC work units for global climate prediction are too huge, 500 hrs /
workunit!

[Martin 53N 1W]

Post by HireMe.geek.nz
BIONIC work units for global climate prediction are too huge, 500 hrs /
workunit!

Yes, they are huge. You get very clearly warned of the commitment
necessary to complete scientifically useful results.

You do get credit 'trickled' to keep the stats junkies happy with
regular increments (:-))


For the amount of working data required with the simulation, breaking
apart the simulation across multiple work units and clients is too
cumbersome a problem. The present idea works well enough. As PCs get
still faster, the 'hugeness' will soon diminish.

Happy crunchin',
Martin

[Rolf Leggewie]

Post by Martin 53N 1W

Post by HireMe.geek.nz
BIONIC work units for global climate prediction are too huge, 500 hrs
/ workunit!

Yes, they are huge. You get very clearly warned of the commitment
necessary to complete scientifically useful results.

Is it possible to run climateprection.net models on an openMosix
cluster? IOW will the BOINC SW run in several processes (not just
threads) on a Linux client?

Is there any venture similar to climateprediction.net (global climate)?

[Martin 53N 1W]

Rolf Leggewie wrote:
[...]

Post by Rolf Leggewie
Is it possible to run climateprection.net models on an openMosix
cluster? IOW will the BOINC SW run in several processes (not just
threads) on a Linux client?

You can tell the boinc software to utilise multiple CPUs on a machine.
Multiple clients are then started for those CPUs.


The idea of boinc is to easily exploit parallelism at the coarse grained
level of each WU. If you want finer parallelism, you must create smaller
WUs.

However, special clients could exploit parallelisms within a WU. I would
expect some extra features would need to be added to boinc to properly
manage such clients.

One of the ideas behind boinc is to keep it simple so that many projects
can quickly get coded to share in the online world's idle CPU resource.
Hence the use of well defined (compute independent) easily managed "WU"s.

See:
http://boinc.berkeley.edu/

Post by Rolf Leggewie
Is there any venture similar to climateprediction.net (global climate)?

Check:
http://boinc.berkeley.edu/
http://www.aspenleaf.com/distributed/distrib-projects.html


Good luck,
Martin

[Rolf Leggewie]

Martin,

thank you for the quick reply.

Post by Rolf Leggewie
Is it possible to run climateprection.net models on an openMosix
cluster? IOW will the BOINC SW run in several processes (not just
threads) on a Linux client?

If you want finer parallelism, you must create smaller WUs.

So I assume, the answer is no? Of course it will run on openMosix but
will not be able to benefit from it any more than what would have been
possible with just starting one copy of climateprediction.net each.
This negates the whole idea of the openMosix cluster, or am I mistaken?

The reason I was asking about whether BOINC will start several processes
is that with the huge demands of climateprediction.net it would be very
nice to throw some extra computing power towards it from time to time to
finish SOONER (in case the deadline is approaching for example). If I
need to start one WU on each node, I will at best finish at the same
time, in the worst (and most likely) case, it will take longer,
especially if the extra CPUs are not as powerful. Of course, while it
will have taken longer, I would have completed one WU for each node
instead of just one WU for the whole cluster. I'd still be interested
to have one cluster tackle only one WU at a time. IMO this makes more
sense.

Best

Rolf

[Martin 53N 1W]

Post by Rolf Leggewie
Martin,

[...]

Post by Rolf Leggewie

If you want finer parallelism, you must create smaller WUs.

So I assume, the answer is no? Of course it will run on openMosix but
will not be able to benefit from it any more than what would have been
possible with just starting one copy of climateprediction.net each.

Correct.

Post by Rolf Leggewie
This
negates the whole idea of the openMosix cluster, or am I mistaken?

Well, you still get parallelism in that you get 'n' simulations
completed. You just get 'n' results after the time needed for a
simulation rather than one result after only 'sim-time/n'.

Post by Rolf Leggewie
The reason I was asking about whether BOINC will start several processes
is that with the huge demands of climateprediction.net it would be very
nice to throw some extra computing power towards it from time to time to
finish SOONER (in case the deadline is approaching for example). If I

Nice but fast returns are not required. The deadlines are typically one
YEAR or so. The main danger is whether you upgrade to new hardware
before a simulation completes!

You could still have a cpdn client as a background task on each node.
Your only penalty is if cpdn causes memory swapping for the main jobs.


[...]

Post by Rolf Leggewie
instead of just one WU for the whole cluster. I'd still be interested
to have one cluster tackle only one WU at a time. IMO this makes more
sense.

There would be a lot of work required to parallelise the client and
balance the algorithm communication requirements between multiple
processors. Likely, hardware advances will make your best efforts
obsolete before your first sim can be run and validated!

One /big/ performance boost might be to utilise GPU vector processing on
cheap fast PC graphics cards...


Regards,
Martin

[Rolf Leggewie]

Post by Martin 53N 1W
Nice but fast returns are not required. The deadlines are typically one
YEAR or so.

Good. I had gotten the false impression from somewhere that they were
much shorter.

Post by Martin 53N 1W
The main danger is whether you upgrade to new hardware
before a simulation completes!

This would not be a problem with an openMosix cluster ;-) You can just
add and remove computers at will.

Post by Martin 53N 1W
There would be a lot of work required to parallelise the client and
balance the algorithm communication requirements between multiple
processors.

I am not so sure. I think you interpret an openMosix cluster more along
the lines of a Beowulf cluster where changes to the source are necessary
to take advantage of the parallelism. The beauty of openmosix though is
that there is no need to make special changes in the software as long as
it spawns several processes(!). This is well explained at the openMosix
FAQ http://howto.x-tend.be/openMosixWiki/index.php/FAQ, specifically

- What is openMosix useful for?
- What is openMosix not useful for?:
- Generally, how do I write an openMosix-aware program?

There is *no* need to do things like message passing to benefit from the
parallelism of an openMosix cluster. All that is necessary is that the
process spawns several sub-processes which will then automatically
migrate over the openMosix cluster.

Post by Martin 53N 1W
One /big/ performance boost might be to utilise GPU vector processing on
cheap fast PC graphics cards...

That would probably indeed require some changes to the source code and
logic.

Best

Rolf

[Martin 53N 1W]

Rolf Leggewie wrote:
[...]

Post by Rolf Leggewie
I am not so sure. I think you interpret an openMosix cluster more along
the lines of a Beowulf cluster where changes to the source are necessary
to take advantage of the parallelism. The beauty of openmosix though is
that there is no need to make special changes in the software as long as
it spawns several processes(!). This is well explained at the openMosix
FAQ http://howto.x-tend.be/openMosixWiki/index.php/FAQ, specifically

Good point!

Post by Rolf Leggewie

Post by Martin 53N 1W
One /big/ performance boost might be to utilise GPU vector processing
on cheap fast PC graphics cards...

That would probably indeed require some changes to the source code and
logic.

Yes, and it is already being tried. The hoped for performance gains are
easily worth the effort.


Regards,
Martin

[Rolf Leggewie]

Post by Martin 53N 1W

Post by Rolf Leggewie
The beauty of
openmosix though is that there is no need to make special changes in
the software as long as it spawns several processes(!). This is well
explained at the openMosix FAQ
http://howto.x-tend.be/openMosixWiki/index.php/FAQ, specifically

Good point!

Do you happen to know whether BOINC itself can spawn several
subprocesses or whether that depends on the project software?
IOW, should I contact the BOINC developers or the
climateprediction.net developers about this?

Best

Rolf

[Martin 53N 1W]

Rolf Leggewie wrote:
[...]

Post by Rolf Leggewie
Do you happen to know whether BOINC itself can spawn several
subprocesses or whether that depends on the project software?

That depends on what you mean exactly...

Post by Rolf Leggewie
IOW, should I contact the BOINC developers or the
climateprediction.net developers about this?

You could try reminding them of Openmosix. However, they have a highly
parallel system that works well and they are still busy with
developments to support the 99.99% of non-mosix users across various
projects.

All the boinc code is open source. You can take a look, or try joining
their mailing list. Or try experimenting and monitor what happens on
your system.

The cpdn client is closed source, but there is no harm in asking if
you're really interested.


Let us know what you find!

Good luck,
Martin

[Rolf Leggewie]

Post by Martin 53N 1W

Post by Rolf Leggewie
Do you happen to know whether BOINC itself can spawn several
subprocesses or whether that depends on the project software?

That depends on what you mean exactly...

I tried and found out that BOINC runs CPDN as a single process. My
question is whether BOINC or the CPDN client were the appropriate
targets in trying to have that software run in several processes. (the
concept of process vs. thread is well explained in the openMosix FAQ and
is probably familiar terrain to most programmers)

Post by Martin 53N 1W
Let us know what you find!

http://www.ssl.berkeley.edu/pipermail/boinc_projects/2004-November/000635.html

et ff.

I will probably raise the question on
http://news.gmane.org/gmane.linux.cluster.openmosix.general one of the
next days.