Re: distributed processing ad multiThreading
Re: distributed processing ad multiThreading
- Subject: Re: distributed processing ad multiThreading
- From: Martin Reed <email@hidden>
- Date: Mon, 17 Sep 2001 12:02:00 +0100
Robert,
On 17/9/01 11:34, "Robert S Goldsmith" <email@hidden> wrote:
<snip>
>
The problem I have is that I would like to try and
>
parallelise the evaluation process.
>
>
I have two options. I have access to a 4 processor solaris
>
machine and a duel processor G4 if I want to try threading
>
in an SMP enviroment and I have access to about 120 solaris
>
workstations if I want to go for distributed processing.
>
>
SMP I don't think will be much of a problem to try - except
>
the Apple docs on NSThread etc. are a little thin on the
>
ground. However, with the distributed processing, how would
>
i go about:
>
>
1) launching 'threads' on all the machines
>
2) getting them to communicate small and large abounts of data
>
>
As you can understand, I do not want a huge overhead because
>
time is an important concideration.
I'll take a nibble at your distributed solution (which seems the way to go
if you can). I know of this as "coarse grain parallelism". First off, I'm
assuming that you can partition your algorithm such that the *significant*
overheads of distribution don't kill your efficiency. To put that in number,
if you can make each chunk ("work unit") of distributed work last at least
10s of seconds (and preferably 10s or 100s of minutes) then I would say you
have a winner. A work unit could be several be a batch of several smaller
computations. Also, I have to assume that the volume of data being thrown
around a network is reasonable - say O(100kB) per work unit.
If you actually need fine grained activity, perhaps because the result of
each short computation affects the input to the next, then stop reading and
hit delete :-)
Given all that, you basically need a (preferably cross-platform) mechanism
for running code on other systems which will allow network comms. Yes, you
will need to install and maintain network aware programs on each client
system - is that viable? Basic Unix networking skills would be required to
get comms in place. Then you need to agree on how to package up data and
results for transmission over the network. I'd look towards XML as an
infrastructure for that. You might even want to look to SOAP (plug these
words in to google.com if they are not familiar). Obviously you'll need a
controlling program to dish out work and save results away.
One possibility (especially if you are in Education ie cheap licences) would
be to look at Apple's WebObjects. It won't write your solution for you, but
it *would* give you a relatively efficient infrastructure for distributing
an application on a cross platform basis, because that is inherent in how WO
does scalability.
Just some initial thoughts, not knowing quite where you are coming from or
looking for.
cheers
martin