Multicast peer discovery

[Andreas]

Hello,
the company I work for make a program that performs heavy computations.
To speed things up it is possible to distribute the work onto other
machines.
This is done by starting a separate process with ssh on the remote
machines.
Maintaining the list of available remote machines however is a
difficult task.

I have therefore been thinking about each available machine notifying
all other machines on the subnet about its availability trough the use
of multicasting.

What would be the drawbacks to such an approach?
Have anyone implemented such a system?

Thanks in advance for any response!
Andreas Werner Paulsen

[stephen]

Post by Andreas
Hello,
the company I work for make a program that performs heavy computations.
To speed things up it is possible to distribute the work onto other
machines.
This is done by starting a separate process with ssh on the remote
machines.
Maintaining the list of available remote machines however is a
difficult task.
I have therefore been thinking about each available machine notifying
all other machines on the subnet about its availability trough the use
of multicasting.
What would be the drawbacks to such an approach?
Have anyone implemented such a system?

have a look at protocols that already use this approach - OSPF neighbour
discovery on a multicast LAN is 1 place to start.

Post by Andreas
Thanks in advance for any response!
Andreas Werner Paulsen

[Xabi]

You also could issue a cron task on every available machine that
informs you on a period basis of machine's availability.

I don't know what type of tasks are you distributing and I'm by no
means an expert on this topic but you could implement a middleware
layer using the boss-worker pattern.

Post by stephen

Post by Andreas
Hello,
the company I work for make a program that performs heavy computations.
To speed things up it is possible to distribute the work onto other
machines.
This is done by starting a separate process with ssh on the remote
machines.
Maintaining the list of available remote machines however is a
difficult task.
I have therefore been thinking about each available machine notifying
all other machines on the subnet about its availability trough the use
of multicasting.
What would be the drawbacks to such an approach?
Have anyone implemented such a system?

have a look at protocols that already use this approach - OSPF neighbour
discovery on a multicast LAN is 1 place to start.

Post by Andreas
Thanks in advance for any response!
Andreas Werner Paulsen

--
Regards

[Chris Wilk]

Hello Andreas,

Post by Andreas
the company I work for make a program that performs heavy computations.
To speed things up it is possible to distribute the work onto other
machines.
This is done by starting a separate process with ssh on the remote
machines.
Maintaining the list of available remote machines however is a
difficult task.
I have therefore been thinking about each available machine notifying
all other machines on the subnet about its availability trough the use
of multicasting.
What would be the drawbacks to such an approach?
Have anyone implemented such a system?

Why don't you use a DRM / Job Scheduler system
<http://en.wikipedia.org/wiki/Distributed_Resource_Manager>?
It would take away most of distribution, monitoring and synchronisation
burden from your shoulders.

Best,
Chris