DisPyTE - Distributing Python Task Environment
Dispyte is a tool that aims at distributing tasks across a heterogeneous
network. It is written in pure Python. Thus, it can be simultaneously used
on various operating systems. Dispyte has been
successfully tested under GNU/Linux and MS Windows NT (2000 and XP).
Because of its modular object oriented design it can be conveniently
extended: allowing not only for numerous different applications, but also
for integrating alternative communication mechanisms. Owing to its
implicit fault tolerance and load balancing, it can be used as a powerful
alternative for MPI or OpenMP.
Dispyte's core consists of collection of interfaces. Additionally, all
common classes, for network communications or task and receiver
administrations, are fully implemented. The user then has to derive his/her
application-specific implementations from the provided interface, mainly in
order to specify how tasks are ``produced''. In the current implementation
the network communications are handled using remote calls. As its
underlying network engine, Dispyte uses Twisted, an event-driven networking
framework, also implemented in Python.
It also provides implementations for common classes. A Dispyte master is not
only able to re-distribute (unsuccessful) tasks, but also to check for new
available workers at run-time. Thus, allowing for a fairly fault-tolerant
operation, as worker errors do not stop or interrupt the process. Another
feature of Dispyte is its implicit load balance: If the master is waiting
for unfinished tasks and discovers idle workers it re-sends these tasks to
the vacant nodes. Once the master receives a task's result the calculation
processes on all nodes that are redundantly working on the same task are
stopped. These nodes are then available to work on new tasks.
Features:
- pure Python
- easy extensible
- works with heterogenous networks
- implicit load balance
- fail save network communication with Twisted
- runs on GNU/Linux and MS Windows NT (2000 and XP)