Can Python do parallel computing?

Python can do parallel computing. The following is the relevant introduction:

1. Overview

Parallel Python is a python module that provides a mechanism for parallel execution of python code on SMPs (systems with multiple processors or multi-cores) and clusters (computers connected through a network). It is lightweight, easy to install and integrate with other python software. Parallel Python is an open source and cross-platform module written in pure Python. 2. Features

Execute python code in parallel on SMP and clusters

Easy to understand and implement Job-based parallelization technology (easy to convert serial applications in parallel)

Automatic detection of optimal configuration (number of worker processes is set to the number of effective processors by default)

Dynamic processor allocation (number of worker processes can be changed at runtime)

Low overhead for subsequent jobs with the same functionality (implement transparent caching to reduce overhead)

Dynamic load balancing (jobs are distributed across processors while running)

Fault tolerance (if one of Node failure, tasks are rescheduled on other nodes)

Automatic discovery of computing resources

Dynamic allocation of computing resources (the result of automatic discovery and fault tolerance)

Network connection SHA-based authentication

Cross-platform portability and interoperability (Windows, Linux, Unix, Mac OS X)

Cross-architecture portability and interoperability (x86, x86 -64 etc.)

Open source

Related recommendations: "python video tutorial"

3. Motivation

Nowadays, software written in python is used in many applications, including business logic, data analysis and scientific computing. This, together with the wide availability of SMP computers (multi-processor or multi-core) and clusters (computers connected via a network) on the market, creates a need for parallel execution of python code.

The simplest and most common way to write parallel applications for SMP computers is to use threads. Although, if the application is computationally bound using threads or the threaded python module will not allow running python bytecode in parallel. The reason is that the python interpreter uses the GIL (Global Interpreter Lock) for internal accounting. This lock allows only one python bytecode instruction to be executed at a time, even on SMP machines.

The PP module overcomes this limitation and provides an easy way to write parallel python applications. Internally ppsmp uses processes and IPC (inter-process communication) to organize parallel computations. All details and complexities of the latter are completely taken care of, the application just submits the job and retrieves its results (the simplest way to write parallel applications).

To make things even better, software written in PP works in parallel, even on many computers connected via a local network or the Internet. Cross-platform portability and dynamic load balancing allow PP to efficiently parallelize computing even on heterogeneous and multi-platform clusters.

4. Installation

Any platform: Download the module archive and extract it to a local directory. Run the installation script: python setup.py install

Windows: Download and execute the Windows installer binary.

5. Example

import math, sys, time
import pp
def isprime(n):
    """Returns True if n is prime and False otherwise"""
    if not isinstance(n, int):
        raise TypeError("argument passed to is_prime is not of 'int' type")
    if n < 2:
        return False
    if n == 2:
        return True
    max = int(math.ceil(math.sqrt(n)))
    i = 2
    while i <= max:
        if n % i == 0:
            return False
        i += 1
    return True
def sum_primes(n):
    """Calculates sum of all primes below given integer n"""
    return sum([x for x in xrange(2,n) if isprime(x)])
print """Usage: python sum_primes.py [ncpus]
    [ncpus] - the number of workers to run in parallel, 
    if omitted it will be set to the number of processors in the system
"""
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)
if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"
# Submit a job of calulating sum_primes(100) for execution. 
# sum_primes - the function
# (100,) - tuple with arguments for sum_primes
# (isprime,) - tuple with functions on which function sum_primes depends
# ("math",) - tuple with module names which must be imported before sum_primes execution
# Execution starts as soon as one of the workers will become available
job1 = job_server.submit(sum_primes, (100,), (isprime,), ("math",))
# Retrieves the result calculated by job1
# The value of job1() is the same as sum_primes(100)
# If the job has not been finished yet, execution will wait here until result is available
result = job1()
print "Sum of primes below 100 is", result
start_time = time.time()
# The following submits 8 jobs and then retrieves the results
inputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700)
jobs = [(input, job_server.submit(sum_primes,(input,), (isprime,), ("math",))) for input in inputs]
for input, job in jobs:
    print "Sum of primes below", input, "is", job()
print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

The above is the detailed content of Can Python do parallel computing?. For more information, please follow other related articles on the PHP Chinese website!