Concurrency

Concurrency#

Concurrent execution means that multiple tasks are in progress at the same time, but not necessarily running at the exact same time.

Check the offcial python reference on Concurrent Execution.

Asyncio#

Python offers tools for organizing asynchronous programming. The main components are the async and await keywords, along with the asyncio library.

Here are a few basic terms that you may find useful in an asyncio context:

  • Coroutine: A function defined with async def, which returns a coroutine object.

  • Task: A wrapper around a coroutine that runs it concurrently in the event loop.

  • Event Loop: The core of asyncio. It runs and manages async tasks and callbacks.

The following cell demonstrates a classical asynchronous program. It uses the functions fun1 and fun2, which are essentially the same - each prints a number in a loop. Each record identifies which function produced it.

import asyncio
from random import uniform

async def fun1():
    for i in range(10):
        print(f"fun 1|iteration {i}")
        await asyncio.sleep(uniform(0, 0.1))

async def fun2():
    for i in range(10):
        print(f"fun 2|iteration {i}")
        await asyncio.sleep(uniform(0, 0.1))

print("=========begin cycle=========")
await asyncio.gather(fun1(), fun2())
print("=========end cycle=========")

=========begin cycle=========
fun 1|iteration 0
fun 2|iteration 0
fun 1|iteration 1
fun 1|iteration 2
fun 2|iteration 1
fun 2|iteration 2
fun 1|iteration 3
fun 2|iteration 3
fun 2|iteration 4
fun 1|iteration 4
fun 2|iteration 5
fun 1|iteration 5
fun 2|iteration 6
fun 2|iteration 7
fun 1|iteration 6
fun 1|iteration 7
fun 2|iteration 8
fun 1|iteration 8
fun 1|iteration 9
fun 2|iteration 9
=========end cycle=========

The output clearly shows that the functions are executed simultaneously, rather than one by one.

Multiprocessing#

Runs a sompletely spearate process in the system and executes the code you need there. Find our more in:

The following cell defines a funciton that prints the id of the current process. And run it in the main process and in multiprocessing.

import os
import multiprocessing

def print_pid():
    print(os.getpid())

print_pid()

p = multiprocessing.Process(target=print_pid)
p.start()
p.join()

22512
23316

As the result there are two different numbers, which showe that the second execution of the function was recognized by the system as completely different program.

Subprocess#

The subprocess package enables the spwaning of processes that are called by commands available in the parent shell. Check the subprocess - Subprocess management page.

The following table shows the most popular elements of the subprocess package:

Element

Description

run function

Runs the process and blocks the parent process

Popen function

Runs the process but the python process continues to execution

The following cell shows the output of the pwd command executed from the parent shell.

import subprocess
subprocess.run(["pwd"])

/home/user/Documents/code/python/standard_library

CompletedProcess(args=['pwd'], returncode=0)
Contents