Run from jupyter

Run from jupyter#

This page give an overview of the details of running a jupyter kernel from the jupyter notebook. It doesn’t have much practical sence but extremely usefull for purposes of this project.

import sys
import importlib
import multiprocessing
from ipykernel.kernelbase import Kernel
from ipykernel.kernelapp import IPKernelApp
from jupyter_client.blocking import BlockingKernelClient

sys.path.append("/tmp")

Create new instance#

The main problem associated with running kernel from jupyter environment - is that in execution environment already created kernel object. It follows singleton pattern which means that it has one instance over whole program, so if you try to create a new kernel you’ll just get the same object - not create a new one. This can be overcome by running the kernel in the new process.

Note The process must be started in spawn or forkserver context so that it not to share resources with the parent process. For more details on this issue, see description on the different startup methods in the multiprocessing package.

The following cell shows the instance of the Kernel class.

Kernel.instance()

<ipykernel.ipkernel.IPythonKernel at 0x7377f016b650>

As the result we got reference to the IPythonKernel - this is the kernel of the environment this notebook runned in.

So if you directly call IPKernelApp.launch_instance directly, you will just get error related to the fact that resources the kernel is trying to use are occupied, as shown in the next cell.

try:
    IPKernelApp.launch_instance()
except Exception as e:
    print(e)

init_sockets cannot be called twice!

The next code creates prints Kernel.instance() from the separate process.

%%writefile /tmp/show_kernel_instance.py
from ipykernel.kernelbase import Kernel

def process_target():
    print(Kernel.instance())

Overwriting /tmp/show_kernel_instance.py

from show_kernel_instance import process_target
context = multiprocessing.get_context("spawn")
p = context.Process(target=process_target)
p.start()

<ipykernel.kernelbase.Kernel object at 0x77d29a450d70>

In the separate process, Kernel.instance refers to the different object.

CLI arguments#

Another issue related to starting the kernel in the jupyter notebook is that it can use command line arguments used by the parent process. Specifically, if a connection file was specified when setting up the kernel’s new IPKernelApp object, these parameters will be inherited by the child process, and IPKernelApp will read and use these parameter.

  • The most obvious solution is to simply skip the parameters of the parent process sys.argv = ['python3'].

  • In some cases it’s more beneficial to pass parameters directly as list[str] to IPKernelApp.launch_instance which will be used later in IPKernelApp.initialize.

Note The following example was run in vscode jupyter notebook implementation and in different implementation result may be different.

Two subsequent cells start a new process that prints sys.argv and calls the IPKernelApp.parse_command_line method which is an important part of the IPKernelApp.launch_instance instance. Dsiplays the connection_file attribute of the IPKernelApp.

%%writefile /tmp/show_arv.py
import sys
from ipykernel.kernelapp import IPKernelApp

def process_target():
    print(sys.argv)
    app = IPKernelApp.instance()
    app.parse_command_line()
    print("connection file:", app.connection_file)

Overwriting /tmp/show_arv.py

import show_arv
context = multiprocessing.get_context("spawn")
p = context.Process(target=show_arv.process_target)
p.start()

['/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/ipykernel_launcher.py', '--f=/run/user/1000/jupyter/runtime/kernel-v3230a012ea78203d37cfc76381637f9f4dec70282.json']
connection file: /run/user/1000/jupyter/runtime/kernel-v3230a012ea78203d37cfc76381637f9f4dec70282.json

As a result, IPKernelApp will use the same connection file as the main kernel.

The following cell shows the result of parsing_command_line after resetting of the command line arguments from the parent process.

%%writefile /tmp/show_arv.py
import sys
from ipykernel.kernelapp import IPKernelApp

def process_target():
    sys.argv = ["python3"]
    app = IPKernelApp.instance()
    app.parse_command_line()
    print("connection file:", app.connection_file)

Overwriting /tmp/show_arv.py

importlib.reload(show_arv)
context = multiprocessing.get_context("spawn")
p = context.Process(target=show_arv.process_target)
p.start()

connection file:

As a result, there is no connection file in the child process.

An alternative solution is to pass the arguments as to the system as List[str]. The following cell runs parse_command_line that resets the connection_file.

%%writefile /tmp/show_arv.py
import sys
from ipykernel.kernelapp import IPKernelApp

def process_target():
    app = IPKernelApp.instance()
    app.parse_command_line(["--f", "value"])
    print("connection file:", app.connection_file)

Overwriting /tmp/show_arv.py

importlib.reload(show_arv)
context = multiprocessing.get_context("spawn")
p = context.Process(target=show_arv.process_target)
p.start()

connection file: value

As a result, the connection_file attribute of the IPKernelApp takes the values specified in the command line argument list.

Implementation#

This section describes the minimal working setup of the code that runs the kernel inside jupyter notebook.

The following cell defines the target for the process. It is only the IPKernelApp.launch_instance that redefines the connection file.

%%writefile /tmp/run_kernel.py
from ipykernel.kernelapp import IPKernelApp

def process_target():
    IPKernelApp.launch_instance(["-f", "/tmp/example_connection.json"])

Overwriting /tmp/run_kernel.py

The following code runs this kernel in a separate process.

from run_kernel import process_target
context = multiprocessing.get_context("spawn")
p = context.Process(target=process_target)
p.start()

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing /tmp/example_connection.json

To make sure that everything is working properly, the following cell creates a client that connects to this kernel and shows that it’s alive.

client = BlockingKernelClient()
client.load_connection_file("/tmp/example_connection.json")
client.start_channels()
client.is_alive()

True

Special tool#

To make it easier to run jupyter kernels from juptyer notebook developed, a special tool has been developed. Use src.run_jupyter_kernel.IPKernelAppProcess. It takes arguments:

  • connection_file: Path to the kernel connection file.

  • kernel_class: Class of the kernel that determines the behaviour of the kernel.

The following cell runs kernel starts a new jupyter kernel.

from ipykernel.ipkernel import IPythonKernel
from src.run_jupyter_kernel import IPKernelAppProcess

p = IPKernelAppProcess(
    connection_file="/tmp/run_kernel_example.json",
    kernel_class=IPythonKernel
)

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing /tmp/run_kernel_example.json

The following cell creates a client that proves that kernels works fine.

client = BlockingKernelClient()
client.load_connection_file("/tmp/run_kernel_example.json")
client.start_channels()
client.is_alive()

True

Note You cannot create two kernels on the same connection file. The following cell shows an attempt to create one.

IPKernelAppProcess(connection_file="/tmp/run_kernel_example.json")

<utils.run_jupyter_kernel.IPKernelAppProcess at 0x7d4ddafe00e0>

Process SpawnProcess-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fedor/Documents/python/utils/run_jupyter_kernel.py", line 24, in _run_kernel_target
    IPKernelApp.launch_instance(
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 692, in initialize
    self.init_sockets()
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 331, in init_sockets
    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 253, in _bind_socket
    return self._try_bind_socket(s, port)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 229, in _try_bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "/home/fedor/.virtualenvs/python/lib/python3.12/site-packages/zmq/sugar/socket.py", line 320, in bind
    super().bind(addr)
  File "_zmq.py", line 917, in zmq.backend.cython._zmq.Socket.bind
  File "_zmq.py", line 179, in zmq.backend.cython._zmq._check_rc
zmq.error.ZMQError: Address already in use (addr='tcp://127.0.0.1:33255')

The result there is an error about corrupted resources. But IPKernelAppProcess terminates the process that holds the kernel when the object is deleted. The following cell shows that everything goes fine with creating the new kernel when the object that keeps the previous one is deleted.

del p
IPKernelAppProcess(connection_file="/tmp/run_kernel_example.json")

<utils.run_jupyter_kernel.IPKernelAppProcess at 0x7d4df04ebc80>

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing /tmp/run_kernel_example.json
Contents