Kernel

Kernel#

A Jupyter kernel is a special program that procedures code it got from the front end and pushes result back to the frontend.

import sys
from src.run_jupyter_kernel import IPKernelAppProcess, get_messages

sys.path.append("/tmp")

Manipulate kernels#

The jupyter kernelspec command is designed to work manipulate with jupyter kernels. Use the jupyter kernelspec list command to list available in the current jupyter setup kernels.

!jupyter kernelspec list

Available kernels:
  sql_kernel         /home/fedor/.virtualenvs/knowledge/share/jupyter/kernels/SQL_kernel
  postgres_kernel    /home/fedor/.virtualenvs/knowledge/share/jupyter/kernels/postgres_kernel
  python3            /home/fedor/.virtualenvs/knowledge/share/jupyter/kernels/python3
  universal_sql      /home/fedor/.virtualenvs/knowledge/share/jupyter/kernels/universal_sql
  bash               /home/fedor/.local/share/jupyter/kernels/bash

CLI#

jupyter-kernel tool comes with jupyter_client package.

It confuses me that jupyter_client provides the jupyter-kernel CLI tool. If you have the same doubts, the following cells will prove it.

The code in the following cell shows the file to which jupyter-kernel refers in the system.

!which jupyter-kernel

/usr/local/bin/jupyter-kernel

And behind it is Python code that refers to jupyter_client.kernelapp.main.

!cat $(which jupyter-kernel)

#!/usr/local/bin/python3.12
# -*- coding: utf-8 -*-
import re
import sys
from jupyter_client.kernelapp import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Run from jupyter#

For the purposes of this project, it’s useful to be able to run jupyter kernels in jupyter notebooks. There are a number of problems associated with this, mainly related mainly to the fact that jupyter notebook uses it’s own kernel to work, and this fact blocks the creation of new jupyter kernels. Read more about this and possible solutions in the related page.

As a ready solution to be able to experiment with different jupyter kernel use src.run_jupyter_kernel.IPKernelAppProcess function.

The following cell shows the process of running of the kernel with src.run_jupyter_kernel.IPKernelAppProcess.

p = IPKernelAppProcess("/tmp/example_connection_file.json")

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing /tmp/example_connection_file.json

As a result, we got a typical ipykernel starting log.

Build kernel#

There are three ways to create your own Jupyter kernel; check more here. This page focuses on the simplest method: simple Python wrapper kernels.

Kernel is actually a subclass of the ipykernel.kernelbase.Kernel, which implements custom logic in for the kernel.

The following cell shows the implementation of the kernel. This is kernel that to any execution sends to the client {'name': 'stdout', 'text': <input>} where input is a code that is required to be executed.

%%writefile /tmp/my_kernel.py
from ipykernel.kernelbase import Kernel

class EchoKernel(Kernel):
    implementation = 'Echo'
    implementation_version = '1.0'
    language_info = {
        'name': 'Any text',
        'mimetype': 'text/plain',
        'file_extension': '.txt',
    }
    banner = "Echo kernel - as useful as a parrot"

    def do_execute(
        self,
        code,
        silent,
        store_history=True,
        user_expressions=None,
        allow_stdin=False
    ):
        if not silent:
            stream_content = {'name': 'stdout', 'text': code}
            self.send_response(self.iopub_socket, 'stream', stream_content)

        return {
            'status': 'ok',
            'execution_count': self.execution_count,
            'payload': [],
            'user_expressions': {},
        }

if __name__ == '__main__':
    from ipykernel.kernelapp import IPKernelApp
    IPKernelApp.launch_instance(kernel_class=EchoKernel)

Overwriting /tmp/my_kernel.py

The following cell runs EchoKernel.

from my_kernel import EchoKernel
p = IPKernelAppProcess(
    connection_file="/tmp/echo_kernel.json",
    kernel_class=EchoKernel
)

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing /tmp/echo_kernel.json

The next code runs this is message in the kernel we just created.

messages = get_messages(
    connection_file="/tmp/echo_kernel.json",
    code="this is message"
)
for msg in messages:
    if msg["header"]["msg_type"] == "stream":
        ans = msg["content"]
ans["text"]

'this is message'

And we got back this is message just as specified in the kernel.

Register kernel#

In the last step of creating jupyter kernels is registering them - so you would be able to run kernel from one of jupyter’s CLI/GUI interfaces. Registering a kernel involves adding a json configuration file that specifies how to run the kernel have to runned and some additional information about the kernel.

The following cell shows kernels already added to the environment.

!jupyter kernelspec list

Available kernels:
  bash       /usr/local/share/jupyter/kernels/bash
  python3    /usr/local/share/jupyter/kernels/python3

At the same path, we need to create a folder for the kernel and place kernel.json inside it. The argv key in kernel.json should execute the module we created earlier.

!mkdir -p /usr/local/share/jupyter/kernels/echo

%%writefile /usr/local/share/jupyter/kernels/echo/kernel.json
{
    "argv":[
        "python3", 
        "/tmp/my_kernel.py", 
        "-f", 
        "{connection_file}"
    ],
    "display_name":"Echo"
}

Writing /usr/local/share/jupyter/kernels/echo/kernel.json

Now the result of the jupyter kernelspec list command is the kernel you just created.

!jupyter kernelspec list

Available kernels:
  bash       /usr/local/share/jupyter/kernels/bash
  echo       /usr/local/share/jupyter/kernels/echo
  python3    /usr/local/share/jupyter/kernels/python3

To ensure that everything works correct let’s try to run echo kernel and send hello, parrot? to be executed.

from jupyter_client import KernelManager
km = KernelManager(kernel_name="echo")
km.start_kernel()

kc = km.client()
kc.start_channels()

kc.execute("hello, parrot?", reply=True)
while True:
   msg = kc.get_iopub_msg(timeout=5)
   if msg['msg_type'] == 'stream':
        break

kc.shutdown(reply=True)
km.shutdown_kernel(now=True)

As a result, the kernel returns exactly what we’ve sent it - exactly as specified in its logic.

msg["content"]["text"]

'hello, parrot?'

Even better, you can run a Jupyter Notebook file, select echo from the list of kernels, and use it just like any other notebook.

Contents