Run airflow

Run airflow#

To play with airflow we run airflow. In this section I will describe how to get airflow up in a docker container, and by default I will use this method in any airflow related questions onthis site.

Image#

The following docker file should be used to build the image.

Features an image built from this dockerfile:

  • Admin username is admin and password is admin;

  • Cofigure disables automatic loading of example dags, so it will be more convenient to concentrate on writing your own dags.

%%writefile run_airflow/dockerfile
FROM python:3.10

# Script for airflow installation
# described below
COPY install_airflow.sh install_airflow.sh
RUN bash install_airflow.sh

# we need to create user in other case
# airflow will create it by itself with
# random password
# for some reason it asks to run airflow db init
# before creating a user
RUN airflow db init; \
    airflow users create \
        --username admin \
        --password admin \
        --firstname Fedor \
        --lastname Kobak \
        --role Admin \
        --email spiderman@superhero.org;

# Here is the command that replaces the load_examples
# parameter from True to False. So if you
# airflow, you won't get any examples in the list of dags.
RUN sed -i 's/load_examples = True/load_examples = False/g' /root/airflow/airflow.cfg

CMD ["airflow", "standalone"]

Overwriting run_airflow/dockerfile

install_airflow.sh just copied from airflow quick start page.

%%writefile run_airflow/isntall_airflow.sh
AIRFLOW_VERSION=2.7.1

# Extract the version of Python you have installed. If you're currently using Python 3.11 you may want to set this manually as noted above, Python 3.11 is not yet supported.
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example this would install 2.7.1 with python 3.8: https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.8.txt

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Writing run_airflow/isntall_airflow.sh

Start the container#

In the following cell command to start the container. We need port 8080 to get access to the airflow browser client.

!docker run -itd --rm\
    --name start_airflow\
    -p 8080:8080\
    airflow_tests &> /dev/null

Now you can try localhost:8080 on your browser (use username - admin and password - admin).

Don’t forget to stop the container when you don’t need it.

!docker stop start_airflow &> /dev/null
Contents