Real Time Application.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.8"
    },
    "colab": {
      "name": "Real Time Application.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dh_Sa_FTdqfD"
      },
      "source": [
        "Davide Cremonini 14412 - Davide Sbetti 14032\r\n",
        "\r\n",
        "----\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sz5pU4VVdpDg"
      },
      "source": [
        "# Real Time Application"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tT66-LbKdwBg"
      },
      "source": [
        "In this notebook we would like to apply the previously trained model in real time, on the frames delivered by a webcam. \r\n",
        "\r\n",
        "**NOTE**: this notebook was not executed on Google Colab, since the model was saved and downloaded locally. All necessary files are located in the repository and the notebook can so be executed from there. \r\n",
        "\r\n",
        "We start by importing the necessary libraries. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WxCjOUdWdpEG",
        "outputId": "09c7be94-675d-4feb-cf73-61db27925ddf"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import cv2\n",
        "import csv\n",
        "import numpy as np\n",
        "import time\n",
        "from pygame import mixer"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "pygame 2.0.1 (SDL 2.0.14, Python 3.7.8)\n",
            "Hello from the pygame community. https://www.pygame.org/contribute.html\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acmQ0mked7cQ"
      },
      "source": [
        "We then initialise the mixer object, loading the defined alarm we have chosen to use when drowsiness is detected conan. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNdW3RmwdpES"
      },
      "source": [
        "mixer.init()\n",
        "mixer.music.load(\"kikeriki.mp3\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJDc_4UJeFag"
      },
      "source": [
        "We load then the model we saved previously. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "etIYQTHPdpEV"
      },
      "source": [
        "model = keras.models.load_model('model.h5')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGElVTf9efFI"
      },
      "source": [
        "We load the files for the opencv cascade classifier, files downloaded from the official repository of OpenCv (https://github.com/opencv/opencv/tree/master/data/haarcascades). \r\n",
        "\r\n",
        "We have decided to use the frontalface default, to detect faces, while we opted for two different classifiers in order to detect the two eyes. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U4-Ka3todpEW"
      },
      "source": [
        "xml_path = \"haarcascade_xml/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xzQozI0pdpEY"
      },
      "source": [
        "face_cascade = cv2.CascadeClassifier(xml_path + \"haarcascade_frontalface_default.xml\")\n",
        "right_eye_cascade = cv2.CascadeClassifier(xml_path + \"haarcascade_righteye_2splits.xml\")\n",
        "left_eye_cascade = cv2.CascadeClassifier(xml_path + \"haarcascade_lefteye_2splits.xml\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DalgIQmAggbV"
      },
      "source": [
        "We set the image dimension to 80x80, since this is the input size of the model, and we start the capture. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RlUw7jz6dpEc"
      },
      "source": [
        "dim = (80,80)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VqYZpmzgvk4"
      },
      "source": [
        "Each time we obtain a new frame from the webcam, we convert it to gray, keeping 3 channels, as done also in the training of the model. \r\n",
        "\r\n",
        "We then try to detect a face in the frame. If the face is detected, we split vertically the resulting area in two parts and we apply the single classifiers to each one, in order to detect the eyes. \r\n",
        "\r\n",
        "If we are able to detect both eyes, we \"cut\" the interested area and we resize it to a 80x80 dimension. We standardise the two images (applying so the same transformations of the training phase), passing then them to the model. \r\n",
        "\r\n",
        "If both eyes are closed, we increase a counter of closed eyes, that stores the number of subsequent closed eyes frames. Once the counter reaches a certain threshold, we play the defined alarm. \r\n",
        "\r\n",
        "We reset the counter whenever two consecutive frames show open eyes (around half a second). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOvqh9OYdpEl",
        "outputId": "25781028-c646-466d-9a96-270b8cc408a3"
      },
      "source": [
        "cap = cv2.VideoCapture(0)\n",
        "\n",
        "closed_count = 0\n",
        "frame_count = 0\n",
        "closed_tol_count = 0\n",
        "\n",
        "start = time.time()\n",
        "while(True):\n",
        "    # Capture frame-by-frame\n",
        "    ret, frame = cap.read()\n",
        "    frame_count += 1\n",
        "    \n",
        "    # Our operations on the frame come here\n",
        "    gray_single = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n",
        "    \n",
        "    gray = np.zeros_like(frame)\n",
        "    gray[:,:,0] = gray_single\n",
        "    gray[:,:,1] = gray_single\n",
        "    gray[:,:,2] = gray_single\n",
        "    \n",
        "    faces = face_cascade.detectMultiScale(gray, 1.3, 5)\n",
        "\n",
        "    if len(faces) > 0:\n",
        "        (x,y,w,h) = faces[0]\n",
        "\n",
        "        #img = cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)\n",
        "        half_int = int(np.ceil(w/2))\n",
        "\n",
        "        roi_gray_left = gray[y:y+h, x:x+half_int]\n",
        "        roi_gray_right = gray[y:y+h, x+half_int:x+w]\n",
        "\n",
        "        #frame = cv2.rectangle(frame,(x,y),(x+half_int,y+h),(255,0,0),2) # left part\n",
        "        #frame = cv2.rectangle(frame,(x+half_int,y),(x+w,y+h),(0,255,0),2) # right part\n",
        "\n",
        "        roi_color_left = frame[y:y+h, x:x+half_int]\n",
        "        roi_color_right = frame[y:y+h, x+half_int:x+w]\n",
        "\n",
        "        right_eyes = right_eye_cascade.detectMultiScale(roi_gray_left)\n",
        "        left_eyes = left_eye_cascade.detectMultiScale(roi_gray_right)\n",
        "\n",
        "        # check we have detected something on both sides\n",
        "        if len(right_eyes) > 0 and len(left_eyes) > 0:\n",
        "            (rx,ry,rw,rh) = right_eyes[0]\n",
        "            (lx,ly,lw,lh) = left_eyes[0]\n",
        "\n",
        "\n",
        "            cv2.rectangle(roi_color_left,(rx,ry),(rx+rw,ry+rh),(0,255,0),2)\n",
        "            cv2.rectangle(roi_color_right,(lx,ly),(lx+lw,ly+lh),(0,255,0),2)\n",
        "\n",
        "            #take right eye image\n",
        "            right_eye = roi_gray_left[ry:ry+rh, rx:rx+rw]\n",
        "            left_eye = roi_gray_right[ly:ly+lh, lx:lx+lw]\n",
        "            \n",
        "            #cv2.imshow('frame',left_eye)\n",
        "            \n",
        "            right_eye_resized = cv2.resize(right_eye, dim, interpolation = cv2.INTER_AREA)\n",
        "            left_eye_resized = cv2.resize(left_eye, dim, interpolation = cv2.INTER_AREA)\n",
        "            \n",
        "            right_eye_resized = right_eye_resized/255.0\n",
        "            left_eye_resized = left_eye_resized/255.0\n",
        "            \n",
        "            left_mirror = cv2.flip(left_eye_resized, 1)\n",
        "            #cv2.imshow('frame',left_mirror)\n",
        "            \n",
        "            right_final = tf.data.Dataset.from_tensor_slices([right_eye_resized])\n",
        "            left_final = tf.data.Dataset.from_tensor_slices([left_mirror])\n",
        "            \n",
        "            \n",
        "            right_open = model.predict(right_final.batch(32))\n",
        "            left_open = model.predict(left_final.batch(32))            \n",
        "            \n",
        "            if right_open[0][0] < 0.5 and left_open[0][0] < 0.5:\n",
        "                closed_count += 1\n",
        "                cv2.putText(frame,'CLOSED', \n",
        "                    (500,460), \n",
        "                    cv2.FONT_HERSHEY_SIMPLEX, \n",
        "                    1,\n",
        "                    (0,0,255),\n",
        "                    2)\n",
        "                close_tol_count = 0\n",
        "            elif right_open[0][0] < 0.5 or left_open[0][0] < 0.5:\n",
        "                cv2.putText(frame,'CLOSED', \n",
        "                    (500,460), \n",
        "                    cv2.FONT_HERSHEY_SIMPLEX, \n",
        "                    1,\n",
        "                    (0,0,255),\n",
        "                    2)\n",
        "                #print(\"Closed eyes found\")\n",
        "            else:\n",
        "                closed_tol_count += 1\n",
        "                if closed_tol_count > 2:\n",
        "                    closed_count = 0\n",
        "                #print(\"Open eyes found\")\n",
        "            \n",
        "            if closed_count >= 4 and not mixer.music.get_busy():\n",
        "                mixer.music.play()\n",
        "            \n",
        "    # Display the resulting frame\n",
        "    cv2.imshow('frame',frame)\n",
        "    if cv2.waitKey(1) & 0xFF == ord('q'):\n",
        "        stop = time.time()\n",
        "        break\n",
        "        \n",
        "cap.release()\n",
        "cv2.waitKey(0)\n",
        "cv2.destroyAllWindows()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Right: [[0.99675643]]  Left: [[0.93149817]]\n",
            "Right: [[0.99999726]]  Left: [[0.99997807]]\n",
            "Right: [[0.9999447]]  Left: [[0.9999995]]\n",
            "Right: [[1.]]  Left: [[0.99935263]]\n",
            "Right: [[0.9998214]]  Left: [[1.]]\n",
            "Right: [[0.99998873]]  Left: [[1.]]\n",
            "Right: [[0.9995645]]  Left: [[1.]]\n",
            "Right: [[0.9999842]]  Left: [[0.99999666]]\n",
            "Right: [[0.99999976]]  Left: [[0.9999932]]\n",
            "Right: [[0.99992776]]  Left: [[0.9999964]]\n",
            "Right: [[0.9999335]]  Left: [[0.99999654]]\n",
            "Right: [[0.99988234]]  Left: [[0.99998164]]\n",
            "Right: [[0.99959695]]  Left: [[0.9793924]]\n",
            "Right: [[0.9924703]]  Left: [[0.99545854]]\n",
            "Right: [[0.99981594]]  Left: [[0.9908395]]\n",
            "Right: [[0.9999395]]  Left: [[0.93750596]]\n",
            "Right: [[0.77440953]]  Left: [[0.88221455]]\n",
            "Right: [[0.80850667]]  Left: [[0.72917914]]\n",
            "Right: [[0.6175787]]  Left: [[0.3947398]]\n",
            "Right: [[0.98582107]]  Left: [[0.22462872]]\n",
            "Right: [[0.94399774]]  Left: [[0.15245667]]\n",
            "Right: [[0.9998834]]  Left: [[0.9993943]]\n",
            "Right: [[0.9951675]]  Left: [[0.99918395]]\n",
            "Right: [[0.9987079]]  Left: [[0.9997264]]\n",
            "Right: [[0.99452627]]  Left: [[0.99130887]]\n",
            "Right: [[0.9825563]]  Left: [[0.9615432]]\n",
            "Right: [[0.99724483]]  Left: [[0.9579252]]\n",
            "Right: [[0.9788339]]  Left: [[0.9896512]]\n",
            "Right: [[0.9497014]]  Left: [[0.9998353]]\n",
            "Right: [[0.9317776]]  Left: [[0.8846876]]\n",
            "Right: [[0.93386054]]  Left: [[0.88976824]]\n",
            "Right: [[0.96638775]]  Left: [[0.99885684]]\n",
            "Right: [[0.9759625]]  Left: [[0.99375004]]\n",
            "Right: [[0.93680394]]  Left: [[0.9951527]]\n",
            "Right: [[0.94223166]]  Left: [[0.97443664]]\n",
            "Right: [[0.9448638]]  Left: [[0.9848112]]\n",
            "Right: [[0.9453782]]  Left: [[0.9623014]]\n",
            "Right: [[0.9629195]]  Left: [[0.9539853]]\n",
            "Right: [[0.91644186]]  Left: [[0.38778293]]\n",
            "Right: [[0.9047394]]  Left: [[0.9101336]]\n",
            "Right: [[0.95512533]]  Left: [[0.9988091]]\n",
            "Right: [[0.9715396]]  Left: [[0.9979192]]\n",
            "Right: [[0.99997896]]  Left: [[0.9945974]]\n",
            "Right: [[0.9976944]]  Left: [[0.99914986]]\n",
            "Right: [[0.27200747]]  Left: [[0.07783392]]\n",
            "Right: [[0.676885]]  Left: [[0.691141]]\n",
            "Right: [[0.9494642]]  Left: [[0.17632169]]\n",
            "Right: [[0.9208664]]  Left: [[0.06992662]]\n",
            "Right: [[0.8391921]]  Left: [[0.16804895]]\n",
            "Right: [[0.99999917]]  Left: [[0.9980503]]\n",
            "Right: [[0.99994004]]  Left: [[0.99999136]]\n",
            "Right: [[0.9973124]]  Left: [[0.99992454]]\n",
            "Right: [[0.99994814]]  Left: [[0.9993714]]\n",
            "Right: [[0.9999728]]  Left: [[0.999699]]\n",
            "Right: [[0.99991596]]  Left: [[1.]]\n",
            "Right: [[0.99999976]]  Left: [[0.9997008]]\n",
            "Right: [[0.99999875]]  Left: [[0.9998635]]\n",
            "Right: [[0.9999954]]  Left: [[0.9999995]]\n",
            "Right: [[1.]]  Left: [[0.99990636]]\n",
            "Right: [[1.]]  Left: [[0.9995339]]\n",
            "Right: [[0.99996746]]  Left: [[0.99999976]]\n",
            "Right: [[0.9998524]]  Left: [[0.9994521]]\n",
            "Right: [[0.9988633]]  Left: [[0.99989784]]\n",
            "Right: [[0.9999977]]  Left: [[0.999904]]\n",
            "Right: [[0.9999989]]  Left: [[0.9999979]]\n",
            "Right: [[0.99999475]]  Left: [[1.]]\n",
            "Right: [[0.9999995]]  Left: [[0.999985]]\n",
            "Right: [[0.9988675]]  Left: [[0.99888456]]\n",
            "Right: [[0.99997425]]  Left: [[1.]]\n",
            "Right: [[0.9999491]]  Left: [[0.9999629]]\n",
            "Right: [[0.2789143]]  Left: [[0.21334079]]\n",
            "Right: [[0.12297696]]  Left: [[0.01715085]]\n",
            "Right: [[0.23158991]]  Left: [[0.00892025]]\n",
            "Right: [[0.03607383]]  Left: [[0.03055584]]\n",
            "Pay attention!\n",
            "Right: [[0.13803348]]  Left: [[0.05338842]]\n",
            "Right: [[0.02494246]]  Left: [[0.14508638]]\n",
            "Right: [[0.01479334]]  Left: [[0.32642102]]\n",
            "Right: [[1.]]  Left: [[0.99999774]]\n",
            "Right: [[1.]]  Left: [[0.99983144]]\n",
            "Right: [[1.]]  Left: [[0.99791086]]\n",
            "Right: [[1.]]  Left: [[0.99989235]]\n",
            "Right: [[1.]]  Left: [[0.9997953]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[0.9999242]]  Left: [[0.9954028]]\n",
            "Right: [[1.]]  Left: [[0.99999416]]\n",
            "Right: [[0.9998409]]  Left: [[0.9930458]]\n",
            "Right: [[0.05996817]]  Left: [[0.03171217]]\n",
            "Right: [[0.00979194]]  Left: [[0.04202956]]\n",
            "Right: [[0.01787028]]  Left: [[0.00092798]]\n",
            "Right: [[0.00226367]]  Left: [[0.00642684]]\n",
            "Pay attention!\n",
            "Right: [[0.08523419]]  Left: [[0.7327167]]\n",
            "Right: [[0.00660479]]  Left: [[0.0231024]]\n",
            "Right: [[0.99490625]]  Left: [[0.0569019]]\n",
            "Right: [[1.]]  Left: [[0.99989676]]\n",
            "Right: [[0.9999995]]  Left: [[0.9999829]]\n",
            "Right: [[1.]]  Left: [[0.9999994]]\n",
            "Right: [[0.9999976]]  Left: [[0.99999905]]\n",
            "Right: [[0.99999917]]  Left: [[0.9997493]]\n",
            "Right: [[0.99998385]]  Left: [[0.999676]]\n",
            "Right: [[1.]]  Left: [[0.9999984]]\n",
            "Right: [[0.99300826]]  Left: [[0.9884093]]\n",
            "Right: [[0.9999966]]  Left: [[0.99998975]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[0.9993633]]  Left: [[0.998754]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[0.00138828]]  Left: [[0.00955862]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[1.]]  Left: [[1.]]\n",
            "Right: [[0.99996346]]  Left: [[0.9999411]]\n",
            "Right: [[0.99783754]]  Left: [[0.93551904]]\n",
            "Right: [[1.]]  Left: [[0.99999166]]\n",
            "Right: [[1.]]  Left: [[1.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aFGw2OiDi7C0"
      },
      "source": [
        "We can see how, on average, we have 3.4 frames each second with the camera we decided to use for our demo. This value allowed us to tune better the threshold value, which can be customised. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2BZJ4tWHdpEq",
        "outputId": "a7f0143c-81c4-453b-f686-262b416766c2"
      },
      "source": [
        "frame_count/(stop-start)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3.4194400062487587"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    }
  ]
}