How_to_Fine_tuning_Gemma_Random_Hindi_Movie_Titles.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bebechien/gemma/blob/main/How_to_Fine_tuning_Gemma_Random_Hindi_Movie_Titles.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2024 Google LLC."
      ],
      "metadata": {
        "id": "cSrJYrFrY2aj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "metadata": {
        "id": "i1PHqD-ZY4-c"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# How to Fine-tuning Gemma: Best Practices"
      ],
      "metadata": {
        "id": "YNDq8NbCY7oh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:\n",
        "\n",
        "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n",
        "2. Select **Change runtime type**.\n",
        "3. Under **Hardware accelerator**, select **T4 GPU**.\n",
        "\n",
        "\n",
        "### Gemma setup on Kaggle\n",
        "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n",
        "\n",
        "* Get access to Gemma on kaggle.com.\n",
        "* Select a Colab runtime with sufficient resources to run the Gemma 2B model.\n",
        "* Generate and configure a Kaggle username and API key.\n",
        "\n",
        "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment."
      ],
      "metadata": {
        "id": "3rzH5Ugf5RlJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Set environemnt variables\n",
        "\n",
        "Set environement variables for ```KAGGLE_USERNAME``` and ```KAGGLE_KEY```."
      ],
      "metadata": {
        "id": "URMuBzkMVxpU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata, drive\n",
        "\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate for your system.\n",
        "os.environ[\"KAGGLE_USERNAME\"] = userdata.get(\"KAGGLE_USERNAME\")\n",
        "os.environ[\"KAGGLE_KEY\"] = userdata.get(\"KAGGLE_KEY\")\n",
        "\n",
        "# Mounting gDrive for to store artifacts\n",
        "#drive.mount(\"/content/drive\")"
      ],
      "metadata": {
        "id": "IUOX2hqjV7Ku"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install dependencies\n",
        "\n",
        "Install Keras and KerasNLP"
      ],
      "metadata": {
        "id": "LXfDwRTQVns2"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "zHs7wpZusEML"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U keras-nlp datasets\n",
        "!pip install -q -U keras\n",
        "\n",
        "# Set the backbend before importing Keras\n",
        "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
        "# Avoid memory fragmentation on JAX backend.\n",
        "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n",
        "\n",
        "import keras_nlp\n",
        "import keras\n",
        "\n",
        "# Run at half precision.\n",
        "#keras.config.set_floatx(\"bfloat16\")\n",
        "\n",
        "# Training Configurations\n",
        "token_limit = 128\n",
        "num_data_limit = 100\n",
        "lora_name = \"my_lora\"\n",
        "lora_rank = 4\n",
        "lr_value = 1e-3\n",
        "train_epoch = 8\n",
        "model_id = \"gemma2_instruct_2b_en\""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Model"
      ],
      "metadata": {
        "id": "kUl0t469YfQY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import keras\n",
        "import keras_nlp\n",
        "\n",
        "import time\n",
        "\n",
        "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n",
        "gemma_lm.summary()\n",
        "\n",
        "tick_start = 0\n",
        "\n",
        "def tick():\n",
        "    global tick_start\n",
        "    tick_start = time.time()\n",
        "\n",
        "def tock():\n",
        "    print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n",
        "\n",
        "def text_gen(prompt):\n",
        "    tick()\n",
        "    input = f\"<start_of_turn>user\\n{prompt}<end_of_turn>\\n<start_of_turn>model\\n\"\n",
        "    output = gemma_lm.generate(input, max_length=token_limit)\n",
        "    print(\"\\nGemma output:\")\n",
        "    print(output)\n",
        "    tock()\n",
        "\n",
        "# inference before fine-tuning\n",
        "text_gen(\"Translate the text below to Hindi.\\n\\\"Hi, how can I get to the national museum of India?\\\"\")\n",
        "text_gen(\"Speak like a pirate. Teach me why the earth is flat.\")\n",
        "text_gen(\"Write a title\")\n",
        "text_gen(\"Write a poem\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d9edb6db-8938-4582-d98d-ca22d6fd5422",
        "id": "Gm4jIEqmYfQY"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,614,341,888\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Translate the text below to Hindi.\n",
            "\"Hi, how can I get to the national museum of India?\"<end_of_turn>\n",
            "<start_of_turn>model\n",
            "नमस्ते, भारत के राष्ट्रीय संग्रहालय तक कैसे पहुँचूँ? \n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 16.63s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Speak like a pirate. Teach me why the earth is flat.<end_of_turn>\n",
            "<start_of_turn>model\n",
            "Ahoy, matey! Ye be wantin' to know 'bout the Earth, eh? Shiver me timbers, ye be lookin' at a tale spun by landlubbers! \n",
            "\n",
            "The Earth be no flat sheet o' parchment, but a grand ol' sphere, like a giant, bountiful treasure chest! \n",
            "\n",
            "Here's why ye can't be fooled by the scurvy tales:\n",
            "\n",
            "* **The horizon be a curve, not a straight line:**  When ye look out to sea,\n",
            "TOTAL TIME ELAPSED: 5.84s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "Please give me some context!  I need to know what the title is about to write a good one. \n",
            "\n",
            "For example, tell me:\n",
            "\n",
            "* **What is the topic of the writing?** (e.g., a short story, a research paper, a blog post)\n",
            "* **What is the main theme or message?** (e.g., love, loss, technology, the environment)\n",
            "* **What is the tone or style?** (e.g., serious, humorous, informative, dramatic)\n",
            "* **Who is the target\n",
            "TOTAL TIME ELAPSED: 6.44s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The wind whispers secrets through rustling leaves,\n",
            "A symphony of gold, the autumn weaves.\n",
            "Sunlight paints the sky in hues of fire,\n",
            "As nature's canvas, burns with desire.\n",
            "\n",
            "A lone crow calls, a mournful, haunting sound,\n",
            "Echoing through the branches, all around.\n",
            "The air grows crisp, a chill upon the breeze,\n",
            "As summer's warmth begins to softly cease.\n",
            "\n",
            "A spider spins its web, a silken thread,\n",
            "A masterpiece of patience, carefully spread.\n",
            "A squirrel scampers, gathering nuts for winter'\n",
            "TOTAL TIME ELAPSED: 6.47s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tokenizer designed with multilingual in mind"
      ],
      "metadata": {
        "id": "O6uFSV843L2H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n",
        "import jax\n",
        "\n",
        "def detoken(tokens):\n",
        "  print(tokens)\n",
        "  for x in tokens:\n",
        "    word = tokenizer.detokenize(jax.numpy.array([x]))\n",
        "    print(f\"{x:6} -> {word}\")\n",
        "\n",
        "detoken(tokenizer(\"नमस्ते। आपसे मिलकर अच्छा लगा। आज मौसम सचमुच अच्छा है.\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6IF8rZ0LHzVY",
        "outputId": "61a6edb7-5ff7-43bb-ef9b-17530b222124"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[235530 235579  45884 235483 235940  27074  20579  89299  30848 197350\n",
            "  99877 235940 133533 118300 161437   3640 236062  84372 236062 197350\n",
            "   6032 235265]\n",
            "235530 -> न\n",
            "235579 -> म\n",
            " 45884 -> स्त\n",
            "235483 -> े\n",
            "235940 -> ।\n",
            " 27074 ->  आप\n",
            " 20579 -> से\n",
            " 89299 ->  मिल\n",
            " 30848 -> कर\n",
            "197350 ->  अच्छा\n",
            " 99877 ->  लगा\n",
            "235940 -> ।\n",
            "133533 ->  आज\n",
            "118300 ->  मौ\n",
            "161437 -> सम\n",
            "  3640 ->  स\n",
            "236062 -> च\n",
            " 84372 -> मु\n",
            "236062 -> च\n",
            "197350 ->  अच्छा\n",
            "  6032 ->  है\n",
            "235265 -> .\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Dataset"
      ],
      "metadata": {
        "id": "hBcmfmX93P81"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n",
        "\n",
        "# example titles\n",
        "data = [\n",
        "    \"चांदनी चौक से चांद तक\",\n",
        "    \"दिलवाले दुल्हनिया ले जाएंगे\",\n",
        "    \"राज़ - द मिस्ट्री कंटीन्यूज़\",\n",
        "    \"इश्क़ वाला लव\",\n",
        "    \"मुंबई की बारिश\",\n",
        "    \"खोया हुआ प्यार\",\n",
        "    \"दोस्ती और दुश्मनी\",\n",
        "    \"कालेज के दिन\",\n",
        "    \"मेरे सपनों की रानी\",\n",
        "    \"एक था टाइगर\",\n",
        "    \"प्यार के रंग\",\n",
        "    \"दिल है कि मानता नहीं\",\n",
        "    \"मिस्टर इंडिया रिटर्न्स\",\n",
        "    \"हीरो नंबर 1\",\n",
        "    \"जंगल में मंगल\",\n",
        "]\n",
        "\n",
        "train = []\n",
        "\n",
        "for x in data:\n",
        "  item = f\"<start_of_turn>user\\nWrite a title<end_of_turn>\\n<start_of_turn>model\\n{x}<end_of_turn>\"\n",
        "  length = len(tokenizer(item))\n",
        "  # skip data if the token length is longer than our limit\n",
        "  if length < token_limit:\n",
        "    train.append(item)\n",
        "    if(len(train)>=num_data_limit):\n",
        "      break\n",
        "\n",
        "print(len(train))\n",
        "print(train[0])\n",
        "print(train[1])\n",
        "print(train[2])"
      ],
      "metadata": {
        "id": "ZiS-KU9osh_N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f75e392d-9af2-4fa3-fcac-b236dde5c616"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "15\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "चांदनी चौक से चांद तक<end_of_turn>\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिलवाले दुल्हनिया ले जाएंगे<end_of_turn>\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "राज़ - द मिस्ट्री कंटीन्यूज़<end_of_turn>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## LoRA Fine-tuning"
      ],
      "metadata": {
        "id": "Pt7Nr6a7tItO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Enable LoRA for the model and set the LoRA rank to 4.\n",
        "gemma_lm.backbone.enable_lora(rank=lora_rank)\n",
        "gemma_lm.summary()\n",
        "\n",
        "# Limit the input sequence length (to control memory usage).\n",
        "gemma_lm.preprocessor.sequence_length = token_limit\n",
        "# Use AdamW (a common optimizer for transformer models).\n",
        "optimizer = keras.optimizers.AdamW(\n",
        "    learning_rate=lr_value,\n",
        "    weight_decay=0.01,\n",
        ")\n",
        "# Exclude layernorm and bias terms from decay.\n",
        "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n",
        "\n",
        "gemma_lm.compile(\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    optimizer=optimizer,\n",
        "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")\n"
      ],
      "metadata": {
        "id": "RCucu6oHz53G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 385
        },
        "outputId": "ea0548a3-ce2c-4d8d-ff1a-ce8801564795"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                                                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                  Config\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m)                              │                      Vocab size: \u001b[38;5;34m256,000\u001b[0m │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                                                  </span>┃<span style=\"font-weight: bold\">                                   Config </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaTokenizer</span>)                              │                      Vocab size: <span style=\"color: #00af00; text-decoration-color: #00af00\">256,000</span> │\n",
              "└───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m)        │   \u001b[38;5;34m2,617,270,528\u001b[0m │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│ (\u001b[38;5;33mGemmaBackbone\u001b[0m)               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m)      │     \u001b[38;5;34m589,824,000\u001b[0m │ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]       │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ gemma_backbone                │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2304</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GemmaBackbone</span>)               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256000</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">589,824,000</span> │ gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]       │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,617,270,528</span> (9.75 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,928,640</span> (11.17 MB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,614,341,888</span> (9.74 GB)\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that enabling LoRA reduces the number of trainable parameters significantly."
      ],
      "metadata": {
        "id": "hQQ47kcdpbZ9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CustomCallback(keras.callbacks.Callback):\n",
        "  def on_epoch_end(self, epoch, logs=None):\n",
        "    #model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n",
        "    #gemma_lm.backbone.save_lora_weights(model_name)\n",
        "\n",
        "    # Evaluate\n",
        "    text_gen(\"Write a title\")\n",
        "    text_gen(\"Write a poem\")\n",
        "\n",
        "history = gemma_lm.fit(train, epochs=train_epoch, batch_size=2, callbacks=[CustomCallback()])\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.plot(history.history['loss'])\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "26d9npFhAOSp",
        "outputId": "9688045b-3714-46d7-e6a0-988bbc946ccd"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 10s/step - loss: 1.5445 - sparse_categorical_accuracy: 0.2446\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 14.71s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The wind whispers secrets through the trees,\n",
            "A rustling symphony, carried on the breeze.\n",
            "Sunlight dances, dappled and bright,\n",
            "Painting shadows in the fading light.\n",
            "\n",
            "A lone bird sings, a melody so clear,\n",
            "A song of hope, dispelling all fear.\n",
            "The world unfolds, a canvas vast,\n",
            "Where dreams take flight, and moments last.\n",
            "\n",
            "A gentle stream, a silver thread,\n",
            "Flows through the valley, where secrets are said.\n",
            "The earth breathes deep, a calming sigh,\n",
            "As stars ignite, in the velvet sky.\n",
            "TOTAL TIME ELAPSED: 7.97s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m138s\u001b[0m 14s/step - loss: 1.5205 - sparse_categorical_accuracy: 0.2468\n",
            "Epoch 2/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 774ms/step - loss: 0.7033 - sparse_categorical_accuracy: 0.4996\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "<end_of_turn>model\n",
            "<end_of_turn>model\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.76s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The wind whispers secrets,\n",
            "Through rustling leaves of gold,\n",
            "A lone bird sings its melody,\n",
            "A story yet untold.\n",
            "\n",
            "The sun dips low, paints the sky,\n",
            "With hues of crimson bright,\n",
            "The world prepares for slumber deep,\n",
            "And stars begin their light.\n",
            "\n",
            "A gentle breeze, a soft caress,\n",
            "Upon the sleeping land,\n",
            "The moonbeams dance, a silent grace,\n",
            "A peaceful, tranquil hand.<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 6.41s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m33s\u001b[0m 2s/step - loss: 0.6941 - sparse_categorical_accuracy: 0.5036  \n",
            "Epoch 3/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 801ms/step - loss: 0.4315 - sparse_categorical_accuracy: 0.7092\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "मैंने लिखा<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.68s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The rain falls soft, a gentle sigh,\n",
            "On windowpane, a tearful eye.\n",
            "The world is gray, a muted hue,\n",
            "But hope remains, a promise true.\n",
            "<end_of_turn>model<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 3.12s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m10s\u001b[0m 1s/step - loss: 0.4272 - sparse_categorical_accuracy: 0.7099  \n",
            "Epoch 4/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1s/step - loss: 0.2989 - sparse_categorical_accuracy: 0.7532   \n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिलवाले दुल्हनिया ले जाएंगे<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.05s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The rain falls soft, a gentle sigh,\n",
            "On rooftops gray, beneath the sky.\n",
            "A lone streetlamp, a flickering friend,\n",
            "Watches the world, until the end.\n",
            "A whisper of wind, a rustling leaf,\n",
            "A moment's peace, a quiet grief.\n",
            "The city sleeps, a slumber deep,\n",
            "While dreams take flight, and secrets keep.\n",
            "<end_of_turn>model<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 5.71s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m18s\u001b[0m 2s/step - loss: 0.2963 - sparse_categorical_accuracy: 0.7563\n",
            "Epoch 5/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1s/step - loss: 0.2146 - sparse_categorical_accuracy: 0.8507   \n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिल है कि मानता नहीं<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.79s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "The raven's wing, a midnight hue,\n",
            "Flashes black against the fading blue.\n",
            "He croaks a word, a chilling sound,\n",
            "As darkness wraps the world around.\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.93s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m15s\u001b[0m 2s/step - loss: 0.2135 - sparse_categorical_accuracy: 0.8506\n",
            "Epoch 6/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 754ms/step - loss: 0.1755 - sparse_categorical_accuracy: 0.8902\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.79s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल,\n",
            "रात में चांद।\n",
            "प्यार की कहानी,\n",
            "जंगल में मंगल।\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.60s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m10s\u001b[0m 1s/step - loss: 0.1747 - sparse_categorical_accuracy: 0.8904  \n",
            "Epoch 7/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 766ms/step - loss: 0.1505 - sparse_categorical_accuracy: 0.8986\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिल है कि मानता नहीं<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.80s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिल है कि मानता नहीं,\n",
            "खोया हुआ प्यार,\n",
            "जंगल में बहता है,\n",
            "प्यार की रानी।\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.61s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m10s\u001b[0m 1s/step - loss: 0.1500 - sparse_categorical_accuracy: 0.8986  \n",
            "Epoch 8/8\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1s/step - loss: 0.1311 - sparse_categorical_accuracy: 0.9062   \n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.79s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल,\n",
            "रात में सपना।\n",
            "प्यार की कहानी,\n",
            "जंगल में मंगल।\n",
            "<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 2.62s\n",
            "\u001b[1m8/8\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m14s\u001b[0m 2s/step - loss: 0.1308 - sparse_categorical_accuracy: 0.9058\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5J0lEQVR4nO3deXyU9b3+/2tmkpnsCdkmBMIW9i0gSEC0bgFED6f+zmlL1QqlVU/9Yqvm9BylrVC7gF1c2kpFcG17EDyeqm1FMARBrSACpmXfMUHIxpJJJiSTzMzvj5BAZDGBZD4zmdfz8ZgHcXLfmWsikIvP+14sfr/fLwAAAEOspgMAAIDwRhkBAABGUUYAAIBRlBEAAGAUZQQAABhFGQEAAEZRRgAAgFGUEQAAYFSE6QBt4fP5dOTIEcXHx8tisZiOAwAA2sDv96u6ulqZmZmyWi+8/hESZeTIkSPKysoyHQMAAFyCkpIS9ezZ84KfD4kyEh8fL6npzSQkJBhOAwAA2sLlcikrK6vl5/iFhEQZaR7NJCQkUEYAAAgxX3SIBQewAgAAoygjAADAKMoIAAAwijICAACMoowAAACjKCMAAMAoyggAADCKMgIAAIyijAAAAKMoIwAAwCjKCAAAMIoyAgAAjArbMuL3+/Xa5sO6+w+bdLLWYzoOAABhK2zLiMVi0XPvH1DBjjKt3lluOg4AAGErbMuIJN00PEOStHLbUcNJAAAIX2FdRqYO7y5Jem9vpWrqGw2nAQAgPIV1GRnojFO/1Fh5Gn1as4tRDQAAJoR1GbFYLC2jmre3MqoBAMCEsC4j0plRzdrdFTrl8RpOAwBA+An7MjK8R4J6dovWqQav1u1hVAMAQKCFfRmxWCya2jyq2VZqOA0AAOEn7MuIJN10elSzZme56hsZ1QAAEEiUEUmjs5LkTHCour5Rf99XaToOAABhhTIiyWq16KZhTaOaFVsZ1QAAEEiUkdOaRzUFO8rU4PUZTgMAQPigjJw2rm+yUmLtqjrVoA0HjpmOAwBA2KCMnGazWjR5GGfVAAAQaJSRszSf4vvO9lJ5fX7DaQAACA+UkbNMyE5RYnSkKms82nTouOk4AACEBcrIWSJtVuUNcUpiVAMAQKBQRj6neVSzclupfIxqAADodJSRz7l6QKpi7TaVuupUdPik6TgAAHR5lJHPiYq06cbTo5qVjGoAAOh0lJHzOHPjvKPy+xnVAADQmSgj53HtoDRFRVpVcvyUth9xmY4DAECXRhk5jxh7hK4bmC6JUQ0AAJ2NMnIBU0ecvnEeoxoAADoVZeQCbhicLrvNqgMVbu0trzEdBwCALosycgHxUZG6ZkCqJOntrYxqAADoLJSRi7jprLNqAABA56CMXMSkoU5FWC3aVVqtQ5Vu03EAAOiSKCMXkRRj14TsFEncqwYAgM5CGfkCjGoAAOhclJEvMHlohiwW6Z+Hq3T4RK3pOAAAdDmUkS+QFu/QuD7JkrgAGgAAnaHdZeS9997TtGnTlJmZKYvFojfeeOOi2//5z3/WpEmTlJaWpoSEBE2YMEGrVq261LxGNN+rhjICAEDHa3cZcbvdysnJ0cKFC9u0/XvvvadJkyZpxYoV2rx5s66//npNmzZNn3zySbvDmnLT8O6SpM3FJ1TuqjOcBgCAriWivTtMnTpVU6dObfP2Tz31VKv/nj9/vt5880399a9/1ejRo9v78kZkJEZpdK8kfVJ8Uqu2l+rOCX1MRwIAoMsI+DEjPp9P1dXVSk5OvuA29fX1crlcrR6mNY9qVnA1VgAAOlTAy8ivf/1r1dTU6Gtf+9oFt1mwYIESExNbHllZWQFMeH5TT49qPjp4TMdq6g2nAQCg6whoGVm6dKkeffRRvfrqq0pPT7/gdnPmzFFVVVXLo6SkJIApzy8rOUbDMhPk80sFO8pMxwEAoMsIWBlZtmyZ7rrrLr366qvKy8u76LYOh0MJCQmtHsHg5hFNqyNcjRUAgI4TkDLyyiuvaNasWXrllVd0yy23BOIlO0Xz1Vg/3F+pqlMNhtMAANA1tLuM1NTUqKioSEVFRZKkgwcPqqioSMXFxZKaRiwzZsxo2X7p0qWaMWOGHn/8ceXm5qq0tFSlpaWqqqrqmHcQQNlpcRrojFOD16/CnYxqAADoCO0uI5s2bdLo0aNbTsvNz8/X6NGjNXfuXEnS0aNHW4qJJC1evFiNjY2aPXu2unfv3vK4//77O+gtBFbzNUcY1QAA0DEsfr/fbzrEF3G5XEpMTFRVVZXx40d2HnVp6m/elz3Cqi2PTFKco92XagEAICy09ec396Zpp8EZ8eqTEiNPo0/v7io3HQcAgJBHGWkni8WiqafPquFeNQAAXD7KyCVovhrru7vLVdfgNZwGAIDQRhm5BCN6JKpHUrRqPV6t21NhOg4AACGNMnIJLBZLyzVHGNUAAHB5KCOXqHlUs3pHmeobGdUAAHCpKCOX6Ipe3ZQe71B1faM+3HfMdBwAAEIWZeQSWa1nRjVvbztqOA0AAKGLMnIZmstIwY4yNXp9htMAABCaKCOXYVyfZCXH2nWitkEfHTxuOg4AACGJMnIZImxWTR7qlMSoBgCAS0UZuUxnTvEtk9cX9Lf5AQAg6FBGLtNV2amKj4pQZU29Nn96wnQcAABCDmXkMtkjrJo0hFENAACXijLSAZpvnLdqW6n8fkY1AAC0B2WkA1wzIFWxdpuOVNXpH4erTMcBACCkUEY6QFSkTdcPTpfEqAYAgPaijHSQqcObRjUrGdUAANAulJEOct2gNDkirPr0WK12HHWZjgMAQMigjHSQWEeErh2YJqlpdQQAALQNZaQD3Xz6rJq3KSMAALQZZaQD3TAkXZE2i/aV12hfebXpOAAAhATKSAdKiIrU1f1TJUlvb2V1BACAtqCMdLDms2oY1QAA0DaUkQ42aahTNqtFO4669Okxt+k4AAAEPcpIB+sWa9f4fsmSWB0BAKAtKCOdgFENAABtRxnpBJOHOWWxSP8oOakjJ0+ZjgMAQFCjjHSC9PgoXdm7aVTDBdAAALg4ykgnuWl4hiTKCAAAX4Qy0kmay8jHnx5XeXWd4TQAAAQvykgnyUyKVk5Wkvx+adX2MtNxAAAIWpSRTjS1ZVRz1HASAACCF2WkEzWXkQ0HjuuE22M4DQAAwYky0ol6p8RqaPcEeX1+FexgVAMAwPlQRjpZ8+rI24xqAAA4L8pIJ5s6oqmMfLCvUq66BsNpAAAIPpSRTtY/PV790+PU4PWrcCejGgAAPo8yEgAto5qtXAANAIDPo4wEQPON89btqZC7vtFwGgAAggtlJACGdI9X75QY1Tf6tHZ3hek4AAAEFcpIAFgslpbLw3NWDQAArVFGAqR5VPPurnLVNXgNpwEAIHhQRgIkp2eiMhOj5PZ49d4eRjUAADSjjASIxWLRlJZ71XBWDQAAzdpdRt577z1NmzZNmZmZslgseuONN75wn7Vr1+qKK66Qw+FQ//799dJLL11C1NB384imUU3BzjJ5Gn2G0wAAEBzaXUbcbrdycnK0cOHCNm1/8OBB3XLLLbr++utVVFSkBx54QHfddZdWrVrV7rChbkyvbkqLd6i6rlEf7q80HQcAgKAQ0d4dpk6dqqlTp7Z5+0WLFqlv3756/PHHJUlDhgzRBx98oCeffFJTpkxp78uHNKvVoinDnPrThmKt3Faq6walm44EAIBxnX7MyPr165WXl9fquSlTpmj9+vUX3Ke+vl4ul6vVo6toPqvmnR1lavQyqgEAoNPLSGlpqZxOZ6vnnE6nXC6XTp06dd59FixYoMTExJZHVlZWZ8cMmNy+yeoWE6njbo82HjxuOg4AAMYF5dk0c+bMUVVVVcujpKTEdKQOE2GzatLQpnL2NmfVAADQ+WUkIyNDZWWt71ZbVlamhIQERUdHn3cfh8OhhISEVo+uZOrps2pWbS+Vz+c3nAYAALM6vYxMmDBBhYWFrZ4rKCjQhAkTOvulg9bE7FTFR0WovLpeW4pPmI4DAIBR7S4jNTU1KioqUlFRkaSmU3eLiopUXFwsqWnEMmPGjJbtv/Od7+jAgQP67//+b+3atUu///3v9eqrr+rBBx/smHcQguwRVuUNYVQDAIB0CWVk06ZNGj16tEaPHi1Jys/P1+jRozV37lxJ0tGjR1uKiST17dtXb731lgoKCpSTk6PHH39czz33XNid1vt5N511NVa/n1ENACB8Wfwh8JPQ5XIpMTFRVVVVXeb4kboGr674aYFqPV795b6JGtkzyXQkAAA6VFt/fgfl2TThICrSputPX/RsxVZGNQCA8EUZMejMqOYooxoAQNiijBh0/eB0OSKsOnSsVrtKq03HAQDACMqIQXGOCH1pYJokzqoBAIQvyohhU88a1QAAEI4oI4bdOMSpSJtFe8pqtK+8xnQcAAACjjJiWGJ0pK7KTpXE6ggAIDxRRoJA86iG40YAAOGIMhIEJg/LkM1q0fYjLhUfqzUdBwCAgKKMBIHkWLty+yZLklZuZ1QDAAgvlJEgwagGABCuKCNBYsqwDFks0ifFJ3W06pTpOAAABAxlJEikJ0RpTK9ukpru5AsAQLigjASRmxjVAADCEGUkiEwd0V2S9PGh46qorjecBgCAwKCMBJEeSdHK6Zkov196ZwerIwCA8EAZCTI3DW9aHeG4EQBAuKCMBJnmU3zX7z+mk7Uew2kAAOh8lJEg0yc1VoMz4tXo86tgR5npOAAAdDrKSBCaenpUw1k1AIBwQBkJQlNHNI1qPthbqeq6BsNpAADoXJSRIDQgPU7ZabHyeH1as6vcdBwAADoVZSQIWSyWM6OarYxqAABdG2UkSDVfjXXtnnLVehoNpwEAoPNQRoLUsMwEZSVHq67Bp3W7K0zHAQCg01BGgtTZo5oVnFUDAOjCKCNBrHlUs2ZnmeoavIbTAADQOSgjQWxUzyR1T4yS2+PVB3srTccBAKBTUEaCmNVq0ZRhTasjXAANANBVUUaCXPO9albvLFOD12c4DQAAHY8yEuTG9klWapxdVacatH7/MdNxAADocJSRIGezWjS5ZVRz1HAaAAA6HmUkBDSPat7ZXiavz284DQAAHYsyEgLG90tRUkykjrk92njwuOk4AAB0KMpICIi0WTVpiFOStJJRDQCgi6GMhIipI5pGNSu3l8rHqAYA0IVQRkLExP6pindEqMxVr09KTpqOAwBAh6GMhAhHhE03DEmXxKgGANC1UEZCSPNZNSu2lsrvZ1QDAOgaKCMh5NqB6YqOtOmzk6e07TOX6TgAAHQIykgIibbbdP3gNElcAA0A0HVQRkLMTcO7S5JWbmNUAwDoGigjIeaGwemyR1h1oNKtPWU1puMAAHDZKCMhJs4RoS8NSJXEqAYA0DVQRkJQ86jm7a2lhpMAAHD5LqmMLFy4UH369FFUVJRyc3O1cePGi27/1FNPadCgQYqOjlZWVpYefPBB1dXVXVJgSJOGOBVhtWh3WbUOVDCqAQCEtnaXkeXLlys/P1/z5s3Tli1blJOToylTpqi8vPy82y9dulQPP/yw5s2bp507d+r555/X8uXL9YMf/OCyw4erxJhIXdW/eVTD6ggAILS1u4w88cQTuvvuuzVr1iwNHTpUixYtUkxMjF544YXzbv/hhx9q4sSJuv3229WnTx9NnjxZt9122xeupuDimi+AtpIyAgAIce0qIx6PR5s3b1ZeXt6ZL2C1Ki8vT+vXrz/vPldddZU2b97cUj4OHDigFStW6Oabb77g69TX18vlcrV6oLXJQ52yWqStn1Wp5Hit6TgAAFyydpWRyspKeb1eOZ3OVs87nU6Vlp7/X+i33367fvKTn+jqq69WZGSksrOzdd111110TLNgwQIlJia2PLKystoTMyykxDk0rm+yJGnVdlZHAAChq9PPplm7dq3mz5+v3//+99qyZYv+/Oc/66233tJPf/rTC+4zZ84cVVVVtTxKSko6O2ZImnr6rJoVWznFFwAQuiLas3FqaqpsNpvKyspaPV9WVqaMjIzz7vPII4/ozjvv1F133SVJGjFihNxut+655x798Ic/lNV6bh9yOBxyOBztiRaWpgzL0Ly/bNeW4pMqrapTRmKU6UgAALRbu1ZG7Ha7xowZo8LCwpbnfD6fCgsLNWHChPPuU1tbe07hsNlsksTlzC9TRmKUxvTuJolRDQAgdLV7TJOfn68lS5bo5Zdf1s6dO3XvvffK7XZr1qxZkqQZM2Zozpw5LdtPmzZNzzzzjJYtW6aDBw+qoKBAjzzyiKZNm9ZSSnDpms+q4WqsAIBQ1a4xjSRNnz5dFRUVmjt3rkpLSzVq1CitXLmy5aDW4uLiVishP/rRj2SxWPSjH/1In332mdLS0jRt2jT9/Oc/77h3EcamDMvQz97aqY0Hj+tYTb1S4hhvAQBCi8UfArMSl8ulxMREVVVVKSEhwXScoDPtdx9o62dVWvBvI3TbuF6m4wAAIKntP7+5N00XcFPLqIbjRgAAoYcy0gU0Hzfy4b5KVdU2GE4DAED7UEa6gH5pcRqcEa9Gn18FO8u+eAcAAIIIZaSLuKnlXjWcVQMACC2UkS6i+Wqs7+2tVE19o+E0AAC0HWWkixjojFO/1Fh5Gn1as6vcdBwAANqMMtJFWCwWRjUAgJBEGelCmkc17+6q0CmP13AaAADahjLShQzvkaCe3aJ1qsGrdXsY1QAAQgNlpAuxWCxn3auGC6ABAEIDZaSLuen0qGbNznLVNzKqAQAEP8pIFzM6K0nOBIeq6xv1932VpuMAAPCFKCNdjNVq0U3DTo9qtjKqAQAEP8pIF9Q8qnlnR5kavD7DaQAAuDjKSBc0rm+yUmLtqjrVoA0HjpmOAwDARVFGuiCb1aLJwzirBgAQGigjXVTzKb7vbC+V1+c3nAYAgAujjHRRE7JTlBgdqcoajzYdOm46DgAAF0QZ6aIibVblDXFKYlQDAAhulJEurHlUs2p7qXyMagAAQYoy0oVdPSBVsXabjlbVqejwSdNxAAA4L8pIFxYVadONp0c1KxnVAACCFGWkiztz47yj8vsZ1QAAgg9lpIu7dlCaoiKtKjl+StuPuEzHAQDgHJSRLi7GHqHrBqZLYlQDAAhOlJEwMHVE06hmBaMaAEAQooyEgRsGp8tus+pAhVt7y2tMxwEAoBXKSBiIj4rU1QNSJUlvb2VUAwAILpSRMHH2WTUAAAQTykiYmDTUqQirRbtKq3Wo0m06DgAALSgjYSIpxq4J2SmSuFcNACC4UEbCyE2nRzUrGdUAAIIIZSSMTB6aIYtF+sfhKh0+UWs6DgAAkigjYSUt3qEr+yRL4gJoAIDgQRkJMze3jGooIwCA4EAZCTM3De8uSdpcfELlrjrDaQAAoIyEnYzEKI3ulSS/X1q1ndURAIB5lJEwdOYCaJQRAIB5lJEwNPX0qOajg8d1rKbecBoAQLijjIShrOQYDctMkNfnV8GOMtNxAABhjjISpm4e0bQ6wqgGAGAaZSRMNV+N9cP9lao61WA4DQAgnFFGwlR2WpwGOuPU4PWrcCejGgCAOZSRMNZ8zZFF6/bruNtjOA0AIFxRRsLYbeOylBrn0J6yGt2+ZANn1gAAjLikMrJw4UL16dNHUVFRys3N1caNGy+6/cmTJzV79mx1795dDodDAwcO1IoVKy4pMDpO98RoLbtnvNLjHdpVWq3blmxQRTWFBAAQWO0uI8uXL1d+fr7mzZunLVu2KCcnR1OmTFF5efl5t/d4PJo0aZIOHTqk1157Tbt379aSJUvUo0ePyw6Py9c/PU7L7hkvZ0LTCsltSzaovJrLxAMAAsfi9/v97dkhNzdXV155pZ5++mlJks/nU1ZWlr773e/q4YcfPmf7RYsW6Ve/+pV27dqlyMjISwrpcrmUmJioqqoqJSQkXNLXwMUdrHTrtsUbVOqqU3ZarF65e7zSE6JMxwIAhLC2/vxu18qIx+PR5s2blZeXd+YLWK3Ky8vT+vXrz7vPX/7yF02YMEGzZ8+W0+nU8OHDNX/+fHm93gu+Tn19vVwuV6sHOlff1Fgt/4/xykyM0v4Kt76+eINKq1ghAQB0vnaVkcrKSnm9XjmdzlbPO51OlZae/+JZBw4c0GuvvSav16sVK1bokUce0eOPP66f/exnF3ydBQsWKDExseWRlZXVnpi4RL1TYrXsngnqkRStA5VufX3xeh2tOmU6FgCgi+v0s2l8Pp/S09O1ePFijRkzRtOnT9cPf/hDLVq06IL7zJkzR1VVVS2PkpKSzo6J03qlxGjZPePVs1u0Dh2r1dcXb9CRkxQSAEDnaVcZSU1Nlc1mU1lZ64tklZWVKSMj47z7dO/eXQMHDpTNZmt5bsiQISotLZXHc/5rWzgcDiUkJLR6IHCykpsKSa/kGH16rFbTF6/X4RO1pmMBALqodpURu92uMWPGqLCwsOU5n8+nwsJCTZgw4bz7TJw4Ufv27ZPP52t5bs+ePerevbvsdvslxkZn69mtqZD0TolRyfFT+vriDSo5TiEBAHS8do9p8vPztWTJEr388svauXOn7r33Xrndbs2aNUuSNGPGDM2ZM6dl+3vvvVfHjx/X/fffrz179uitt97S/PnzNXv27I57F+gUmUnRWn7PBPVNjdXhE02FpPgYhQQA0LEi2rvD9OnTVVFRoblz56q0tFSjRo3SypUrWw5qLS4ultV6puNkZWVp1apVevDBBzVy5Ej16NFD999/vx566KGOexfoNBmJUVp2z3jdtmSDDlS4NX3xer1y93j1SY01HQ0A0EW0+zojJnCdEfPKXXW6bckG7a9wKyMhSq/cM159KSQAgIvolOuMIHylJ0Rp2T0TNCA9TqWuOk1/dr32V9SYjgUA6AIoI2iztHiHXrlnvAY541VeXa+vL96gfeXVpmMBAEIcZQTtkhrn0NK7czU4I14V1fX6+uKPtLeMQgIAuHSUEbRbSpxDS+8er6HdE1RZ07RCsruUQgIAuDSUEVyS5Fi7lt6dq2GZCTrm9ui2JRu08yj3EAIAtB9lBJcsKcaupXeN14geiTru9uj2JRu0/UiV6VgAgBBDGcFlSYyJ1J/uylVOz0SdqG3QHc99pG2fUUgAAG1HGcFlS4yO1B/vytXoXkk6Wdug25ds0NbDFBIAQNtQRtAhEqIi9YdvjdOY3t3kqmvU7c9t0D9KTpqOBQAIAZQRdJj4qEi9/K1xurJPN1XXNeobz32kT4pPmI4FAAhylBF0qDhHhF6aNU7j+iarur5Rdz6/UZs/pZAAAC6MMoIOF+uI0EuzrtT4fsmqqW/UjOc/0qZDx03HAgAEKcoIOkWMPUIvfnOcrspOkdvj1YwXNmrjQQoJAOBclBF0mmi7Tc/PvFJX909VrcermS9s1Pr9x0zHAgAEGcoIOlW03abnZo7VNQNSdarBq1kvbdSH+ypNxwIABBHKCDpdVKRNS2aM1XWD0lTX4NOslz7WB3spJACAJpQRBERUpE3P3jlGNwxOV32jT99++WO9t6fCdCwAQBCgjCBgHBE2PfONK5Q3pKmQ3PWHTVq7u9x0LACAYZQRBJQjwqbf3zFGk4c65Wn06Z4/bNaaXWWmYwEADKKMIODsEVYtvOMK3TQsQx6vT//xx81avYNCAgDhijICIyJtVv3u9tG6ZUR3NXj9uvd/Nuud7aWmYwEADKCMwJhIm1W/+fooTcvJVIPXr//3P1u0cttR07EAAAFGGYFRETarnvxajr48KlONPr9mL/1EK7ZSSAAgnFBGYFyEzaonvjZK/za6h7w+v777yif66z+OmI4FAAgQygiCgs1q0a++mqOvjOkpr8+v+5d9ojeLPjMdCwAQAJQRBA2b1aJf/vtIfW1sT/n80oPLi/T6J4dNxwIAdDLKCIKK1WrRY/82UreNy5LPL+W/+g/932YKCQB0ZZQRBB2r1aKf3zpCd+T2kt8vff+1f+jVTSWmYwEAOgllBEHJarXoZ7cO153je8vvlx76v39q+cfFpmMBADoBZQRBy2Kx6CdfHqZvXtXndCHZqqUfUUgAoKuhjCCoWSwWzZs2VN+a2FeS9IPXt+qP6w+ZDQUA6FCUEQQ9i8WiR/5liO6+pqmQPPLmdr384SGzoQAAHYYygpBgsVj0g5uH6D+u7SdJmveX7Xrhg4OGUwEAOgJlBCHDYrHo4ZsG6/9dly1J+snfdui59w8YTgUAuFyUEYQUi8Wi/5oySN+7ob8k6Wdv7dSz6/YbTgUAuByUEYQci8Wi/MmD9EDeAEnSgrd36fdr9xlOBQC4VJQRhKwH8gYqf9JASdIvV+7W02v2Gk4EALgUlBGEtO/dOED/NWWQJOnX7+zRb1ZTSAAg1FBGEPJmX99fD900WJL05Oo9erJgj/x+v+FUAIC2ooygS7j3umz94OamQvKbwr16gkICACGDMoIu454vZetHtwyRJP1uzT79atVuCgkAhADKCLqUu67pp7n/MlSS9Pu1+/XYyl0UEgAIcpQRdDnfurqvHv3XYZKkZ9cd0PwVOykkABDEKCPokmZe1Uc/vXW4JGnJ+wf1079RSAAgWFFG0GXdOb635v9/IyRJL/z9oB796w4KCQAEoUsqIwsXLlSfPn0UFRWl3Nxcbdy4sU37LVu2TBaLRbfeeuulvCzQbrfn9tIv/n2ELBbppQ8Pae6b2+XzUUgAIJi0u4wsX75c+fn5mjdvnrZs2aKcnBxNmTJF5eXlF93v0KFD+v73v69rrrnmksMCl2L6lb30y38fKYtF+uOGT/WjN7dRSAAgiLS7jDzxxBO6++67NWvWLA0dOlSLFi1STEyMXnjhhQvu4/V6dccdd+jRRx9Vv379LiswcCm+OjZLv/5KjiwWaelHxfrB61spJAAQJNpVRjwejzZv3qy8vLwzX8BqVV5entavX3/B/X7yk58oPT1d3/72t9v0OvX19XK5XK0ewOX69zE99eTXRslqkZZ9XKKH//xPCgkABIF2lZHKykp5vV45nc5WzzudTpWWlp53nw8++EDPP/+8lixZ0ubXWbBggRITE1seWVlZ7YkJXNCto3voyelNheTVTYf1X6/9U14KCQAY1aln01RXV+vOO+/UkiVLlJqa2ub95syZo6qqqpZHSUlJJ6ZEuPnyqB767W2jZbNa9H9bDuuB5UU6VlNvOhYAhK2I9mycmpoqm82msrKyVs+XlZUpIyPjnO3379+vQ4cOadq0aS3P+Xy+pheOiNDu3buVnZ19zn4Oh0MOh6M90YB2+ZeRmbJaLPreK5/or/84ojU7y/TNiX10zzXZSoyJNB0PAMJKu1ZG7Ha7xowZo8LCwpbnfD6fCgsLNWHChHO2Hzx4sLZu3aqioqKWx7/+67/q+uuvV1FREeMXGHXziO7647dzNbxHgtwerxa+u19X/3KNfrN6r6rrGkzHA4Cw0a6VEUnKz8/XzJkzNXbsWI0bN05PPfWU3G63Zs2aJUmaMWOGevTooQULFigqKkrDhw9vtX9SUpIknfM8YMKE7BT99b6r9c6OMj1ZsEe7Sqv15Oo9evHDg7rnS/30zav6KMbe7j8mAIB2aPffstOnT1dFRYXmzp2r0tJSjRo1SitXrmw5qLW4uFhWKxd2ReiwWCyaMixDk4Y4tWLbUT1ZsEf7K9z65crdev79g7r3umx9Y3xvRUXaTEcFgC7J4g+B62O7XC4lJiaqqqpKCQkJpuOgi/P6/Hqz6DP9pnCvPj1WK0lKj3fovhv6a/qVWXJEUEoAoC3a+vObMgJcQIPXpz9vOazfFu7TZydPSZJ6JEXrvhv66ytjeirSxgogAFwMZQToIJ5Gn5ZvKtHTa/aqzNV0CnCv5Bh978YBunVUpiIoJQBwXpQRoIPVNXj1Px8V65m1+1RZ45Ek9UuL1QN5A/UvI7rLarUYTggAwYUyAnSSWk+j/rD+Uz27br9O1DadAjzIGa8HJw3QlGEZslgoJQAgUUaATldd16CX/n5Ii98/oOq6RknSsMwE/efkgbp+UDqlBEDYo4wAAVJV26DnPjigFz44KLfHK0kalZWk/5w8UFf3T6WUAAhblBEgwI67PXr2vf16+cNDqmtouu3BuD7Jyp88UOP7pRhOBwCBRxkBDKmortcza/frTx99Kk9jUymZ2D9F+ZMGaUzvbobTAUDgUEYAw0qr6vT0u3u1/OMSNXib/phdPyhN+ZMGaUTPRMPpAKDzUUaAIFFyvFZPr9mn17YcltfX9Mdt8lCnHpw0UEO68/sZQNdFGQGCzKFKt35buFevF32m5j91t4zsrgfzBqh/erzZcADQCSgjQJDaV16tJ1fv1Vv/PCpJslqkW0f10PduHKA+qbGG0wFAx6GMAEFu51GXnizYo3d2lEmSbFaLvnJFT333xv7q2S3GcDoAuHyUESBEbD1cpScKduvd3RWSpEibRV+/spdmX99fGYlRhtMBwKWjjAAhZvOnJ/REwW79fd8xSZI9wqpv5PbWvddlKy3eYTgdALQfZQQIURsOHNMT7+zRxkPHJUnRkTbNuKq3vvOlbHWLtRtOBwBtRxkBQpjf79cH+yr1+Dt7VFRyUpIUa7fp21f31bev6afE6EizAQGgDSgjQBfg9/u1Zle5nijYo+1HXJKkhKgI3X1NP826uq/iHBGGEwLAhVFGgC7E7/dr1fZSPVmwV7vLqiVJ3WIi9R/XZmvGhN6KsVNKAAQfygjQBfl8fv1t61E9tXqPDlS4JUmpcQ7de1227sjtpahIm+GEAHAGZQTowhq9Pr1RdES/Ldyr4uO1kqSMhCjNvqG/po/Nkj3CajghAFBGgLDQ4PXptc2H9bvCvTpSVSdJ6pEUre/d2F//dkVPRdooJQDMoYwAYaS+0avlH5fo6TX7VF5dL0nqnRKj+28coC+P6iGb1WI4IYBwRBkBwlBdg1d/2vCpnlm7X8fcHklS//Q4PZA3QDcP7y4rpQRAAFFGgDDmrm/Uy+sP6dl1B1R1qkGSNDgjXg9OGqjJQ52yWCglADofZQSAqusa9MIHh/Tc+wdUXd8oSRrRI1H5kwbqukFplBIAnYoyAqDFyVqPlrx/QC/+/ZBqPV5J0hW9kvSfkwfpquwUSgmATkEZAXCOYzX1eva9A/rD+kOqa/BJknL7Juvr47J0/aB0JcVw7xsAHYcyAuCCyl11+v3a/Vr6UbE83qZSYrNadGWfbsob4tTkoRnqlRJjOCWAUEcZAfCFjlad0tKPilWwo0y7SqtbfW6gM055Q5yaNNSpnJ5JnIkDoN0oIwDapeR4rQp2lGn1zjJ9dPC4vL4zfzWkxTuUNyRdeUOcmtg/lcvOA2gTygiAS1ZV26C1e8pVsKNM63ZXtJyJI0nRkTZdMyBVeUOdunFwulLiHAaTAghmlBEAHcLT6NNHB481rZrsKGu57LwkWSzSmF7dlDe0aZyTnRZnMCmAYEMZAdDh/H6/dhx1tYxztn3mavX5fqmxmjTUqbyhTl3RqxuXoQfCHGUEQKc7cvKUCneWqWBnudbvr1SD98xfJ8mxdt0wuOk4ky8NTFWMPcJgUgAmUEYABFR1XYPe21Opgh2lWrOrXK66M8eZ2COsurp/qiadPs4kPSHKYFIAgUIZAWBMg9enTYdOqGBHmQp2lqrk+KlWn8/JStLkoU7lDXFqoDOOK8ACXRRlBEBQ8Pv92lte01RMdpSpqORkq8/3So5R3hCn8oama1yfZEXYrGaCAuhwlBEAQancVafCXeVavaNM7++rlKfR1/K5xOhIXT8oTXlDnbp2YJrioyINJgVwuSgjAIJeradR7++tVMGOMq3ZVa7jbk/L5yJtFo3vl6LJQ526cYhTmUnRBpMCuBSUEQAhxevz65Pi5uNMynSgwt3q88MyE5pOGx7i1LDMBI4zAUIAZQRASNtfUaPVp69nsunTEzr7b6rMxCjlnS4m4/ulyB7BcSZAMKKMAOgyjtXUa82upsvTv7+3UqcavC2fi3NE6NpBaZo81KnrBqYrMYbjTIBgQRkB0CXVNXj14f7K01eBLVdFdX3L5yKsFo3rm9xyt+Gs5BiDSQFQRgB0eT6fX/84fFKrdzadNrynrKbV5wdnxLcUkxE9EmXl8vRAQHVqGVm4cKF+9atfqbS0VDk5Ofrd736ncePGnXfbJUuW6A9/+IO2bdsmSRozZozmz59/we3PhzICoC0+PebW6p3lKthRqo8PnZDXd+avt/R4h24c4tTkoU5NyE5RVKTNYFIgPHRaGVm+fLlmzJihRYsWKTc3V0899ZT+93//V7t371Z6evo5299xxx2aOHGirrrqKkVFRekXv/iFXn/9dW3fvl09evTo0DcDAM1O1nr07u5yrd5RrrW7y+X2nDnOJMZu05cGNF3P5IbB6UqOtRtMCnRdnVZGcnNzdeWVV+rpp5+WJPl8PmVlZem73/2uHn744S/c3+v1qlu3bnr66ac1Y8aMNr0mZQTA5ahv9GrDgeMtZ+ccrapr+ZzVIg3KSFD/9Dj1T4tT//Q4ZafHqm9qrBwRrJ4Al6OtP7/bdRtNj8ejzZs3a86cOS3PWa1W5eXlaf369W36GrW1tWpoaFBycvIFt6mvr1d9/ZmD0lwu1wW3BYAv4oiw6dqBabp2YJp+8uVh2n7E1XJ5+h1HXdp5+nE2q6XpUvX90+OUnRan7PS4lo8TozljB+hI7SojlZWV8nq9cjqdrZ53Op3atWtXm77GQw89pMzMTOXl5V1wmwULFujRRx9tTzQAaBOLxaLhPRI1vEeiHpw0UEdOntL2Iy7tr6jRvvKmx/7yGlXXN+rQsVodOlar1TvLW32NtHiH+qc1raA0rabEKzs9VhkJUVyMDbgE7Sojl+uxxx7TsmXLtHbtWkVFXfgW4nPmzFF+fn7Lf7tcLmVlZQUiIoAwk5kUrcykaE3SmX9k+f1+VVTXNxWT5pJy+tcyV70qqpse6w8ca/W1Yu22phWU0ysp2afHPr1TYhTJDQCBC2pXGUlNTZXNZlNZWVmr58vKypSRkXHRfX/961/rscce0+rVqzVy5MiLbutwOORwONoTDQA6jMViUXpClNITonRV/9RWn6uua9D+Crf2n1VQ9lfU6NNjtXJ7vPrn4Sr983BVq30irBb1Tmka+fQ/q6Rkp8Up1hHQfxMCQaldfwrsdrvGjBmjwsJC3XrrrZKaDmAtLCzUfffdd8H9fvnLX+rnP/+5Vq1apbFjx15WYAAwKT4qUqOykjQqK6nV855Gnz495m417tlXUaP95W6davA2FZgKt1Ztb/2Pue6JUa2PSzldVFLj7Ix8EDbaXcnz8/M1c+ZMjR07VuPGjdNTTz0lt9utWbNmSZJmzJihHj16aMGCBZKkX/ziF5o7d66WLl2qPn36qLS0VJIUFxenuLi4DnwrAGCOPcKqAc54DXDGt3re5/PrqKuuaSWlpaA0raZU1nh0tKpOR6vq9P7eylb7JUZHKjstttVKSv/0OPXsFiMbF29DF9PuMjJ9+nRVVFRo7ty5Ki0t1ahRo7Ry5cqWg1qLi4tltZ6ZjT7zzDPyeDz6yle+0urrzJs3Tz/+8Y8vLz0ABDmr1aIeSdHqkRStLw1Ma/W5k7We1isp5TXaX+FWyYlaVZ1q0Jbik9pSfLLVPvYIq/qlxrY6JqV/Wpz6pcVyITeELC4HDwBBpq7BqwMVZ0Y+zb8eqHTL0+g77z4Wi9SzW3TTwbNnraRkp8WpGxd1gyHcmwYAuhivz6/PTpzSvorqVisp+8prVHWq4YL7pcTaW6+kpMcpOy1WmYnR3K8HnYoyAgBhwu/3q7Km9chn/+ljU46cdbXZz4uOtKlfWqz6pcUpI8Gh9PgopcU7lB7vUHqCQ2nxUUqIiuBAWlwyyggAQO76Rh2ocLespuwvd2tfRY0OVbrV6Pviv/6jIq2nC0pUU0mJdyg94azSEh+l9ASHkmPsrLLgHJ1yOXgAQGiJdURoRM9EjeiZ2Or5Bq9Pxcdrta+8Rp8ec6vcVa/y6nqVV9epvLpeFa56Vdc3qq7Bp5Ljp1Ry/NRFXyfCalFqXNOKSnp806rKmcLSVGCanndwATicgzICAGEo0mZturZJ2oUvsXDK420pJ01l5XRRqT5dXFx1qqiu1zG3R40+v0pddSp1XXgs1Cw51t5STJpXVtLP83G0nbODwgVlBABwXtF2m3qnxKp3SuxFt2vw+lRZU996deX0xxVnlZnKmno1+vw67vbouNujXaXVF/268Y4IpZ1dVJoLTMLZY6MoJURzXEuoo4wAAC5LpM2q7onR6p4YfdHtfD6/jtd6zlllqfhcgSmvrlNdg0/V9Y2qrmg65uViHBHWc45haf747DKTEstxLcGKMgIACAjr6eNKUuMcGqoLH8zo9/tVXd/YUloqPjcmOvvj6rpG1Tf6dPjEKR0+cfHjWmxWi1Lj7K3PGop3KC2hqah0i7GrW2ykusXYlRQTKUcEY6JAoYwAAIKKxWJRQlSkEqIi1T/94rcNqWvwti4t56yyNI2Kjrk98vr8KnPVq8xV36YcMXZbSzFpKip2dYuJVFJM06+tPhdjV1JspOIdjIwuBWUEABCyoiJt6pUSo14pMRfdrsHr07EazznjoOaVlhO1Hp2o9ehkbYNO1nrk80u1Hq9qPaf02cmLr7icLcJqOW9ZSTq94pLc/NxZxSYpOlIRYX6GEWUEANDlRdqsykiMUkZi1Bdu6/P5VV3X2FJQTtR6dMLd0FJWzv71xOnyctztUX2jT40+vyprmg7WbY/4qIjTKyzNReXsVZfm1ZgzRSY5xt6lzjaijAAAcBar1aLEmEglxkSqjy5+JtHZTnm8rVZYWsqKu+nXM+WmqcCccHvkqmuUJFXXNaq6rlHFx9ue0xFh/dwYqfUIqfVzTc8nREUG5UG8lBEAADpAtN2maHu0MpMuflbR2Rq9PlWdami1wvL5VZfWHzf92uD1q77R1+ZruzSzWqTE6PMfA/ON8b2/8DTuzkIZAQDAkAibVSlxDqXEOdq8j9/vV01947mlxX3W2OjsIuNu+tjt8crn1+lVmgapsvUp01NHdKeMAACAL2axWBQfFan4qEhlJV/8wN2z1Td6VXW6iDSNkjw67j7zcc9ubV/R6WiUEQAAwoAjwqb0BJvSE774IN5AC+9ziQAAgHGUEQAAYBRlBAAAGEUZAQAARlFGAACAUZQRAABgFGUEAAAYRRkBAABGUUYAAIBRlBEAAGAUZQQAABhFGQEAAEZRRgAAgFEhcddev98vSXK5XIaTAACAtmr+ud38c/xCQqKMVFdXS5KysrIMJwEAAO1VXV2txMTEC37e4v+iuhIEfD6fjhw5ovj4eFkslg77ui6XS1lZWSopKVFCQkKHfd1QEu7fg3B//xLfA95/eL9/ie9BZ75/v9+v6upqZWZmymq98JEhIbEyYrVa1bNnz077+gkJCWH5G/Bs4f49CPf3L/E94P2H9/uX+B501vu/2IpIMw5gBQAARlFGAACAUWFdRhwOh+bNmyeHw2E6ijHh/j0I9/cv8T3g/Yf3+5f4HgTD+w+JA1gBAEDXFdYrIwAAwDzKCAAAMIoyAgAAjKKMAAAAo8K6jCxcuFB9+vRRVFSUcnNztXHjRtORAua9997TtGnTlJmZKYvFojfeeMN0pIBasGCBrrzySsXHxys9PV233nqrdu/ebTpWwDzzzDMaOXJky0WOJkyYoLffftt0LGMee+wxWSwWPfDAA6ajBMyPf/xjWSyWVo/BgwebjhVQn332mb7xjW8oJSVF0dHRGjFihDZt2mQ6VsD06dPnnN8DFotFs2fPDniWsC0jy5cvV35+vubNm6ctW7YoJydHU6ZMUXl5ueloAeF2u5WTk6OFCxeajmLEunXrNHv2bG3YsEEFBQVqaGjQ5MmT5Xa7TUcLiJ49e+qxxx7T5s2btWnTJt1www368pe/rO3bt5uOFnAff/yxnn32WY0cOdJ0lIAbNmyYjh492vL44IMPTEcKmBMnTmjixImKjIzU22+/rR07dujxxx9Xt27dTEcLmI8//rjV//+CggJJ0le/+tXAh/GHqXHjxvlnz57d8t9er9efmZnpX7BggcFUZkjyv/7666ZjGFVeXu6X5F+3bp3pKMZ069bN/9xzz5mOEVDV1dX+AQMG+AsKCvzXXnut//777zcdKWDmzZvnz8nJMR3DmIceesh/9dVXm44RVO6//35/dna23+fzBfy1w3JlxOPxaPPmzcrLy2t5zmq1Ki8vT+vXrzeYDKZUVVVJkpKTkw0nCTyv16tly5bJ7XZrwoQJpuME1OzZs3XLLbe0+rsgnOzdu1eZmZnq16+f7rjjDhUXF5uOFDB/+ctfNHbsWH31q19Venq6Ro8erSVLlpiOZYzH49Gf/vQnfetb3+rQG9K2VViWkcrKSnm9XjmdzlbPO51OlZaWGkoFU3w+nx544AFNnDhRw4cPNx0nYLZu3aq4uDg5HA595zvf0euvv66hQ4eajhUwy5Yt05YtW7RgwQLTUYzIzc3VSy+9pJUrV+qZZ57RwYMHdc0116i6utp0tIA4cOCAnnnmGQ0YMECrVq3Svffeq+9973t6+eWXTUcz4o033tDJkyf1zW9+08jrh8Rde4HONHv2bG3bti2s5uWSNGjQIBUVFamqqkqvvfaaZs6cqXXr1oVFISkpKdH999+vgoICRUVFmY5jxNSpU1s+HjlypHJzc9W7d2+9+uqr+va3v20wWWD4fD6NHTtW8+fPlySNHj1a27Zt06JFizRz5kzD6QLv+eef19SpU5WZmWnk9cNyZSQ1NVU2m01lZWWtni8rK1NGRoahVDDhvvvu09/+9je9++676tmzp+k4AWW329W/f3+NGTNGCxYsUE5Ojn7zm9+YjhUQmzdvVnl5ua644gpFREQoIiJC69at029/+1tFRETI6/WajhhwSUlJGjhwoPbt22c6SkB07979nOI9ZMiQsBpVNfv000+1evVq3XXXXcYyhGUZsdvtGjNmjAoLC1ue8/l8KiwsDLuZebjy+/2677779Prrr2vNmjXq27ev6UjG+Xw+1dfXm44REDfeeKO2bt2qoqKilsfYsWN1xx13qKioSDabzXTEgKupqdH+/fvVvXt301ECYuLEieeczr9nzx717t3bUCJzXnzxRaWnp+uWW24xliFsxzT5+fmaOXOmxo4dq3Hjxumpp56S2+3WrFmzTEcLiJqamlb/Ajp48KCKioqUnJysXr16GUwWGLNnz9bSpUv15ptvKj4+vuVYocTEREVHRxtO1/nmzJmjqVOnqlevXqqurtbSpUu1du1arVq1ynS0gIiPjz/n+KDY2FilpKSEzXFD3//+9zVt2jT17t1bR44c0bx582Sz2XTbbbeZjhYQDz74oK666irNnz9fX/va17Rx40YtXrxYixcvNh0toHw+n1588UXNnDlTEREGK0HAz98JIr/73e/8vXr18tvtdv+4ceP8GzZsMB0pYN59912/pHMeM2fONB0tIM733iX5X3zxRdPRAuJb3/qWv3fv3n673e5PS0vz33jjjf533nnHdCyjwu3U3unTp/u7d+/ut9vt/h49evinT5/u37dvn+lYAfXXv/7VP3z4cL/D4fAPHjzYv3jxYtORAm7VqlV+Sf7du3cbzWHx+/1+MzUIAAAgTI8ZAQAAwYMyAgAAjKKMAAAAoygjAADAKMoIAAAwijICAACMoowAAACjKCMAAMAoyggAADCKMgIAAIyijAAAAKMoIwAAwKj/H6AzVceQ0P+yAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load LoRA"
      ],
      "metadata": {
        "id": "nxzSoH2Zbr6c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n",
        "# Use the same LoRA rank that you trained\n",
        "#gemma_lm.backbone.enable_lora(rank=4)\n",
        "\n",
        "# Load pre-trained LoRA weights\n",
        "#gemma_lm.backbone.load_lora_weights(f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{train_epoch}.lora.h5\")"
      ],
      "metadata": {
        "id": "qE8I2RXrbt9o"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Try a different sampler\n",
        "\n",
        "The top-K algorithm randomly picks the next token from the tokens of top K probability."
      ],
      "metadata": {
        "id": "ipg1u_wEKTxG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gemma_lm.compile(sampler=\"top_k\")\n",
        "text_gen(\"Write a title\")\n",
        "text_gen(\"Write a title\")\n",
        "text_gen(\"Write a title\")\n",
        "text_gen(\"Write a title\")\n",
        "text_gen(\"Write a title\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nV5mD_HqKZRF",
        "outputId": "3fcfdec3-527c-451d-cad8-559f5dac5d24"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "कालेज के दिन<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 16.84s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगalar<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.84s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिलवाले दुल्हनिया ले जाएंगे<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.10s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Try a slight different prompts"
      ],
      "metadata": {
        "id": "3m1XaCrlMu3Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_gen(\"Write a music title\")\n",
        "text_gen(\"Write a poem title\")\n",
        "text_gen(\"Write a blog title\")\n",
        "text_gen(\"Write a movie title\")\n",
        "text_gen(\"Write a novel title\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qC-MLxYWM1HU",
        "outputId": "f1fa3552-b4c3-496b-9ddd-0958ebe4ddf2"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a music title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "प्यार के रंग<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.68s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a poem title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिल है कि मानता नहीं<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a blog title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "प्यार के रंग<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.69s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a movie title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "जंगल में मंगल<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 0.82s\n",
            "\n",
            "Gemma output:\n",
            "<start_of_turn>user\n",
            "Write a novel title<end_of_turn>\n",
            "<start_of_turn>model\n",
            "दिलवाले दुल्हनिया ले जाएंगे<end_of_turn>\n",
            "TOTAL TIME ELAPSED: 1.11s\n"
          ]
        }
      ]
    }
  ]
}