notebooks/part-2-annotate-with-gliner-review-in-argilla.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "82G2vlxphIlD"
   },
   "source": [
    "# 💫  Explore and analyze NER predictions\n",
    "\n",
    "In this tutorial, we will learn to log [spaCy](https://spacy.io/) Name Entity Recognition (NER) predictions.\n",
    "\n",
    "This is useful for:\n",
    "\n",
    "- 🧐Evaluating pre-trained models.\n",
    "- 🔎Spotting frequent errors both during development and production.\n",
    "- 📈Annotating records to create an gold-standard evaluation dataset.\n",
    "\n",
    "\n",
    "Reference: https://docs.argilla.io/en/latest/tutorials/notebooks/labelling-tokenclassification-spacy-pretrained.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u8phxyG-hIlF"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "In this tutorial, we will learn how to explore and analyze spaCy NER pipelines in an easy way.\n",
    "\n",
    "We will load the [*Gutenberg Time*](https://huggingface.co/datasets/gutenberg_time) dataset from the Hugging Face Hub and use a transformer-based spaCy model for detecting entities in this dataset and log the detected entities into an Argilla dataset. This dataset can be used for exploring the quality of predictions and for creating a new training set, by correcting, adding and validating entities via human annotation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, run your Argilla server if you haven't already:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the argilla docker container in your terminal (or use the %%bash magic)\n",
    "# This may take a couple of minutes to spin up the container\n",
    "# docker run -d --name quickstart -p 6900:6900 argilla/argilla-quickstart:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zp0MOU97hIlG"
   },
   "source": [
    "Let's import the Argilla module for reading and writing data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "gG2ZaK1OhIlG"
   },
   "outputs": [],
   "source": [
    "import argilla as rg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H1rV-N6phIlG"
   },
   "source": [
    "If you are running Argilla using the Docker quickstart image or Hugging Face Spaces, you need to init the Argilla client with the `URL` and `API_KEY`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "dT-F6e7zhIlG"
   },
   "outputs": [],
   "source": [
    "# Replace api_url with the url to your HF Spaces URL if using Spaces\n",
    "# Replace api_key if you configured a custom API key\n",
    "# Replace workspace with the name of your workspace\n",
    "# default argilla username is 'argilla'\n",
    "# default argilla password is '12345678'\n",
    "WORKSPACE='admin'\n",
    "rg.init(\n",
    "    api_url=\"http://localhost:6900\",                      # If you are using the docker container\n",
    "    # api_url=\"https://jackboyla-zero-shot-lab.hf.space\", # If you are using HF Spaces\n",
    "    api_key=\"owner.apikey\",\n",
    "    workspace=WORKSPACE\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yEU7UgpthIlH"
   },
   "source": [
    "Finally, let's include the imports we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "GktYugZxhIlH"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from datasets import load_dataset, load_from_disk\n",
    "import pandas as pd\n",
    "import spacy\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QPJ5kkX7hIlH"
   },
   "source": [
    "## Our dataset\n",
    "For this tutorial, our default dataset is the [*Gutenberg Time*](https://huggingface.co/datasets/gutenberg_time) dataset from the Hugging Face Hub. It contains all explicit time references in a dataset of 52,183 novels whose full text is available via Project Gutenberg. From extracts of novels, we are surely going to find some NER entities.\n",
    "\n",
    "If you are following the full lab, you can also load the dataset you generated in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "MEURV43lhIlH",
    "outputId": "c8ef60fe-55d6-441f-b65b-a2ce6e3801cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                               input  \\\n",
      "0  When you would like the property photographed....   \n",
      "1  I want to go see this property in person! MLS#...   \n",
      "2  Va. 7 property for Preston Propane before the ...   \n",
      "3  I was searching for a Property and found this ...   \n",
      "4  I would like more information about 7362 Horiz...   \n",
      "\n",
      "                                              output  \\\n",
      "0  ['property <> real estate <> physical entity r...   \n",
      "1  ['MLS# 234382 <> Real Estate Property <> Uniqu...   \n",
      "2  ['Va. 7 <> Street Address <> Physical location...   \n",
      "3  ['Property <> Real estate <> Physical asset wi...   \n",
      "4  ['7362 Horizon Drive West Palm Beach, FL 33412...   \n",
      "\n",
      "                                          embeddings  \n",
      "0  [0.016540057957172394, 0.04142068699002266, -0...  \n",
      "1  [0.003052351763471961, 0.013402777723968029, -...  \n",
      "2  [0.01409836020320654, 0.02464674413204193, 0.0...  \n",
      "3  [0.009898669086396694, 0.002850162098184228, -...  \n",
      "4  [0.011095961555838585, 0.03647858276963234, 0....  \n"
     ]
    }
   ],
   "source": [
    "# when using this notebook standalone, choose a dataset from the hub\n",
    "# dataset = load_dataset(\"gutenberg_time\", split=\"train\", streaming=True)\n",
    "\n",
    "# when using this notebook as part of the full lab, load the dataset you created in the previous step\n",
    "DATASET_PATH = 'data/sampled_dataset/'\n",
    "dataset = load_from_disk(DATASET_PATH)\n",
    "\n",
    "# Let's have a look at the first 5 examples of the train set.\n",
    "try:\n",
    "    print(pd.DataFrame(dataset.take(5)))\n",
    "except AttributeError:\n",
    "    print(pd.DataFrame(dataset[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XVmbkS2WhIlH"
   },
   "source": [
    "## Annotating with GLiNER and Logging NER entities into Argilla\n",
    "\n",
    "\n",
    "Let's instantiate a spaCy transformer `nlp` pipeline and apply it to the first N examples in our dataset, collecting the *tokens* and *NER entities*.\n",
    "\n",
    "We're going to use a [GLiNER](https://github.com/urchade/GLiNER) model to perform zero shot NER. This means we can provide any entity labels we like!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataset_name = \"gutenberg_spacy_ner\"\n",
    "dataset_name = \"generated_dataset\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5KuStg_vhIlH"
   },
   "source": [
    "If you go to the `<dataset_name>` dataset in Argilla you can explore the predictions of this model.\n",
    "\n",
    "You can:\n",
    "\n",
    "- Filter records containing specific entity types,\n",
    "- See the most frequent \"mentions\" or surface forms for each entity. Mentions are the string values of specific entity types, for example, \"1 month\" can be the mention of a duration entity. This is useful for error analysis, to quickly see potential issues and problematic entity types,\n",
    "- Use the free-text search to find records containing specific words,\n",
    "- And validate, include or reject specific entity annotations to build a new training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jackboylan/miniconda3/envs/dcu_masters/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:550: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bill Gates person\n",
      "Microsoft business\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from gliner_spacy.pipeline import GlinerSpacy\n",
    "\n",
    "# Gliner model options https://huggingface.co/urchade \n",
    "gliner_model = \"urchade/gliner_largev2\"\n",
    "\n",
    "# Define your domain here: the list of entity types you expect to see\n",
    "zero_shot_labels = [\"person\", \"organization\", \"email\", \"sports team\", \"business\"]\n",
    "\n",
    "# Configuration for GLiNER integration\n",
    "custom_spacy_config = {\n",
    "    \"gliner_model\": gliner_model,\n",
    "    \"chunk_size\": 250,\n",
    "    \"labels\": zero_shot_labels,\n",
    "    \"style\": \"ent\"\n",
    "}\n",
    "\n",
    "# Initialize a blank English spaCy pipeline and add GLiNER\n",
    "nlp = spacy.blank(\"en\")\n",
    "nlp.add_pipe(\"gliner_spacy\", config=custom_spacy_config)\n",
    "\n",
    "# Example\n",
    "text = \"This is a text about Bill Gates and Microsoft.\"\n",
    "\n",
    "# Process the text with the pipeline\n",
    "doc = nlp(text)\n",
    "\n",
    "# Output detected entities\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97ff91751746440296f119885a11ea6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14a0ae060233470ba45b3ee2fba22554",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span> records logged to <a href=\"https://jackboyla-zero-shot-lab.hf.space/datasets/admin/generated_dataset\" target=\"_blank\"><span style=\"color: #0000ff; text-decoration-color: #0000ff; text-decoration: underline\">https://jackboyla-zero-shot-lab.hf.space/datasets/admin/generated_dataset</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36m5\u001b[0m records logged to \u001b]8;id=142127;https://jackboyla-zero-shot-lab.hf.space/datasets/admin/generated_dataset\u001b\\\u001b[4;94mhttps://jackboyla-zero-shot-lab.hf.space/datasets/admin/generated_dataset\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "BulkResponse(dataset='generated_dataset', processed=5, failed=0)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# Creating an empty record list to save all the records\n",
    "records = []\n",
    "\n",
    "# Gutenberg Dataset text field\n",
    "# TEXT_FIELD = \"tok_context\"\n",
    "# Generated Dataset text field\n",
    "TEXT_FIELD = \"input\"\n",
    "# Iterate over the first 50 examples of the dataset\n",
    "NUM_EXAMPLES = 5\n",
    "try:\n",
    "    examples_to_annotate = list(dataset.take(NUM_EXAMPLES))\n",
    "except AttributeError:\n",
    "    examples_to_annotate = dataset.select(range(NUM_EXAMPLES))\n",
    "\n",
    "for record in tqdm(examples_to_annotate):\n",
    "    # We only need the text of each instance\n",
    "    text = record[TEXT_FIELD]\n",
    "\n",
    "    # spaCy Doc creation\n",
    "    doc = nlp(text)\n",
    "\n",
    "    # Entity annotations\n",
    "    entities = [(ent.label_, ent.start_char, ent.end_char) for ent in doc.ents]\n",
    "\n",
    "    # Pre-tokenized input text\n",
    "    tokens = [token.text for token in doc]\n",
    "\n",
    "    # Argilla TokenClassificationRecord list\n",
    "    records.append(\n",
    "        rg.TokenClassificationRecord(\n",
    "            text=text,\n",
    "            tokens=tokens,\n",
    "            prediction=entities,\n",
    "            prediction_agent=gliner_model,\n",
    "        )\n",
    "    )\n",
    "\n",
    "rg.log(records=records, name=dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tLqpJHcDhIlI"
   },
   "source": [
    "If you now go to you dataset in the argilla UI, you can explore and compare the results of the model.\n",
    "\n",
    "To only see predictions of a specific model, you can use the `predicted by` filter, which comes from the `prediction_agent` parameter of your `TextClassificationRecord`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export Annotated Argilla Dataset\n",
    "\n",
    "GLiNER models return the character indices of the detected entities. For fine-tuning, we need the token indices. So we have to do some data gymnastics below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def char_to_token_indices(text, tokens, entities):\n",
    "    \"\"\"\n",
    "    Convert character span indices in entities to token indices.\n",
    "    \n",
    "    Args:\n",
    "    - text: The original text as a single string.\n",
    "    - tokens: A list of tokens.\n",
    "    - entities: A list of entities with character start and end indices.\n",
    "    \n",
    "    Returns:\n",
    "    - A list of entities with token start and end indices.\n",
    "    \"\"\"\n",
    "    # Calculate the character start index of each token\n",
    "    token_char_spans = []\n",
    "    current_char_index = 0\n",
    "    for token in tokens:\n",
    "        start_index = text.find(token, current_char_index)\n",
    "        end_index = start_index + len(token)\n",
    "        token_char_spans.append((start_index, end_index))\n",
    "        current_char_index = end_index\n",
    "\n",
    "    # Convert character indices to token indices for each entity\n",
    "    converted_entities = []\n",
    "    for entity in entities:\n",
    "        entity_start, entity_end = entity['start'], entity['end']\n",
    "        entity_start_token = None\n",
    "        entity_end_token = None\n",
    "        \n",
    "        # Find the tokens that the entity start and end indices fall into\n",
    "        for i, (start, end) in enumerate(token_char_spans):\n",
    "            if start <= entity_start < end:\n",
    "                entity_start_token = i\n",
    "            if start < entity_end <= end:\n",
    "                entity_end_token = i + 1\n",
    "                break  # Stop looking once we've found the end token\n",
    "        \n",
    "        if entity_start_token is not None and entity_end_token is not None:\n",
    "            converted_entities.append([entity_start_token, entity_end_token, entity['label']])\n",
    "        else:\n",
    "            print('Error on entity:', entity, 'Tokens:', tokens)\n",
    "    \n",
    "    return converted_entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "OUTPUT_ROOT = Path('data/')\n",
    "OUTPUT_ROOT.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "# exported dataset after review\n",
    "dataset_rg = rg.load(dataset_name)\n",
    "\n",
    "# export your Argilla Dataset to a datasets Dataset\n",
    "dataset_ds = dataset_rg.to_datasets()\n",
    "\n",
    "'''\n",
    "Dataset({\n",
    "    features: ['text', 'tokens', 'prediction', 'prediction_agent', 'annotation', 'annotation_agent', 'vectors', 'id', 'metadata', 'status', 'event_timestamp', 'metrics'],\n",
    "    num_rows: 200\n",
    "})\n",
    "'''\n",
    "\n",
    "# format the dataset to GLiNER training format {'tokenized_text' [], 'ner': [ [start_token_i, end_token_i, label], ...], ...}\n",
    "\n",
    "# if it's been annotated, it goes to the evaluation set\n",
    "train_set = []\n",
    "eval_set = []\n",
    "for record in dataset_ds:\n",
    "    \n",
    "    converted_entities = char_to_token_indices(record['text'], record['tokens'], record['annotation'] or record['prediction'])\n",
    "\n",
    "    if record['annotation'] is not None:\n",
    "\n",
    "        eval_set.append(\n",
    "            {\n",
    "                'tokenized_text': record['tokens'], \n",
    "                'ner': converted_entities\n",
    "            }\n",
    "        )\n",
    "\n",
    "    # otherwise, it goes to the weakly annotated train set\n",
    "    else:\n",
    "        train_set.append(\n",
    "            {\n",
    "                'tokenized_text': record['tokens'], \n",
    "                'ner': converted_entities\n",
    "            }\n",
    "        )\n",
    "\n",
    "\n",
    "file_path = OUTPUT_ROOT / f\"{dataset_name}_train.jsonl\"\n",
    "with open(file_path, 'w') as file:\n",
    "    for entry in train_set:\n",
    "        json.dump(entry, file)\n",
    "        file.write('\\n')\n",
    "\n",
    "file_path = OUTPUT_ROOT / f\"{dataset_name}_eval.jsonl\"\n",
    "with open(file_path, 'w') as file:\n",
    "    for entry in eval_set:\n",
    "        json.dump(entry, file)\n",
    "        file.write('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data = {\"tokenized_text\": [\"A\", \"portable\", \"bridge\", \"had\", \"been\", \"prepared\", \"for\", \"crossing\", \"the\", \"canals\", \"which\", \"intersected\", \"the\", \"causeway\", \";\", \"the\", \"intention\", \"being\", \"that\", \"it\", \"should\", \"be\", \"laid\", \"across\", \"a\", \"canal\", \",\", \"that\", \"the\", \"army\", \"should\", \"pass\", \"over\", \"it\", \",\", \"and\", \"that\", \"it\", \"should\", \"then\", \"be\", \"carried\", \"forward\", \"to\", \"the\", \"next\", \"gap\", \"in\", \"the\", \"causeway\", \".\", \"This\", \"was\", \"a\", \"most\", \"faulty\", \"arrangement\", \",\", \"necessitating\", \"frequent\", \"and\", \"long\", \"delays\", \",\", \"and\", \"entailing\", \"almost\", \"certain\", \"disaster\", \".\", \"Had\", \"three\", \"such\", \"portable\", \"bridges\", \"been\", \"constructed\", \",\", \"the\", \"column\", \"could\", \"have\", \"crossed\", \"the\", \"causeway\", \"with\", \"comparatively\", \"little\", \"risk\", \";\", \"and\", \"there\", \"was\", \"no\", \"reason\", \"why\", \"these\", \"bridges\", \"should\", \"not\", \"have\", \"been\", \"constructed\", \",\", \"as\", \"they\", \"could\", \"have\", \"been\", \"carried\", \",\", \"without\", \"difficulty\", \",\", \"by\", \"the\", \"Tlascalans\", \".\", \"At\", \"midnight\", \"the\", \"troops\", \"were\", \"in\", \"readiness\", \"for\", \"the\", \"march\", \".\", \"Mass\", \"was\", \"performed\", \"by\", \"Father\", \"Olmedo\", \";\", \"and\", \"at\", \"one\", \"o'clock\", \"on\", \"July\", \"1st\", \",\", \"1520\", \",\", \"the\", \"Spaniards\", \"sallied\", \"out\", \"from\", \"the\", \"fortress\", \"that\", \"they\", \"had\", \"so\", \"stoutly\", \"defended\", \".\", \"Silence\", \"reigned\", \"in\", \"the\", \"city\", \".\", \"As\", \"noiselessly\", \"as\", \"possible\", \",\", \"the\", \"troops\", \"made\", \"their\", \"way\", \"down\", \"the\", \"broad\", \"street\", \",\", \"expecting\", \"every\", \"moment\", \"to\", \"be\", \"attacked\", \";\", \"but\", \"even\", \"the\", \"tramping\", \"of\", \"the\", \"horses\", \",\", \"and\", \"the\", \"rumbling\", \"of\", \"the\", \"baggage\", \"wagons\", \"and\", \"artillery\", \"did\", \"not\", \"awake\", \"the\", \"sleeping\", \"Mexicans\", \",\", \"and\", \"the\", \"head\", \"of\", \"the\", \"column\", \"arrived\", \"at\", \"the\", \"head\", \"of\", \"the\", \"causeway\", \"before\", \"they\", \"were\", \"discovered\", \".\"], \"ner\": [[29, 30, \"organization\"], [116, 117, \"organization\"], [121, 122, \"organization\"], [133, 135, \"person\"], [147, 148, \"organization\"], [210, 211, \"person\"], [217, 218, \"organization\"]]}\n",
    "data = train_set[10]\n",
    "\n",
    "import spacy\n",
    "from spacy.tokens import Span, Doc\n",
    "\n",
    "# Create a Doc from the tokenized text\n",
    "doc = Doc(nlp.vocab, words=data[\"tokenized_text\"])\n",
    "ents = []\n",
    "for start, end, label in data[\"ner\"]:\n",
    "    span = Span(doc, start, end, label=label)\n",
    "    ents.append(span)\n",
    "doc.ents = ents\n",
    "\n",
    "# Visualize the NER entities\n",
    "spacy.displacy.render(doc, style=\"ent\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cKuSFsW8hIlI"
   },
   "source": [
    "## Appendix: Log datasets to the Hugging Face Hub\n",
    "\n",
    "Here we will show you an example of how you can push an Argilla dataset (records) to the [Hugging Face Hub](https://huggingface.co/datasets).\n",
    "In this way, you can effectively version any of your Argilla datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "i7P0CEANhIlI"
   },
   "outputs": [],
   "source": [
    "# records = rg.load(dataset_name)\n",
    "# records.to_datasets().push_to_hub(\"<name of the dataset on the HF Hub>\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "metadata": {
   "interpreter": {
    "hash": "0f338a8622467eba0ef87b9a79c52cc260cef0b0d60c3c739596fb787bf801dd"
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "8874e298d2bce9702a08b32d5709c9f02d53b2045f1d246836c6e4c8123e6782"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}