Chapter06_intro_ginza.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# spaCy と GiNZA\n",
    "\n",
    "**spaCy** [^spaCy] は自然言語を処理するためのフレームワークです。\n",
    "\n",
    "日本語のモデルとしては **GiNZA** [^GiNZA] を利用することができます。GiNZA v5 では２種類の言語モデルが備わっています。1つは v4 まで使われていたディープラーニングの CNN で学習されたモデル(`ja_ginza`)ですが、新たに Transformers で学習されたモデル(`ja_ginza_electra`)が提供されるようになりました。なお、GiNZAでは形態素解析に Sudachi [^Sudachi] が使われています。\n",
    "\n",
    "[^spaCy]: <https://spacy.io/>\n",
    "[^GiNZA]: <https://megagonlabs.github.io/ginza/>\n",
    "[^Sudachi]: <https://github.com/WorksApplications/Sudachi>\n",
    "\n",
    "以下では、`ja_ginza` を使います。\n",
    "\n",
    "Pythonで spaCy+GiNZA を試してみます。まず言語モデルを読み込ます。初めて実行した場合、モデルがダウンロードされるのですこし時間がかかります（モデルの容量は 1GB を超えることが多いので注意してください）。\n",
    "\n",
    "なお、spaCy は Anaconda には付属していませんので、ユーザー自身でインストールする必要がありますが、利用しているパソコン環境によって方法が異なってきます。\n",
    "\n",
    "**なお、本書のp.24で説明した通り、requirements.txt を利用してインストールすると、上記のパッケージはすでに導入されています。ただしginzaについては ディープラーニングの CNN で学習されたモデル(`ja_ginza`) で学習されたモデルがインストールされています。Transformers で学習されたモデル(`ja_ginza_electra`)と `spacy[cuda[*]` を利用したい場合は、いったん `!pip uninstall ja_ginza spacy` でアンインストールし、改めて個別にインストールしてください。\n",
    "requirements.txt を利用せずにインストールする場合は `!pip install` コマンドを使って個別にインストールすることになります。その場合、最新のライブラリがインストールされますが、仕様が変わっていることがあるため、本書の記載どおりでは動作しないコードがあるかもしれません。筆者側で補足できた範囲で、このGitHub レポジトリでサポートしていきます。**\n",
    "\n",
    "詳細は本書第6章の解説を参照してください。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting ginza\n",
      "  Downloading ginza-5.1.2-py3-none-any.whl (20 kB)\n",
      "Collecting ja-ginza\n",
      "  Downloading ja_ginza-5.1.2-py3-none-any.whl (59.1 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.1/59.1 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.5.0,>=3.2.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from ginza) (3.4.1)\n",
      "Collecting SudachiDict-core>=20210802\n",
      "  Downloading SudachiDict-core-20220729.tar.gz (9.1 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting plac>=1.3.3\n",
      "  Downloading plac-1.3.5-py2.py3-none-any.whl (22 kB)\n",
      "Collecting SudachiPy<0.7.0,>=0.6.2\n",
      "  Downloading SudachiPy-0.6.6-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.2 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: catalogue<2.1.0,>=2.0.6 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (2.0.8)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (3.3.0)\n",
      "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (0.4.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (1.0.8)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (1.9.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (2.4.4)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (2.0.6)\n",
      "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (8.1.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (2.28.1)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (1.23.2)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (0.10.1)\n",
      "Requirement already satisfied: setuptools in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (65.1.0)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (3.0.7)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (0.6.2)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (4.64.0)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (3.0.10)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (1.0.3)\n",
      "Requirement already satisfied: packaging>=20.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (21.3)\n",
      "Requirement already satisfied: jinja2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy<3.5.0,>=3.2.0->ginza) (3.1.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from packaging>=20.0->spacy<3.5.0,>=3.2.0->ginza) (3.0.9)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.2.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from pathy>=0.3.5->spacy<3.5.0,>=3.2.0->ginza) (5.2.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4->spacy<3.5.0,>=3.2.0->ginza) (4.3.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.2.0->ginza) (3.3)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.2.0->ginza) (2.1.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.2.0->ginza) (1.26.11)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.2.0->ginza) (2022.6.15)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.2.0->ginza) (0.7.8)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.5.0,>=3.2.0->ginza) (8.1.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from jinja2->spacy<3.5.0,>=3.2.0->ginza) (2.1.1)\n",
      "Building wheels for collected packages: SudachiDict-core\n",
      "  Building wheel for SudachiDict-core (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for SudachiDict-core: filename=SudachiDict_core-20220729-py3-none-any.whl size=71570371 sha256=dd4ad47161ea1b0f323f8c0871e7aa33f165597e781412af354301f383ac293f\n",
      "  Stored in directory: /home/ishida/.cache/pip/wheels/74/3f/5e/ca07f864f6958b359fe55207394428506fcc4f8ba216f85fad\n",
      "Successfully built SudachiDict-core\n",
      "Installing collected packages: SudachiPy, plac, SudachiDict-core, ginza, ja-ginza\n",
      "Successfully installed SudachiDict-core-20220729 SudachiPy-0.6.6 ginza-5.1.2 ja-ginza-5.1.2 plac-1.3.5\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install -U ginza ja-ginza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting spacy\n",
      "  Using cached spacy-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.4 MB)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (2.28.1)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (3.0.7)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (1.23.2)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (3.3.0)\n",
      "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (0.4.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (1.0.8)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.9 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (3.0.10)\n",
      "Requirement already satisfied: jinja2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (3.1.2)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (1.9.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (21.3)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (0.10.1)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (4.64.0)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (2.4.4)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (1.0.3)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (2.0.8)\n",
      "Requirement already satisfied: setuptools in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (65.1.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (2.0.6)\n",
      "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (8.1.0)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from spacy) (0.6.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from packaging>=20.0->spacy) (3.0.9)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.2.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from pathy>=0.3.5->spacy) (5.2.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4->spacy) (4.3.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2022.6.15)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.1.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.11)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from thinc<8.2.0,>=8.1.0->spacy) (0.7.8)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from typer<0.5.0,>=0.3.0->spacy) (8.1.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /mnt/2bddf92b-47f9-4809-95a5-b91e7f25af27/myData/GitHub/myPython/lib/python3.10/site-packages (from jinja2->spacy) (2.1.1)\n",
      "Installing collected packages: spacy\n",
      "Successfully installed spacy-3.4.1\n"
     ]
    }
   ],
   "source": [
    "!pip install -U spacy\n",
    "# !pip install -U spacy[cuda117]\n",
    "## 以下の出力は spacy をインストールした場合の例です"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-22 17:00:02.375717: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-08-22 17:00:02.376205: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-08-22 17:00:02.376353: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('ja_ginza')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "日本語文章を形態素に分解し、それぞれの品詞情報を得るには、テキストから `Doc` クラスオブジェクトを生成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_', '__bytes__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__pyx_vtable__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '_bulk_merge', '_context', '_get_array_attrs', '_realloc', '_vector', '_vector_norm', 'cats', 'char_span', 'copy', 'count_by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', 'from_dict', 'from_disk', 'from_docs', 'from_json', 'get_extension', 'get_lca_matrix', 'has_annotation', 'has_extension', 'has_unknown_spaces', 'has_vector', 'is_nered', 'is_parsed', 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'noun_chunks', 'noun_chunks_iterator', 'remove_extension', 'retokenize', 'sentiment', 'sents', 'set_ents', 'set_extension', 'similarity', 'spans', 'tensor', 'text', 'text_with_ws', 'to_array', 'to_bytes', 'to_dict', 'to_disk', 'to_json', 'to_utf8_array', 'user_data', 'user_hooks', 'user_span_hooks', 'user_token_hooks', 'vector', 'vector_norm', 'vocab']\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('ウクライナとロシアの間で戦争が始まった。この影響で世界中で物価が上昇している。')\n",
    "## docオブジェクトのアトリビュート一覧\n",
    "print(dir(doc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ウクライナとロシアの間で戦争が始まった。\n",
      "この影響で世界中で物価が上昇している。\n"
     ]
    }
   ],
   "source": [
    "for sent in doc.sents:\n",
    "    print(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ウクライナ\n",
      "と\n",
      "ロシア\n",
      "の\n",
      "間\n",
      "で\n",
      "戦争\n",
      "が\n",
      "始まっ\n",
      "た\n",
      "。\n",
      "この\n",
      "影響\n",
      "で\n",
      "世界中\n",
      "で\n",
      "物価\n",
      "が\n",
      "上昇\n",
      "し\n",
      "て\n",
      "いる\n",
      "。\n"
     ]
    }
   ],
   "source": [
    "for sent in doc.sents:\n",
    "    for token in sent:\n",
    "        print(token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------\n",
      "第1文：ウクライナとロシアの間で戦争が始まった。\n",
      "------------\n",
      "レンマ： ウクライナ 、品詞： PROPN 、品詞タグ： 名詞-固有名詞-地名-国\n",
      "レンマ： と 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： ロシア 、品詞： PROPN 、品詞タグ： 名詞-固有名詞-地名-国\n",
      "レンマ： の 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 間 、品詞： NOUN 、品詞タグ： 名詞-普通名詞-副詞可能\n",
      "レンマ： で 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 戦争 、品詞： NOUN 、品詞タグ： 名詞-普通名詞-サ変可能\n",
      "レンマ： が 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 始まる 、品詞： VERB 、品詞タグ： 動詞-一般\n",
      "レンマ： た 、品詞： AUX 、品詞タグ： 助動詞\n",
      "レンマ： 。 、品詞： PUNCT 、品詞タグ： 補助記号-句点\n",
      "---------\n",
      "第2文：この影響で世界中で物価が上昇している。\n",
      "------------\n",
      "レンマ： この 、品詞： DET 、品詞タグ： 連体詞\n",
      "レンマ： 影響 、品詞： NOUN 、品詞タグ： 名詞-普通名詞-サ変可能\n",
      "レンマ： で 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 世界中 、品詞： NOUN 、品詞タグ： 名詞-普通名詞-一般\n",
      "レンマ： で 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 物価 、品詞： NOUN 、品詞タグ： 名詞-普通名詞-一般\n",
      "レンマ： が 、品詞： ADP 、品詞タグ： 助詞-格助詞\n",
      "レンマ： 上昇 、品詞： VERB 、品詞タグ： 名詞-普通名詞-サ変可能\n",
      "レンマ： する 、品詞： AUX 、品詞タグ： 動詞-非自立可能\n",
      "レンマ： て 、品詞： SCONJ 、品詞タグ： 助詞-接続助詞\n",
      "レンマ： いる 、品詞： VERB 、品詞タグ： 動詞-非自立可能\n",
      "レンマ： 。 、品詞： PUNCT 、品詞タグ： 補助記号-句点\n"
     ]
    }
   ],
   "source": [
    "for i, sent in enumerate(doc.sents):\n",
    "    print(f'---------\\n第{i+1}文：{sent}\\n------------')\n",
    "    for token in sent:\n",
    "        print('レンマ：', token.lemma_, '、品詞：', token.pos_, '、品詞タグ：', token.tag_ )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "インデックス\t単語\tレンマ\t正規形\t読み\t品詞タグ\t活用情報\t品詞情報\t依存関係ラベル\t係り先インデックス\n",
      "0 \t ウクライナ \t ウクライナ \t ウクライナ \t ['ウクライナ'] \t PROPN \t [] \t 名詞-固有名詞-地名-国 \t nmod \t 2\n",
      "1 \t と \t と \t と \t ['ト'] \t ADP \t [] \t 助詞-格助詞 \t case \t 0\n",
      "2 \t ロシア \t ロシア \t ロシア \t ['ロシア'] \t PROPN \t [] \t 名詞-固有名詞-地名-国 \t nmod \t 4\n",
      "3 \t の \t の \t の \t ['ノ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 2\n",
      "4 \t 間 \t 間 \t 間 \t ['アイダ'] \t NOUN \t [] \t 名詞-普通名詞-副詞可能 \t obl \t 8\n",
      "5 \t で \t で \t で \t ['デ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 4\n",
      "6 \t 戦争 \t 戦争 \t 戦争 \t ['センソウ'] \t NOUN \t [] \t 名詞-普通名詞-サ変可能 \t nsubj \t 8\n",
      "7 \t が \t が \t が \t ['ガ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 6\n",
      "8 \t 始まっ \t 始まる \t 始まる \t ['ハジマッ'] \t VERB \t ['五段-ラ行;連用形-促音便'] \t 動詞-一般 \t ROOT \t 8\n",
      "9 \t た \t た \t た \t ['タ'] \t AUX \t ['助動詞-タ;終止形-一般'] \t 助動詞 \t aux \t 8\n",
      "10 \t 。 \t 。 \t 。 \t ['。'] \t PUNCT \t [] \t 補助記号-句点 \t punct \t 8\n",
      "EOS\n",
      "インデックス\t単語\tレンマ\t正規形\t読み\t品詞タグ\t活用情報\t品詞情報\t依存関係ラベル\t係り先インデックス\n",
      "11 \t この \t この \t 此の \t ['コノ'] \t DET \t [] \t 連体詞 \t det \t 12\n",
      "12 \t 影響 \t 影響 \t 影響 \t ['エイキョウ'] \t NOUN \t [] \t 名詞-普通名詞-サ変可能 \t obl \t 18\n",
      "13 \t で \t で \t で \t ['デ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 12\n",
      "14 \t 世界中 \t 世界中 \t 世界中 \t ['セカイジュウ'] \t NOUN \t [] \t 名詞-普通名詞-一般 \t obl \t 18\n",
      "15 \t で \t で \t で \t ['デ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 14\n",
      "16 \t 物価 \t 物価 \t 物価 \t ['ブッカ'] \t NOUN \t [] \t 名詞-普通名詞-一般 \t nsubj \t 18\n",
      "17 \t が \t が \t が \t ['ガ'] \t ADP \t [] \t 助詞-格助詞 \t case \t 16\n",
      "18 \t 上昇 \t 上昇 \t 上昇 \t ['ジョウショウ'] \t VERB \t [] \t 名詞-普通名詞-サ変可能 \t ROOT \t 18\n",
      "19 \t し \t する \t 為る \t ['シ'] \t AUX \t ['サ行変格;連用形-一般'] \t 動詞-非自立可能 \t aux \t 18\n",
      "20 \t て \t て \t て \t ['テ'] \t SCONJ \t [] \t 助詞-接続助詞 \t mark \t 18\n",
      "21 \t いる \t いる \t 居る \t ['イル'] \t VERB \t ['上一段-ア行;終止形-一般'] \t 動詞-非自立可能 \t fixed \t 20\n",
      "22 \t 。 \t 。 \t 。 \t ['。'] \t PUNCT \t [] \t 補助記号-句点 \t punct \t 18\n",
      "EOS\n"
     ]
    }
   ],
   "source": [
    "for sent in doc.sents:\n",
    "    print('インデックス\\t単語\\tレンマ\\t正規形\\t読み\\t品詞タグ\\t活用情報\\t品詞情報\\t依存関係ラベル\\t係り先インデックス')\n",
    "    for token in sent:\n",
    "        print(\n",
    "            token.i,\n",
    "            '\\t', token.orth_,\n",
    "            '\\t', token.lemma_,\n",
    "            '\\t', token.norm_,\n",
    "            '\\t', token.morph.get('Reading'),\n",
    "            '\\t', token.pos_,\n",
    "            '\\t', token.morph.get('Inflection'),\n",
    "            '\\t', token.tag_,\n",
    "            '\\t', token.dep_,\n",
    "            '\\t', token.head.i,\n",
    "        )\n",
    "    print('EOS')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOUNやADP, あるいは compound, case、nmod・nsubj というのは、Universal Dependency[^univ_dep] というプロジェクトで定義されているラベルです。\n",
    "依存関係ラベルと係り先インデックスというのは、グラフで表現した方がわかりやすいでしょう。\n",
    "\n",
    "[univ_dep]:  <https://universaldependencies.org/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"ja\" id=\"4852ccd5be2145a0bd0785136df77710-0\" class=\"displacy\" width=\"3725\" height=\"487.0\" direction=\"ltr\" style=\"max-width: none; height: 487.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">ウクライナ</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">と</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">ロシア</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">の</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">間</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">で</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">戦争</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">が</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1450\">始まっ</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1450\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1625\">た。</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1625\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1800\">この</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1800\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1975\">影響</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1975\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2150\">で</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2150\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2325\">世界中</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2325\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2500\">で</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2500\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2675\">物価</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2675\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2850\">が</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2850\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3025\">上昇</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3025\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3200\">し</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3200\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3375\">て</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3375\">SCONJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"397.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3550\">いる。</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3550\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-0\" stroke-width=\"2px\" d=\"M70,352.0 C70,177.0 390.0,177.0 390.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,354.0 L62,342.0 78,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-1\" stroke-width=\"2px\" d=\"M70,352.0 C70,264.5 210.0,264.5 210.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M210.0,354.0 L218.0,342.0 202.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-2\" stroke-width=\"2px\" d=\"M420,352.0 C420,177.0 740.0,177.0 740.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,354.0 L412,342.0 428,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-3\" stroke-width=\"2px\" d=\"M420,352.0 C420,264.5 560.0,264.5 560.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M560.0,354.0 L568.0,342.0 552.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-4\" stroke-width=\"2px\" d=\"M770,352.0 C770,89.5 1445.0,89.5 1445.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">obl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M770,354.0 L762,342.0 778,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-5\" stroke-width=\"2px\" d=\"M770,352.0 C770,264.5 910.0,264.5 910.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M910.0,354.0 L918.0,342.0 902.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-6\" stroke-width=\"2px\" d=\"M1120,352.0 C1120,177.0 1440.0,177.0 1440.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1120,354.0 L1112,342.0 1128,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-7\" stroke-width=\"2px\" d=\"M1120,352.0 C1120,264.5 1260.0,264.5 1260.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-7\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1260.0,354.0 L1268.0,342.0 1252.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-8\" stroke-width=\"2px\" d=\"M1470,352.0 C1470,264.5 1610.0,264.5 1610.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-8\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1610.0,354.0 L1618.0,342.0 1602.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-9\" stroke-width=\"2px\" d=\"M1820,352.0 C1820,264.5 1960.0,264.5 1960.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-9\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1820,354.0 L1812,342.0 1828,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-10\" stroke-width=\"2px\" d=\"M1995,352.0 C1995,2.0 3025.0,2.0 3025.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-10\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">obl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1995,354.0 L1987,342.0 2003,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-11\" stroke-width=\"2px\" d=\"M1995,352.0 C1995,264.5 2135.0,264.5 2135.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-11\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2135.0,354.0 L2143.0,342.0 2127.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-12\" stroke-width=\"2px\" d=\"M2345,352.0 C2345,89.5 3020.0,89.5 3020.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-12\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">obl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2345,354.0 L2337,342.0 2353,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-13\" stroke-width=\"2px\" d=\"M2345,352.0 C2345,264.5 2485.0,264.5 2485.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-13\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2485.0,354.0 L2493.0,342.0 2477.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-14\" stroke-width=\"2px\" d=\"M2695,352.0 C2695,177.0 3015.0,177.0 3015.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-14\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2695,354.0 L2687,342.0 2703,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-15\" stroke-width=\"2px\" d=\"M2695,352.0 C2695,264.5 2835.0,264.5 2835.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-15\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2835.0,354.0 L2843.0,342.0 2827.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-16\" stroke-width=\"2px\" d=\"M3045,352.0 C3045,264.5 3185.0,264.5 3185.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-16\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3185.0,354.0 L3193.0,342.0 3177.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-17\" stroke-width=\"2px\" d=\"M3045,352.0 C3045,177.0 3365.0,177.0 3365.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-17\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">mark</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3365.0,354.0 L3373.0,342.0 3357.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-4852ccd5be2145a0bd0785136df77710-0-18\" stroke-width=\"2px\" d=\"M3395,352.0 C3395,264.5 3535.0,264.5 3535.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-4852ccd5be2145a0bd0785136df77710-0-18\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">fixed</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3535.0,354.0 L3543.0,342.0 3527.0,342.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "displacy.render(doc, style='dep', jupyter=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ウクライナ\n",
      "ロシア\n",
      "間\n",
      "戦争\n",
      "この影響\n",
      "世界中\n",
      "物価\n"
     ]
    }
   ],
   "source": [
    "for noun in doc.noun_chunks:\n",
    "    print(noun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ウクライナ 名詞\n",
      "ロシア 名詞\n",
      "間 名詞\n",
      "戦争 名詞\n",
      "始まる 動詞\n",
      "影響 名詞\n",
      "世界中 名詞\n",
      "物価 名詞\n",
      "上昇 名詞\n",
      "する 動詞\n",
      "いる 動詞\n"
     ]
    }
   ],
   "source": [
    "pos = ['名詞', '形容詞', '動詞']\n",
    "for sent in doc.sents:\n",
    "    for token in sent:\n",
    "        i = (token.tag_.split('-'))[0]\n",
    "        if i in pos:\n",
    "            print(token.lemma_, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自作関数のモジュール化\n",
    "\n",
    "テキストを分析する際、文章を解析するコードをその都度入力するのは手間です。そこで、関数として定義してしまいます。また、この関数定義を別ファイルとして保存して、これをモジュールとして読み込む方法を紹介します。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['今日', '午後', '八宝菜', '食べる']\n"
     ]
    }
   ],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "\n",
    "##  my_ginza.py\n",
    "import spacy\n",
    "\n",
    "# nlp = spacy.load('ja_ginza_electra')\n",
    "nlp = spacy.load('ja_ginza')\n",
    "word_list = []\n",
    "\n",
    "def tokens(sentences, pos=['名詞','形容詞','動詞'], stopwords_list=[]):\n",
    "    doc = nlp(sentences)\n",
    "    for sent in doc.sents:\n",
    "        for token in sent:\n",
    "            i = (token.tag_.split('-'))[0]\n",
    "            if i in pos:\n",
    "                word_list.append(token.lemma_)\n",
    "    return word_list\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    out = tokens('今日の午後は八宝菜を食べました。')\n",
    "    print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ランチ', '食べる']\n"
     ]
    }
   ],
   "source": [
    "import my_ginza\n",
    "out = my_ginza.tokens('ランチを食べました。')\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GiNZAの `nlp()` では多様な処理が連続的に行われます。これを pipeline と表現しています。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('tok2vec', <spacy.pipeline.tok2vec.Tok2Vec object at 0x7f32a4db8c40>)\n",
      "('parser', <spacy.pipeline.dep_parser.DependencyParser object at 0x7f32a4db2f80>)\n",
      "('ner', <spacy.pipeline.ner.EntityRecognizer object at 0x7f32a4db2ff0>)\n",
      "('morphologizer', <spacy.pipeline.morphologizer.Morphologizer object at 0x7f32a4db96c0>)\n",
      "('compound_splitter', <ginza.compound_splitter.CompoundSplitter object at 0x7f32a4d01750>)\n",
      "('bunsetu_recognizer', <ginza.bunsetu_recognizer.BunsetuRecognizer object at 0x7f32a4df7c40>)\n"
     ]
    }
   ],
   "source": [
    "for p in nlp.pipeline:\n",
    "    print(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "徳島\n",
      "は\n",
      "阿波踊り\n",
      "が\n",
      "有名\n",
      "です\n",
      "。\n",
      "高知\n",
      "は\n",
      "よさこい祭り\n",
      "が\n",
      "よく\n",
      "知ら\n",
      "れ\n",
      "て\n",
      "い\n",
      "ます\n",
      "。\n"
     ]
    }
   ],
   "source": [
    "texts = list(nlp.pipe(['徳島は阿波踊りが有名です。', '高知はよさこい祭りがよく知られています。']))\n",
    "for sents in texts:\n",
    "    for sent in sents.sents:\n",
    "        for comp in sent.sents:\n",
    "            for token in comp:\n",
    "                print(token)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "固有表現抽出は以下のような機能です。`doc` オブジェクトの `ents` (entities) リストの要素を取り出します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "徳島 City\n",
      "阿波踊り Person\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('徳島は阿波踊りが有名です。')\n",
    "for ent in doc.ents:\n",
    "    print(ent.text,  ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    徳島\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">City</span>\n",
       "</mark>\n",
       "は\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    阿波踊り\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">Person</span>\n",
       "</mark>\n",
       "が有名です。</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'ja_ginza' では、文章のベクトルと、単語ごとのベクトルがすでに取り出されています。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文章ベクトル\n",
      "----------\n",
      "(300,)\n",
      "(300,)\n",
      "単語ベクトル\n",
      "--------------\n",
      "徳島 True (300,)\n",
      "は True (300,)\n",
      "阿波踊り True (300,)\n",
      "が True (300,)\n",
      "有名 True (300,)\n",
      "です True (300,)\n",
      "。 True (300,)\n",
      "高知 True (300,)\n",
      "は True (300,)\n",
      "よさこい祭り True (300,)\n",
      "が True (300,)\n",
      "よく False (300,)\n",
      "知ら False (300,)\n",
      "れ False (300,)\n",
      "て True (300,)\n",
      "い True (300,)\n",
      "ます True (300,)\n",
      "。 True (300,)\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('徳島は阿波踊りが有名です。高知はよさこい祭りがよく知られています。')\n",
    "print('文章ベクトル\\n----------')\n",
    "for sent in doc.sents:\n",
    "        print(sent.vector.shape)\n",
    "print('単語ベクトル\\n--------------')\n",
    "for token in doc:\n",
    "    print(token.text, token.has_vector, token.vector.shape)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "それぞれが 300 次元のベクトルとして表現されていることが分かります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "徳島は阿波踊りが有名です。\n",
      "類似度は 0.8365065455436707\n"
     ]
    }
   ],
   "source": [
    "docs_l = list(doc.sents)\n",
    "print(docs_l[0])\n",
    "print(f'類似度は {docs_l[0].similarity(docs_l[1])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2つの文の類似度は約 0.83 と求められました。類似度は0から1の範囲の数値ですから、類似度は高いと言えます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `ja_ginza_electra` におけるベクトル表現\n",
    "\n",
    "\n",
    "以下のように実行すると、その一部を確認できます。ただし ja_ginza ではなく ja_giza_electra をインストールしている環境で実行できます。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "## メモリ節約のため ja_ginza オブジェクトは削除しておく\n",
    "del nlp\n",
    "\n",
    "nlp_e = spacy.load('ja_ginza_electra')\n",
    "doc_e = nlp_e('東京タワーで待ちあわせ。')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TransformerData(wordpieces=WordpieceBatch(strings=[['[CLS]', '東京', 'タ', '##ワー', 'で', '待ち', '##あ', '##わせ', '。', '[SEP]']], input_ids=array([[    2, 20375,   390,  7694, 20118, 22454,  3160,  7792, 20110,\n",
      "            3]], dtype=int32), attention_mask=array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], dtype=float32), lengths=[10], token_type_ids=array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)), model_output=ModelOutput([('last_hidden_state', array([[[-0.4881749 , -0.21651527,  0.54330087, ..., -0.10441031,\n",
      "         -0.27446336,  0.17393604],\n",
      "        [ 0.31530297,  0.08611556, -0.46102542, ...,  0.06445421,\n",
      "         -0.12429437,  0.07865894],\n",
      "        [-0.0466472 ,  0.23163442,  0.23410955, ...,  1.1375564 ,\n",
      "          0.1263227 ,  0.27445352],\n",
      "        ...,\n",
      "        [ 0.12068383,  0.8711264 , -0.41709605, ..., -0.4842382 ,\n",
      "         -0.49886122, -0.24800676],\n",
      "        [-0.4594799 , -0.01277845, -0.20572555, ..., -0.7351168 ,\n",
      "          0.12277819,  0.19968231],\n",
      "        [-1.2453946 , -0.2392647 ,  0.51350623, ...,  0.26412916,\n",
      "          0.10113095,  0.46275958]]], dtype=float32))]), align=Ragged(data=array([[1],\n",
      "       [2],\n",
      "       [3],\n",
      "       [4],\n",
      "       [5],\n",
      "       [6],\n",
      "       [7],\n",
      "       [8]], dtype=int32), lengths=array([1, 2, 1, 3, 1], dtype=int32), data_shape=(-1,), cumsums=None))\n"
     ]
    }
   ],
   "source": [
    "print(doc_e._.trf_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "トークンを確認してみます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': tensor([[    2, 20375,   390,  7694, 20118, 22454,  3160,  7792, 20110,     3]],\n",
      "       dtype=torch.int32), 'attention_mask': tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]), 'input_texts': [['[CLS]', '東京', 'タ', '##ワー', 'で', '待ち', '##あ', '##わせ', '。', '[SEP]']], 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=torch.int32)}\n"
     ]
    }
   ],
   "source": [
    "print(doc_e._.trf_data.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ここで「東京タワー」が '東京', 'タ', '##ワー'の3つに分解されているのが確認できます。文章は **サブワード** という単位で分割されます。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 10, 768)\n"
     ]
    }
   ],
   "source": [
    "tensors = doc_e._.trf_data.tensors\n",
    "print(tensors[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 \t 東京\n"
     ]
    }
   ],
   "source": [
    "for sent in doc_e.sents:\n",
    "    for token in sent:\n",
    "        if token.lemma_ == '東京':\n",
    "            print(token.i,\n",
    "            '\\t', token.lemma_\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "東京\n"
     ]
    }
   ],
   "source": [
    "print(doc_e[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 10, 768)\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "\n",
    "## '東京'のインデックス\n",
    "idx = 0\n",
    "token_idxes = list(itertools.chain.from_iterable(doc_e._.trf_data.align[ idx ].data))\n",
    "tensors = doc_e._.trf_data.tensors[0] \n",
    "print(tensors.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WordpieceBatch(strings=[['[CLS]', '東京', 'タ', '##ワー', 'で', '待ち', '##あ', '##わせ', '。', '[SEP]']], input_ids=array([[    2, 20375,   390,  7694, 20118, 22454,  3160,  7792, 20110,\n",
      "            3]], dtype=int32), attention_mask=array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], dtype=float32), lengths=[10], token_type_ids=array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32))\n"
     ]
    }
   ],
   "source": [
    "print(doc_e._.trf_data.wordpieces)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy, GiNZA, transformers を使った自然言語処理、あるいはテキスト処理では、ディープラーニングという技術がベースになっています。\n",
    "ディープラーニングについては、本書の最後の章で簡単に説明しています。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "name": "Chapter06_intro_ginza.ipynb"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}