notebook_multiple_services_moc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f9567c6f",
   "metadata": {},
   "source": [
    "# How to plot one large MOC (union) per servicee\n",
    "\n",
    "In this Notebook, we will see how to get every MOCs of one target and group them by service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f12cb543",
   "metadata": {},
   "source": [
    "## Import required modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b189372e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "from mocpy import MOC\n",
    "from ipyaladin import Aladin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60309946",
   "metadata": {},
   "source": [
    "## Connect to the ElasticSearch server\n",
    "\n",
    "You can change the following url to your ElasticSearch server, and provide a password or a certificate file if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2204873e",
   "metadata": {},
   "outputs": [],
   "source": [
    "es = Elasticsearch('http://voparis-elasticsearch.obspm.fr:9200')\n",
    "indexName = 'moc-index'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974de845",
   "metadata": {},
   "source": [
    "## Get the list of services\n",
    "\n",
    "The following query is used to search all the services with MOC data about Mars.\n",
    "We use an exists query to check if there is MOC data (a coverage field), and a match query to get only those about Mars.\n",
    "There is also an aggregation to group the services together.\n",
    "\n",
    "Here, you can notice that the size parameter is equals to 0 in the search function call. It's because we aren't looking at the hits (the items that match), but the aggregation: the list of services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4dadd463",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mars_craters_dev\n",
      "CRISM\n",
      "hst_planeto\n",
      "hrsc3nd\n",
      "Planmap\n",
      "vvex\n"
     ]
    }
   ],
   "source": [
    "query = {\n",
    "    \"bool\": {\n",
    "         \"must\": [\n",
    "             {\n",
    "                \"exists\" : {\n",
    "                    \"field\" : \"coverage\"\n",
    "                }\n",
    "             },\n",
    "             {\n",
    "                 \"match\" : {\n",
    "                    \"target_name\" : \"mars\"\n",
    "                 }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "aggs = {\n",
    "    \"agg-service\" : {\n",
    "        \"terms\": {\n",
    "            \"field\" : \"service_title.keyword\",\n",
    "            \"size\": 100\n",
    "        },\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "page = es.search(\n",
    "    index=indexName,\n",
    "    query=query,\n",
    "    size=0,\n",
    "    fields=[\"*\"],\n",
    "    aggs = aggs\n",
    ")\n",
    "\n",
    "services = []\n",
    "\n",
    "for bucket in page[\"aggregations\"][\"agg-service\"][\"buckets\"] :\n",
    "    print(bucket[\"key\"])\n",
    "    services.append(bucket[\"key\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f751e6d",
   "metadata": {},
   "source": [
    "## For each service, merge all of its MOCs into one for the target\n",
    "\n",
    "Like the other Notebooks of this repository, we use MOCPy to manipulate the MOCs: to create them from a text in ASCII and to merge them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f799b556",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mergeMocs(mocsList) :\n",
    "    mocUnion = MOC.new_empty(29)\n",
    "    for item in mocsList :\n",
    "        coverageField = item[\"fields\"][\"coverage\"]\n",
    "        if type(coverageField) == type([]) :\n",
    "            for coverage in coverageField :\n",
    "                moc = MOC.from_str(coverage)\n",
    "                mocUnion = mocUnion.union(moc)\n",
    "        else :\n",
    "            moc = MOC.from_str(coverageField)\n",
    "            mocUnion = mocUnion.union(moc)\n",
    "    return mocUnion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba4d77e",
   "metadata": {},
   "source": [
    "For each service we use a query to get all the MOCs on Mars, with this service.\n",
    "It's very similar to the query above, but here we use the scroll API since we have much too much data returned.\n",
    "So, we process it by breaking it into smaller parts, whose size is equal to the size variable defined at the next line.\n",
    "\n",
    "The scroll parameter in the search function is used to define of much time we need to keep the search context (here 1 minute), so the current batch of documents processed, whose size is in the size variable, will be kept for 1 minute and then overwritten by the next batch.\n",
    "\n",
    "scroll_id is associated to the current batch of documents read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "90f683d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Merging MOCs of  Mars_craters_dev ...\n",
      "0.00 %\n",
      "0.52 %\n",
      "1.04 %\n",
      "1.56 %\n",
      "2.08 %\n",
      "2.60 %\n",
      "3.12 %\n",
      "3.64 %\n",
      "4.16 %\n",
      "4.68 %\n",
      "5.20 %\n",
      "5.72 %\n",
      "6.24 %\n",
      "6.76 %\n",
      "7.28 %\n",
      "7.80 %\n",
      "8.32 %\n",
      "8.84 %\n",
      "9.36 %\n",
      "9.88 %\n",
      "10.40 %\n",
      "10.92 %\n",
      "11.44 %\n",
      "11.96 %\n",
      "12.48 %\n",
      "13.00 %\n",
      "13.52 %\n",
      "14.04 %\n",
      "14.56 %\n",
      "15.08 %\n",
      "15.60 %\n",
      "16.12 %\n",
      "16.64 %\n",
      "17.16 %\n",
      "17.68 %\n",
      "18.20 %\n",
      "18.72 %\n",
      "19.24 %\n",
      "19.76 %\n",
      "20.28 %\n",
      "20.80 %\n",
      "21.32 %\n",
      "21.84 %\n",
      "22.36 %\n",
      "22.88 %\n",
      "23.40 %\n",
      "23.92 %\n",
      "24.44 %\n",
      "24.96 %\n",
      "25.48 %\n",
      "26.00 %\n",
      "26.52 %\n",
      "27.04 %\n",
      "27.56 %\n",
      "28.08 %\n",
      "28.60 %\n",
      "29.12 %\n",
      "29.64 %\n",
      "30.16 %\n",
      "30.68 %\n",
      "31.20 %\n",
      "31.72 %\n",
      "32.24 %\n",
      "32.76 %\n",
      "33.28 %\n",
      "33.80 %\n",
      "34.32 %\n",
      "34.84 %\n",
      "35.37 %\n",
      "35.89 %\n",
      "36.41 %\n",
      "36.93 %\n",
      "37.45 %\n",
      "37.97 %\n",
      "38.49 %\n",
      "39.01 %\n",
      "39.53 %\n",
      "40.05 %\n",
      "40.57 %\n",
      "41.09 %\n",
      "41.61 %\n",
      "42.13 %\n",
      "42.65 %\n",
      "43.17 %\n",
      "43.69 %\n",
      "44.21 %\n",
      "44.73 %\n",
      "45.25 %\n",
      "45.77 %\n",
      "46.29 %\n",
      "46.81 %\n",
      "47.33 %\n",
      "47.85 %\n",
      "48.37 %\n",
      "48.89 %\n",
      "49.41 %\n",
      "49.93 %\n",
      "50.45 %\n",
      "50.97 %\n",
      "51.49 %\n",
      "52.01 %\n",
      "52.53 %\n",
      "53.05 %\n",
      "53.57 %\n",
      "54.09 %\n",
      "54.61 %\n",
      "55.13 %\n",
      "55.65 %\n",
      "56.17 %\n",
      "56.69 %\n",
      "57.21 %\n",
      "57.73 %\n",
      "58.25 %\n",
      "58.77 %\n",
      "59.29 %\n",
      "59.81 %\n",
      "60.33 %\n",
      "60.85 %\n",
      "61.37 %\n",
      "61.89 %\n",
      "62.41 %\n",
      "62.93 %\n",
      "63.45 %\n",
      "63.97 %\n",
      "64.49 %\n",
      "65.01 %\n",
      "65.53 %\n",
      "66.05 %\n",
      "66.57 %\n",
      "67.09 %\n",
      "67.61 %\n",
      "68.13 %\n",
      "68.65 %\n",
      "69.17 %\n",
      "69.69 %\n",
      "70.21 %\n",
      "70.73 %\n",
      "71.25 %\n",
      "71.77 %\n",
      "72.29 %\n",
      "72.81 %\n",
      "73.33 %\n",
      "73.85 %\n",
      "74.37 %\n",
      "74.89 %\n",
      "75.41 %\n",
      "75.93 %\n",
      "76.45 %\n",
      "76.97 %\n",
      "77.49 %\n",
      "78.01 %\n",
      "78.53 %\n",
      "79.05 %\n",
      "79.57 %\n",
      "80.09 %\n",
      "80.61 %\n",
      "81.13 %\n",
      "81.65 %\n",
      "82.17 %\n",
      "82.69 %\n",
      "83.21 %\n",
      "83.73 %\n",
      "84.25 %\n",
      "84.77 %\n",
      "85.29 %\n",
      "85.81 %\n",
      "86.33 %\n",
      "86.85 %\n",
      "87.37 %\n",
      "87.89 %\n",
      "88.41 %\n",
      "88.93 %\n",
      "89.45 %\n",
      "89.97 %\n",
      "90.49 %\n",
      "91.01 %\n",
      "91.53 %\n",
      "92.05 %\n",
      "92.57 %\n",
      "93.09 %\n",
      "93.61 %\n",
      "94.13 %\n",
      "94.65 %\n",
      "95.17 %\n",
      "95.69 %\n",
      "96.21 %\n",
      "96.73 %\n",
      "97.25 %\n",
      "97.77 %\n",
      "98.29 %\n",
      "98.81 %\n",
      "99.33 %\n",
      "99.85 %\n",
      "100.00 %\n",
      "Merging MOCs of  CRISM ...\n",
      "0.00 %\n",
      "9.65 %\n",
      "19.30 %\n",
      "28.95 %\n",
      "38.61 %\n",
      "48.26 %\n",
      "57.91 %\n",
      "67.56 %\n",
      "77.21 %\n",
      "86.86 %\n",
      "96.52 %\n",
      "100.00 %\n",
      "Merging MOCs of  hst_planeto ...\n",
      "0.00 %\n",
      "32.07 %\n",
      "64.14 %\n",
      "96.22 %\n",
      "100.00 %\n",
      "Merging MOCs of  hrsc3nd ...\n",
      "0.00 %\n",
      "48.86 %\n",
      "97.73 %\n",
      "100.00 %\n",
      "Merging MOCs of  Planmap ...\n",
      "0.00 %\n",
      "100.00 %\n",
      "Merging MOCs of  vvex ...\n",
      "0.00 %\n",
      "100.00 %\n",
      "DONE\n"
     ]
    }
   ],
   "source": [
    "size = 2000\n",
    "\n",
    "data = []\n",
    "for service in services :\n",
    "    print(\"Merging MOCs of \",service,\"...\")\n",
    "    query = {\n",
    "        \"bool\": {\n",
    "             \"must\": [\n",
    "                 {\n",
    "                    \"exists\" : {\n",
    "                        \"field\" : \"coverage\"\n",
    "                    }\n",
    "                },\n",
    "                 {\n",
    "                     \"match\" : {\n",
    "                        \"target_name\" : \"mars\"\n",
    "                     }\n",
    "                },\n",
    "                {\n",
    "                    \"term\" : {\n",
    "                         \"service_title.keyword\" : service\n",
    "                     } \n",
    "                 } \n",
    "            ]\n",
    "        }\n",
    "    }\n",
    "\n",
    "    page = es.search(\n",
    "        index=indexName,\n",
    "        query=query,\n",
    "        size=size,\n",
    "        scroll=\"1m\",\n",
    "        fields=[\"coverage\"],\n",
    "    )\n",
    "    \n",
    "    sid = page['_scroll_id']\n",
    "    scroll_size = page['hits']['total'][\"value\"]\n",
    "    mocUnion = MOC.new_empty(29)\n",
    "\n",
    "    progress = 0\n",
    "    increment = (100*size)/page['hits']['total'][\"value\"]\n",
    "    while (scroll_size > 0):\n",
    "        print(\"%.2f %%\" % progress)\n",
    "        page = es.scroll(scroll_id = sid, scroll = '2m')\n",
    "        sid = page['_scroll_id']\n",
    "        scroll_size = len(page['hits']['hits'])\n",
    "        mocUnion = mocUnion.union(mergeMocs(page[\"hits\"][\"hits\"]))\n",
    "        progress = progress + increment\n",
    "        \n",
    "    data.append({\"service\":service,\"coverage\":mocUnion})\n",
    "    print(\"100.00 %\")\n",
    "\n",
    "print(\"DONE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "311a3a8a",
   "metadata": {},
   "source": [
    "We can now display those MOCs in Aladin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2d983f39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a110db4263194256b3a7ee241c09b537",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Aladin(coo_frame='body', options=['allow_full_zoomout', 'coo_frame', 'fov', 'full_screen', 'log', 'overlay_sur…"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aladin = Aladin(\n",
    "    coo_frame='body',\n",
    "    survey=\"http://voparis-srv-paris.obspm.fr/vo/planeto/hips/CDS_P_Mars_Viking-MDIM21-color/\"\n",
    ")\n",
    "aladin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "47acf0fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "for mocService in data :\n",
    "    if not mocService[\"coverage\"].empty() :\n",
    "        jsonMoc = mocService[\"coverage\"].serialize(format='json', optional_kw_dict=None, pre_v2=False)\n",
    "        aladin.add_moc_from_dict(jsonMoc, {'opacity' : 0.5, 'name' : mocService[\"service\"]})\n",
    "aladin.fov = aladin.fov+1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60d56e2d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}