MIBItracker_API_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>MIBItracker REST API Tutorial</h1>\n",
    "\n",
    "In this tutorial, we show how to obtain an authorization token for the MIBItracker REST API, and some basic requests that can be performed with it.\n",
    "\n",
    "This tutorial references the BACKEND_URL as the base URL for all requests, which is different from the URL of the frontend application. You can find it listed in the MIBItracker About page, which is available from the menu in the upper right corner under your username.\n",
    "\n",
    "In order to access some sample data to run the tutorials, you can create an account using the following frontend URL: https://mibi-share.ionpath.com. In which case, the backend URL will be: https://backend-dot-mibitracker-share.appspot.com.\n",
    "\n",
    "This notebook shows four different ways of interacting with the API, with the Python example containing the most detail, but as with any REST API you should be able to choose the language/method of your choice even if it is not included here.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "1. [API Reference](#API-Reference)\n",
    "2. [Postman](#Postman)\n",
    "3. [Python](#Python)\n",
    "4. [cURL](#cURL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API Reference\n",
    "\n",
    "A complete reference of all routes available including required and optional parameters can be found using the [Swagger Specification](https://swagger.io/specification/) at [https://[BACKEND_URL]/docs/](). In order to access that, you will first need to log in specifically to the backend with your MIBItracker username and password at the base [https://[BACKEND_URL]](), and *then* navigate to the [/docs/]() route. Note that only the routes which the currently logged-in user is authorized to access will be displayed. A screenshot of the docs with the available routes for tissues expanded is shown below:\n",
    "\n",
    "![API Screenshot](./images/swagger_tissues_endpoints.png)\n",
    "\n",
    "You may try out the GET routes directly in the docs. For example, see below for what the request and response is when listing all tissues. Note that most routes in the API return results in a paginated manner to limit large queries. Executing the GET request for `/images/`, for instance, will return the most recent 10 images:\n",
    "\n",
    "![Tissue Screenshot](./images/swagger_tissues_all.png)\n",
    "\n",
    "For more information on paginated results, see the [Python](#Python) section below.\n",
    "\n",
    "Filtering by `?organ=thymus` reduces the results to a single item:\n",
    "\n",
    "![Filtered Tissue Screenshot](./images/swagger_tissues_filtered.png)\n",
    "\n",
    "You may also query for all images that have a specific tissue type:\n",
    "\n",
    "![Images by Tissue Screenshot](./images/swagger_images_by_tissue.png)\n",
    "\n",
    "Expand out any of the other routes to see their details. Note that onlly the `GET` requests are available to try out directly from the docs; read on for how to make general requests using [Postman](#Postman), [Python](#Python), [cURL](#cURL), or the language of your choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Postman\n",
    "\n",
    "Postman is a GUI application for sending requests to a server, and can be useful to help visualize the way the requests are sent and responses handled. It can be downloaded [here](https://www.getpostman.com/). To begin, you will need to obtain an authorization token by sending a POST request with your email address and password. First, next to the URL bar, change the dropdown from GET to POST and enter https://[BACKEND_URL]/api-token-auth/ into the URL bar. Then, in the Headers tab, add a key: \"Content-Type\" and a value: \"application/json\" as shown in the screenshot below:\n",
    "\n",
    "![Postman Auth Header](./images/postman-auth-header.PNG)\n",
    "\n",
    "Then, in the Body tab add a key-value for your email and password, respectively:\n",
    "\n",
    "![Postman Auth Body](./images/postman-auth-body.PNG)\n",
    "\n",
    "Hit the blue Send button to send the request. When the response is returned, you should see a single key-value pair in the response body area. The value is a long string of characters and that is your authorization token:\n",
    "\n",
    "![Postman Auth Token](./images/postman-auth-token.PNG)\n",
    "\n",
    "To use this token, create a new request and this time, add a header with the key \"Authorization\" and the value \"JWT token_value\" where token_value is the string from above:\n",
    "\n",
    "![Postman Token](./images/postman-token.PNG)\n",
    "\n",
    "As an example, we will then get a specific image from the MIBItracker. The route for retrieving a specific image is `TRACKER_BACKEND_URL/images/[id]`, where 'id' is the primary key of the image. For example, to retrieve the image with id = 429, enter the formatted URL into the URL bar:\n",
    "\n",
    "![Postman Images Retrieve](./images/postman-images-retrieve.PNG)\n",
    "\n",
    "The result is a JSON object containing the properties of the image. As you can see from the first key, it has an id of 429, which is the id we used in the example URL above.\n",
    "\n",
    "For more advanced examples of using the API, such as to perform an advanced search of images, see the Python examples below. As shown in the [API Reference](#API-Reference) section above, visit [https://[INSTANCE_NAME].ionpath.com/tracker/about]() and click the \"API Documentation\" link to view a list of available routes for the API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python\n",
    "\n",
    "Requests to the API can be made in Python using the `MibiRequests` module from [mibilib](https://github.com/ionpath/mibilib), an open source library for interacting with the MIBItracker API. After downloading the library, make sure to add it to your PATH."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "from mibitracker.request_helpers import MibiRequests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step to using `requests_helpers` is to create an instance of `MibiRequests` and obtain an authorization token. Note that once a token has been acquired, it will only be valid for 24 hours and another token must be obtained afterwards.\n",
    "\n",
    "This code assumes your MIBItracker email and password are stored in environment variables. Please use care in properly securing these credentials.\n",
    "\n",
    "One way to do this is to store the credentials in a text file called `MIBItracker_login.dat` with the following content:\n",
    "```bash\n",
    "MIBITRACKER_PUBLIC_URL=https://backend-dot-mibitracker-share.appspot.com\n",
    "MIBITRACKER_PUBLIC_EMAIL=your@email.com\n",
    "MIBITRACKER_PUBLIC_PASSWORD=YourSecurePassword123!?@\n",
    "\n",
    "```\n",
    "\n",
    "Remember to restrict the access to the file. In `bash` this can be done with the following command:<br>\n",
    "`chmod 600 MIBItracker_login.dat`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load MIBItracker credentials from file.\n",
    "from dotenv import load_dotenv\n",
    "fname_login = '/path/to/MIBItracker_login.dat'\n",
    "load_dotenv(fname_login)\n",
    "\n",
    "# This assumes your MIBItracker credentials are saved as environment variables.\n",
    "email = os.getenv('MIBITRACKER_PUBLIC_EMAIL')\n",
    "password = os.getenv('MIBITRACKER_PUBLIC_PASSWORD')\n",
    "BACKEND_URL = os.getenv('MIBITRACKER_PUBLIC_URL')\n",
    "\n",
    "mr = MibiRequests(BACKEND_URL, email, password)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`MibiRequests` contains helper functions for common routes used when accessing the API. These can easily be called using an  authorized instance of `MibiRequests`.\n",
    "\n",
    "For example, to obtain an array of metadata for all images in a particular run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_name = '20180122_1242_1245'\n",
    "image_list = mr.run_images(run_name)\n",
    "\n",
    "print('{} images found from run {}: \\n'.format(len(image_list), run_name))\n",
    "for im in image_list:\n",
    "    print('\\t{}: {}'.format(im['folder'], im['point']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To upload an additional channel to an existing image, i.e. a segmentation mask, use the `MibiRequests.upload_channel` route:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_id = 164\n",
    "png_filename = 'segmentation_labels.png'\n",
    "mr.upload_channel(image_id, png_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all routes available in the MIBItracker API have specific helper functions implemented in the `MibiRequests` class. However, an authenticated `MibiRequests` instance can still be used to access the routes using the correct HTTP verb. The HTTP verbs listed on the documentation page correspond to `MibiRequests` methods.\n",
    "\n",
    "As shown in the [API Reference](#API-Reference) section above, visit [https://[BACKEND_URL/docs/]() to view a list of all available routes for the API. Note that the `mibilib` library is open source, and contributions for helper functions using the API are welcome.\n",
    "\n",
    "For example, to get a list of all images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_images = mr.get('/images/')\n",
    "images_response = response_images.json()\n",
    "print('All images: {} images have been returned.'.format(len(images_response['results'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that this doesn't return *all* the items, as this can get quite large. As a result, most API routes default to returning results in a paginated manner. For instance, if we inspect the response json from the previous query, we can see that it's a dictionary containing the paginated results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paginated dictionary contains four keys:\n",
    "- <b>count</b>: The complete number of records matching the query, regardless of pagination.\n",
    "- <b>next</b>: A request URL that, if executed, will return the *next* page of results.\n",
    "- <b>previous</b>: A request URL that, if executed, will return the *previous* page of results.\n",
    "- <b>results</b>: The records matching the query for the current page of results.\n",
    "\n",
    "To paginate the results, use `limit` and `offset` parameters to specify how many records to return in a response and which index into the list to begin with. If `limit` and `offset` are not specified (as was the case in the `mr.get.('/images')` example above), `limit` and `offset` default to 10 and 0, respectively. To specify exact values of `limit` and `offset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_first_page = mr.get(\n",
    "    '/images/',\n",
    "    params={'limit': 30, 'offset': 0})\n",
    "images_first_page = images_first_page.json()\n",
    "print('First page: {}/{} images have been returned in the first page\\'s results.'.format(\n",
    "        len(images_first_page['results']), images_first_page['count']))\n",
    "\n",
    "images_second_page = mr.get(\n",
    "    '/images/',\n",
    "    params={'limit': 30, 'offset': 30})\n",
    "images_second_page = images_second_page.json()\n",
    "print('Second page: {}/{} images have been returned in the second page\\'s results.'.format(\n",
    "        len(images_second_page['results']), images_second_page['count']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar approach could be used to obtain a list of all images from the run `20171207_1060_1130` (note the use of a double underscore in the query parameter used here):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_name = '20171207_1060_1130'\n",
    "response_images = mr.get(\n",
    "    '/images/?run__name={}'.format(run_name))\n",
    "image_list = response_images.json()['results']\n",
    "\n",
    "print('{} images found from run {}: \\n'.format(len(image_list), run_name))\n",
    "for im in image_list:\n",
    "    print('\\t{}: {}'.format(im['folder'], im['point']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A route exists for advanced searching of images with additional fields available to filter the results, including a range for run date, antibody targets, and others. To use the advanced search for images containing `lamin` with the status `A` (for available):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_advanced = mr.get(\n",
    "    '/images/search_advanced/?antibodyTarget={target}&status={status}'.format(\n",
    "        target='Lamin', status='A'))\n",
    "advanced_list = response_advanced.json()\n",
    "\n",
    "print('{} images found containing \"Lamin\" that have status \"Available\"'.format(\n",
    "         advanced_list['count']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most routes allow to select a single item of that type knowing the primary key (id) of the item in the database. In this case, the id is part of the route and not specified as a query parameter or request data. For example, to get the properties of slide id 5: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "slide_id = 5\n",
    "single_slide = mr.get('/slides/{}/'.format(slide_id))\n",
    "\n",
    "# Print out result with some nice formatting\n",
    "print(json.dumps(single_slide.json(), indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also routes for details such as `/images/{id}/channelnames/` to return only the list of targets for a given image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "image_id = 17\n",
    "image_channelnames = mr.get('/images/{}/channelnames/'.format(image_id))\n",
    "\n",
    "print(json.dumps(image_channelnames.json(), indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "POST and PUT routes generally require data to be sent with the request. Some POST and PUT routes will require multiple pieces of data, such as POSTing a new image set. The data will need to be supplied as a JSON dictionary converted to a string, and the content type of the request will need to be set to JSON. In the example below, the images list is expecting the ID of the images to add to the new image set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\n",
    "    'name': 'New Image Set',\n",
    "    'description': 'This image set was created from the API',\n",
    "    'images': [10, 11, 12, 13]\n",
    "}\n",
    "headers_with_content_type = {\n",
    "    'content-type': 'application/json'\n",
    "}\n",
    "response_imageset = mr.post('/imagesets/',\n",
    "    data=json.dumps(data), \n",
    "    headers=headers_with_content_type)\n",
    "\n",
    "response_imageset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## cURL\n",
    "\n",
    "An alternative that works from the command line is to use cURL to access the API.\n",
    "\n",
    "Before using any of the routes available in the API, an authorization token must be obtained. Note that once a token has been acquired, it will only be valid for 24 hours and another token must be obtained afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# Assuming $TRACKER_BACKEND_URL, $TRACKER_EMAIL and $TRACKER_PASSWORD\n",
    "# have already been set.\n",
    "data='{\"email\": \"'\"$TRACKER_EMAIL\"'\", \"password\": \"'\"$TRACKER_PASSWORD\"'\"}'\n",
    "\n",
    "curl --request POST --header \"Content-Type: application/json\" \\\n",
    "  --data \"$data\" $TRACKER_BACKEND_URL/api-token-auth/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view a list of available routes for the API, visit [https://[INSTANCE_NAME].ionpath.com/tracker/about]() and click the \"API Documentation\" link.\n",
    "\n",
    "For example, to get a list of all tissues, assuming you've stored the authorization token as `$TOKEN`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "curl --header \"Content-Type: application/json\" \\\n",
    "  --header \"Authorization: JWT $TOKEN\" $TRACKER_BACKEND_URL/tissues/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more advanced examples of using the API, such as to perform an advanced search of images, see the [Python](#Python) examples above."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}