Anatomy_of_an_ASDF_file.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This is intended for those that are only interested in some of the details. It is not a full description, but should be treated as a basic introduction. There are more advanced aspects that will be addressed in other tutorials.\n",
    "\n",
    "### Contrasting with FITS\n",
    "\n",
    "As a refresher, FITS files consist of a list of one or more HDU units (Header Data Units). Each header consists of a multiple of 36 80 byte ASCII records (i.e., each header has a size that must be a multiple of 2880 bytes). Most of these 80 byte records (a remnant of the old punched cards size) consist of a keyword and a value, with an optional comment. Keywords are limited to 8 characters, the values may have type Boolean, integer, float or string and may be as long as as there are spaces left on the record. Comments can take up the remaining unused character spots.\n",
    "\n",
    "The data unit must also be of a size that is a multiple of 2880 bytes but the data itself doesn't have to be that size. Data may be an array, table, ascii table and some other more specialized types for special fields (e.g., Radio Astronomy). What type it is and the type of the numbers and elements contained within are specified in the associated header.\n",
    "\n",
    "### ASDF structure\n",
    "\n",
    "ASDF files have only one header, which is pure YAML (Yaml Ain't Markup Language) followed by 0 or more binary blocks.\n",
    "\n",
    "Yaml is a descriptive text format meant to be reasonably readable by people (as opposed to, say, XML). JSON, a common web interchange format is a subset of YAML. YAML is capable of representing complex organization of information by the means of two basic structures (Python terminology will be used here although this will mention the corresponding YAML terms once), these being dictionaries (maps in YAML) and lists (arrays in YAML). YAML is a bit schizophenic in that it has two different syntaxes for representing such structure; one is Python like and uses indentation, the other is more C like and uses grouping delimiters such as `{` an `}`. Both can be used within the same file.\n",
    "\n",
    "Advantages of YAML over FITS headers:\n",
    "\n",
    "- equivalent of keywords can be very long (between 128 and 1024 characters depending on the library)\n",
    "- values lengths are not restricted, and are not restricted to be simple values, values may be dictionaries or lists\n",
    "- tag feature allows libraries to handle specific contents in a custom way (more later)\n",
    "- references\n",
    "- and that implicit grouping is implied by the use of dictionaries and lists, something FITS has no equivalent mechanism other then defining special conventions for specific data sets.\n",
    "\n",
    "The ASDF standard adds specific extra features to YAML capabilities and the ability to include binary data  that will be summarized later.\n",
    "\n",
    "### Examples\n",
    "\n",
    "First an example without binary data, viewed as a simple text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import asdf\n",
    "from astropy.utils.data import download_file\n",
    "asdf_file_url = 'https://data.science.stsci.edu/redirect/ASDF/asdf_tut_simple_1.asdf'\n",
    "filepath = download_file(asdf_file_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(filepath) as asdffile: print(asdffile.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lines appearing before the line starting with `---` are used to indicate that it is an ASDF file, the versions of the appropriate components, and lastly an indication of where tags are defined. We will ignore tags for now. The actual yaml starts with the `---` and the end of the yaml is indicated by the `...`, which in this case is the end of the file. The yaml starts with the indication of the libraries involved and the homepage (the `asdf_library and history` items). The user content starts with the `target` entry. While this is a very simple file, it is apparent that the first `name` key that appears is associated with `target` and that the second `name` key that appears is associated with `proposer` by dint of the indentation and proximity.\n",
    "\n",
    "Now to access this file in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af = asdf.open(filepath)\n",
    "af.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `info` method on the opened ASDF file basically shows the same information as the raw text does along with the type of the values. Accessing particular items is simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af['target']['frame']['name']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af['co-Is'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add a couple small data arrays and write out to a new file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "af['data1'] = np.ones((10,10))\n",
    "af['data2'] = np.zeros((10,))\n",
    "af.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af.write_to('tut_with_data.asdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open as an ordinary file first\n",
    "with open('tut_with_data.asdf','rb') as taf2:\n",
    "    print(taf2.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print as string, but requires conversion of text part\n",
    "with open('tut_with_data.asdf','rb') as taf2:\n",
    "    fbytes = taf2.read()\n",
    "    end = fbytes.find(b'...')\n",
    "    print(fbytes[: end + 3].decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tags\n",
    "\n",
    "Tags are the mechanism asdf uses to indicate what follows is to be interpreted a special way, and in Python this usually means it should be converted to a specific Python object. There is\n",
    "only one kind of tag used in this example (besides the core/asdf-1.1.0, core/software-1.0.0 \n",
    "and core/extension_metadata-1.0.0 tags used in the previous example) and it is to indicate\n",
    "that the following items are to be interpreted as an array and thus converted to a numpy\n",
    "array when reading. This particular tag is !core/ndarray-1.0.0 that the Python library\n",
    "uses to identify numpy arrays. note what follows the tag indicates the type, shape and \n",
    "byte order of the array, as well as which binary block to find the binary data (other\n",
    "attributes are possible in more complex cases). When reading this file, this information\n",
    "becomes part of the resulting numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now open as an asdf file\n",
    "af2 = asdf.open('tut_with_data.asdf')\n",
    "af2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2['data1'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2['data2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Growing the asdf structure\n",
    "\n",
    "Once created, the asdf structure can be dynamically extended in arbitrary ways. The following will exhibit the flexibility available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a new dict node\n",
    "af2['misc'] = {'my_very_lucky_numbers': [3, 7]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2.info(max_rows=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2['misc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify that node in the asdf tree\n",
    "af2['misc']['my_very_lucky_numbers'] += [13, 27]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2['misc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add more items to misc\n",
    "import astropy.units as u\n",
    "cornbread = {\n",
    "    'flour': 0.25 * u.l,\n",
    "    'yellow cornmeal': 0.25 * u.l,\n",
    "    'white sugar': 1/6 * u.l,\n",
    "    'salt': 5 * u.ml,\n",
    "    'egg': 1,\n",
    "    'milk': 0.25 * u.l,\n",
    "    'vegetable oil': 0.25 * u.l\n",
    "}\n",
    "af2['misc']['ingredients_for_cornbread'] = cornbread\n",
    "af2['misc']['speech'] = '''\n",
    "Four score and seven years ago our fathers brought forth on this continent,\n",
    "a new nation, conceived in Liberty,\n",
    "and dedicated to the proposition that all men are created equal.\n",
    "\n",
    "Now we are engaged in a great civil war,\n",
    "testing whether that nation, or any nation so conceived and so dedicated,\n",
    "can long endure. We are met on a great battle-field of that war.\n",
    "We have come to dedicate a portion of that field, as a final resting\n",
    "place for those who here gave their lives that that nation might live.\n",
    "It is altogether fitting and proper that we should do this.\n",
    "\n",
    "But, in a larger sense, we can not dedicate -- we can not consecrate\n",
    "-- we can not hallow -- this ground. The brave men, living and dead,\n",
    "who struggled here, have consecrated it, far above our poor power to\n",
    "add or detract. The world will little note, nor long remember what we\n",
    "say here, but it can never forget what they did here.\n",
    "It is for us the living, rather, to be dedicated here to the\n",
    "unfinished work which they who fought here have thus far so nobly advanced.\n",
    "It is rather for us to be here dedicated to the great task remaining before us\n",
    "-- that from these honored dead we take increased devotion to that cause for\n",
    "which they gave the last full measure of devotion -- \n",
    "that we here highly resolve that these dead shall not have died in vain --\n",
    "that this nation, under God, shall have a new birth of freedom --\n",
    "and that government of the people, by the people, for the people,\n",
    "shall not perish from the earth.\n",
    "'''\n",
    "# Delete a node\n",
    "del af2.tree['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af2['misc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(af2['misc']['speech'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "This provides a way of sharing the same item among different attributes or lists in the file without duplicating the information. Within the python library this is handled automatically if you assign exactly the same object to two different attributes or list items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Illustrate with a small array\n",
    "refdata = np.array([1, 1, 2, 3, 5, 8, 13, 21])\n",
    "af2['ref1'] = refdata\n",
    "af2['misc']['ref2'] = refdata\n",
    "# change one of these\n",
    "af2['ref1'][1] = -100\n",
    "# print the other\n",
    "af2['misc']['ref2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to a new file\n",
    "af2.write_to('tut_with_data2.asdf')\n",
    "# Read back in\n",
    "af3 = asdf.open('tut_with_data2.asdf',mode='rw')\n",
    "af3['ref1']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af3['misc']['ref2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af3['misc']['ref2'][0] = 333\n",
    "af3['ref1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that after saving and reading back in, these two attributes still share the same object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look at the corresponding raw contents of the last file'\n",
    "with open('tut_with_data2.asdf','rb') as taf2: print(taf2.read()[:3204].decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way references are handled is to identify the first occurance with a \n",
    "special identifier (here &id001) that later can be referred to using the syntax *id001,\n",
    "which indicate that attribute or list item is to use exactly the same object. The same\n",
    "reference may be used any number of times.\n",
    "\n",
    "A practical example of utilizing references is to use the same mask or data quality array\n",
    "for all the arrays associated with the data such as error arrays, net integration time,\n",
    "etc.\n",
    "\n",
    "Also note the appearance of the tags to indicate items that are astropy quantities\n",
    "(i.e., values with units, !unit/quantity-1.1.0), and to identify the units themselves \n",
    "(e.g., !unit/unit-1.0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, all arrays can be saved as text instead using the \"inline\" option (or individually designated as inline)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "af3.write_to('tut_with_inline_data.asdf', all_array_storage='inline')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read as text\n",
    "with open('tut_with_inline_data.asdf','rb') as tafinline: print(tafinline.read().decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Using the original example file, set the proposer's name to Zaphod Beeblebrox and the proposer's institution to HHGTTG, and then save the ASDF under a different name, read it back in and verify the new name and institution using info."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "\n",
    "Using the file saved for Exercise 1, add a third data array with the attribute \n",
    "name of data3 that is 3 a dimensional\n",
    "boolean array with dimensions (10, 10, 10).\n",
    "Delete data2 and save the under a different name. Open the saved file and verify\n",
    "it contains the new items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}