RumbleSandbox.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Rumble sandbox</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a Rumble sandbox that allows you to play with simple JSONiq queries.\n",
    "\n",
    "It is a jupyter notebook that you can also download and execute on your own machine, but if you arrived here from the Rumble website, it is likely to be shown within Google's Colab environment.\n",
    "\n",
    "To get started, you first need to execute the cell below to activate the Rumble magic (you do not need to understand what it does, this is just initialization Python code)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import time\n",
    "from IPython.core.magic import register_line_cell_magic\n",
    "\n",
    "@register_line_cell_magic\n",
    "def rumble(line, cell=None):\n",
    "    if cell is None:\n",
    "        data = line\n",
    "    else:\n",
    "        data = cell\n",
    "\n",
    "    start = time.time()                                                         \n",
    "    response = json.loads(requests.post(server, data=data).text)                   \n",
    "    end = time.time()                                                              \n",
    "    print(\"Took: %s ms\" % (end - start))\n",
    "\n",
    "    if 'warning' in response:\n",
    "        print(json.dumps(response['warning']))\n",
    "    if 'values' in response:\n",
    "        for e in response['values']:\n",
    "            print(json.dumps(e))\n",
    "    elif 'error-message' in response:\n",
    "        return response['error-message']\n",
    "    else:\n",
    "        return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, this notebook uses a small public backend provided by us (very limited in CPU and memory, and with only the http scheme activated) that is sufficient to discover Rumble. This is new and experimental, so that it may occasionally break, especially if too many users use it at the same time, so please bear with us!\n",
    "\n",
    "In order to use our backend, just execute the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "server='http://public.rumbledb.org:9090/jsoniq'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is straightforward to execute your own Rumble server on your own Spark cluster (and then you can make full use of all the input file systems and file formats). In this case, just replace the above server with your own hostname and port.\n",
    "\n",
    "Now we are all set! You can now start reading and executing the JSONiq queries as you go, and you can even edit them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON\n",
    "\n",
    "As explained on the [official JSON Web site](http://www.json.org/), JSON is a lightweight data-interchange format designed for humans as well as for computers. It supports as values:\n",
    "- objects (string-to-value maps)\n",
    "- arrays (ordered sequences of values)\n",
    "- strings\n",
    "- numbers\n",
    "- booleans (true, false)\n",
    "- null\n",
    "\n",
    "JSONiq provides declarative querying and updating capabilities on JSON data.\n",
    "\n",
    "## Elevator Pitch\n",
    "\n",
    "JSONiq is based on XQuery, which is a W3C standard (like XML and HTML). XQuery is a very powerful declarative language that originally manipulates XML data, but it turns out that it is also a very good fit for manipulating JSON natively.\n",
    "JSONiq, since it extends XQuery, is a very powerful general-purpose declarative programming language. Our experience is that, for the same task, you will probably write about 80% less code compared to imperative languages like JavaScript, Python or Ruby. Additionally, you get the benefits of strong type checking without actually having to write type declarations.\n",
    "Here is an appetizer before we start the tutorial from scratch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.011076211929321289 ms\n",
      "[{\"nb\": 1, \"state\": \"MA\", \"sold\": \"broiler\"}, {\"nb\": 1, \"state\": \"MA\", \"sold\": \"socks\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"toaster\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"toaster\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"socks\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"toaster\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"blender\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"blender\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"shirt\"}]\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "\n",
    "let $stores :=\n",
    "[\n",
    "  { \"store number\" : 1, \"state\" : \"MA\" },\n",
    "  { \"store number\" : 2, \"state\" : \"MA\" },\n",
    "  { \"store number\" : 3, \"state\" : \"CA\" },\n",
    "  { \"store number\" : 4, \"state\" : \"CA\" }\n",
    "]\n",
    "let $sales := [\n",
    "   { \"product\" : \"broiler\", \"store number\" : 1, \"quantity\" : 20  },\n",
    "   { \"product\" : \"toaster\", \"store number\" : 2, \"quantity\" : 100 },\n",
    "   { \"product\" : \"toaster\", \"store number\" : 2, \"quantity\" : 50 },\n",
    "   { \"product\" : \"toaster\", \"store number\" : 3, \"quantity\" : 50 },\n",
    "   { \"product\" : \"blender\", \"store number\" : 3, \"quantity\" : 100 },\n",
    "   { \"product\" : \"blender\", \"store number\" : 3, \"quantity\" : 150 },\n",
    "   { \"product\" : \"socks\", \"store number\" : 1, \"quantity\" : 500 },\n",
    "   { \"product\" : \"socks\", \"store number\" : 2, \"quantity\" : 10 },\n",
    "   { \"product\" : \"shirt\", \"store number\" : 3, \"quantity\" : 10 }\n",
    "]\n",
    "let $join :=\n",
    "  for $store in $stores[], $sale in $sales[]\n",
    "  where $store.\"store number\" = $sale.\"store number\"\n",
    "  return {\n",
    "    \"nb\" : $store.\"store number\",\n",
    "    \"state\" : $store.state,\n",
    "    \"sold\" : $sale.product\n",
    "  }\n",
    "return [$join]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And here you go\n",
    "\n",
    "### Actually, you already knew some JSONiq\n",
    "\n",
    "The first thing you need to know is that a well-formed JSON document is a JSONiq expression as well.\n",
    "This means that you can copy-and-paste any JSON document into a query. The following are JSONiq queries that are \"idempotent\" (they just output themselves):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006221771240234375 ms\n",
      "{\"pi\": 3.14, \"sq2\": 1.4}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "{ \"pi\" : 3.14, \"sq2\" : 1.4 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0064008235931396484 ms\n",
      "[2, 3, 5, 7, 11, 13]\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "[ 2, 3, 5, 7, 11, 13 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.008172035217285156 ms\n",
      "{\"operations\": [{\"binary\": [\"and\", \"or\"]}, {\"unary\": [\"not\"]}], \"bits\": [0, 1]}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "{\n",
    "      \"operations\" : [\n",
    "        { \"binary\" : [ \"and\", \"or\"] },\n",
    "        { \"unary\" : [\"not\"] }\n",
    "      ],\n",
    "      \"bits\" : [\n",
    "        0, 1\n",
    "      ]\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006453990936279297 ms\n",
      "[{\"Question\": \"Ultimate\"}, [\"Life\", \"the universe\", \"and everything\"]]\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "[ { \"Question\" : \"Ultimate\" }, [\"Life\", \"the universe\", \"and everything\"] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works with objects, arrays (even nested), strings, numbers, booleans, null.\n",
    "\n",
    "It also works the other way round: if your query outputs an object or an array, you can use it as a JSON document.\n",
    "JSONiq is a declarative language. This means that you only need to say what you want - the compiler will take care of the how. \n",
    "\n",
    "In the above queries, you are basically saying: I want to output this JSON content, and here it is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSONiq basics\n",
    "\n",
    "### The real JSONiq Hello, World!\n",
    "\n",
    "Wondering what a hello world program looks like in JSONiq? Here it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0066568851470947266 ms\n",
      "\"Hello, World!\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "\"Hello, World!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not surprisingly, it outputs the string \"Hello, World!\".\n",
    "\n",
    "### Numbers and arithmetic operations\n",
    "\n",
    "Okay, so, now, you might be thinking: \"What is the use of this language if it just outputs what I put in?\" Of course, JSONiq can more than that. And still in a declarative way. Here is how it works with numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0072820186614990234 ms\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "2 + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0064237117767333984 ms\n",
      "42\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    " (38 + 2) div 2 + 11 * 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(mind the division operator which is the \"div\" keyword. The slash operator has different semantics).\n",
    "\n",
    "Like JSON, JSONiq works with decimals and doubles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.007363080978393555 ms\n",
      "25292400000000000000000000\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    " 6.022e23 * 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logical operations\n",
    "\n",
    "JSONiq supports boolean operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006527900695800781 ms\n",
      "false\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "true and false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.007046222686767578 ms\n",
      "true\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "(true or false) and (false or true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unary not is also available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 243.537109375,
      "end_time": 1557235032261.665
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006941080093383789 ms\n",
      "false\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "not true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strings\n",
    "\n",
    "JSONiq is capable of manipulating strings as well, using functions:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 21398.401123046875,
      "end_time": 1557212587338.105
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.005676984786987305 ms\n",
      "\"Hello Captain Kirk\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "concat(\"Hello \", \"Captain \", \"Kirk\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 752.509033203125,
      "end_time": 1557234001018.501
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.00574493408203125 ms\n",
      "\"Spock\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "substring(\"Mister Spock\", 8, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSONiq comes up with a rich string function library out of the box, inherited from its base language. These functions are listed [here](https://www.w3.org/TR/xpath-functions-30/) (actually, you will find many more for numbers, dates, etc).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 17337.39599609375,
      "end_time": 1557230774662.657
     }
    }
   },
   "source": [
    "\n",
    "### Sequences\n",
    "\n",
    "Until now, we have only been working with single values (an object, an array, a number, a string, a boolean). JSONiq supports sequences of values. You can build a sequence using commas:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0066449642181396484 ms\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    " (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 222241.63500976562,
      "end_time": 1557387411306.792
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.00654292106628418 ms\n",
      "1\n",
      "true\n",
      "42\n",
      "\"Life\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "1, true, 4.2e1, \"Life\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"to\" operator is very convenient, too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 39579.559814453125,
      "end_time": 1557387951481.842
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006345033645629883 ms\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n",
      "11\n",
      "12\n",
      "13\n",
      "14\n",
      "15\n",
      "16\n",
      "17\n",
      "18\n",
      "19\n",
      "20\n",
      "21\n",
      "22\n",
      "23\n",
      "24\n",
      "25\n",
      "26\n",
      "27\n",
      "28\n",
      "29\n",
      "30\n",
      "31\n",
      "32\n",
      "33\n",
      "34\n",
      "35\n",
      "36\n",
      "37\n",
      "38\n",
      "39\n",
      "40\n",
      "41\n",
      "42\n",
      "43\n",
      "44\n",
      "45\n",
      "46\n",
      "47\n",
      "48\n",
      "49\n",
      "50\n",
      "51\n",
      "52\n",
      "53\n",
      "54\n",
      "55\n",
      "56\n",
      "57\n",
      "58\n",
      "59\n",
      "60\n",
      "61\n",
      "62\n",
      "63\n",
      "64\n",
      "65\n",
      "66\n",
      "67\n",
      "68\n",
      "69\n",
      "70\n",
      "71\n",
      "72\n",
      "73\n",
      "74\n",
      "75\n",
      "76\n",
      "77\n",
      "78\n",
      "79\n",
      "80\n",
      "81\n",
      "82\n",
      "83\n",
      "84\n",
      "85\n",
      "86\n",
      "87\n",
      "88\n",
      "89\n",
      "90\n",
      "91\n",
      "92\n",
      "93\n",
      "94\n",
      "95\n",
      "96\n",
      "97\n",
      "98\n",
      "99\n",
      "100\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    " (1 to 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some functions even work on sequences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 27601.31787109375,
      "end_time": 1557171455208.987
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.005728006362915039 ms\n",
      "5050\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "sum(1 to 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 55527.44091796875,
      "end_time": 1557229508578.007
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0058438777923583984 ms\n",
      "\"These-are-some-words\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "string-join((\"These\", \"are\", \"some\", \"words\"), \"-\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 13343.739013671875,
      "end_time": 1557224627704.526
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0066111087799072266 ms\n",
      "11\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "count(10 to 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 19340.81201171875,
      "end_time": 1557220260002.04
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.005938053131103516 ms\n",
      "50.5\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "avg(1 to 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike arrays, sequences are flat. The sequence (3) is identical to the integer 3, and (1, (2, 3)) is identical to (1, 2, 3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_status": {
     "execute_time": {
      "duration": 433204.9020996094,
      "end_time": 1557221766692.589
     }
    },
    "scrolled": false
   },
   "source": [
    "## A bit more in depth\n",
    "\n",
    "### Variables\n",
    "\n",
    "You can bind a sequence of values to a (dollar-prefixed) variable, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.007143735885620117 ms\n",
      "\"Bearing 3 1 4 Mark 5. Engage!\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $x := \"Bearing 3 1 4 Mark 5. \"\n",
    "return concat($x, \"Engage!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006165742874145508 ms\n",
      "\"Kirk and Picard and Sisko\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $x := (\"Kirk\", \"Picard\", \"Sisko\")\n",
    "return string-join($x, \" and \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can bind as many variables as you want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006880044937133789 ms\n",
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $x := 1\n",
    "let $y := $x * 2\n",
    "let $z := $y + $x\n",
    "return ($x, $y, $z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and even reuse the same name to hide formerly declared variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006127119064331055 ms\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $x := 1\n",
    "let $x := $x + 2\n",
    "let $x := $x + 3\n",
    "return $x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iteration\n",
    "\n",
    "In a way very similar to let, you can iterate over a sequence of values with the \"for\" keyword. Instead of binding the entire sequence of the variable, it will bind each value of the sequence in turn to this variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006555080413818359 ms\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n",
      "10\n",
      "12\n",
      "14\n",
      "16\n",
      "18\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "for $i in 1 to 10\n",
    "return $i * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More interestingly, you can combine fors and lets like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006516933441162109 ms\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n",
      "10\n",
      "12\n",
      "14\n",
      "16\n",
      "18\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $sequence := 1 to 10\n",
    "for $value in $sequence\n",
    "let $square := $value * 2\n",
    "return $square"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and even filter out some values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0077419281005859375 ms\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $sequence := 1 to 10\n",
    "for $value in $sequence\n",
    "let $square := $value * 2\n",
    "where $square < 10\n",
    "return $square"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you can only iterate over sequences, not arrays. To iterate over an array, you can obtain the sequence of its values with the [] operator, like so:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006000041961669922 ms\n",
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "[1, 2, 3][]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditions\n",
    "\n",
    "You can make the output depend on a condition with an if-then-else construct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0064771175384521484 ms\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "-5\n",
      "-6\n",
      "-7\n",
      "-8\n",
      "-9\n",
      "-10\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "for $x in 1 to 10\n",
    "return if ($x < 5) then $x\n",
    "                   else -$x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the else clause is required - however, it can be the empty sequence () which is often when you need if only the then clause is relevant to you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Composability of Expressions\n",
    "\n",
    "Now that you know of a couple of elementary JSONiq expressions, you can combine them in more elaborate expressions. For example, you can put any sequence of values in an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.007096052169799805 ms\n",
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "[ 1 to 10 ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can dynamically compute the value of object pairs (or their key):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.007810831069946289 ms\n",
      "{\"Greeting\": \"Hello, Mister Spock\", \"Farewell\": \"Live long and prosper\"}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "{\n",
    "      \"Greeting\" : (let $d := \"Mister Spock\"\n",
    "                    return concat(\"Hello, \", $d)),\n",
    "      \"Farewell\" : string-join((\"Live\", \"long\", \"and\", \"prosper\"),\n",
    "                               \" \")\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can dynamically generate object singletons (with a single pair):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006745100021362305 ms\n",
      "{\"Integer 2\": 4}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "{ concat(\"Integer \", 2) : 2 * 2 }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and then merge lots of them into a new object with the {| |} notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006300926208496094 ms\n",
      "{\"Square of 1\": 1, \"Square of 2\": 4, \"Square of 3\": 9, \"Square of 4\": 16, \"Square of 5\": 25, \"Square of 6\": 36, \"Square of 7\": 49, \"Square of 8\": 64, \"Square of 9\": 81, \"Square of 10\": 100}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "{|\n",
    "    for $i in 1 to 10\n",
    "    return { concat(\"Square of \", $i) : $i * $i }\n",
    "|}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON Navigation\n",
    "\n",
    "Up to now, you have learnt how to compose expressions so as to do some computations and to build objects and arrays. It also works the other way round: if you have some JSON data, you can access it and navigate.\n",
    "All you need to know is: JSONiq views\n",
    "an array as an ordered list of values,\n",
    "an object as a set of name/value pairs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objects\n",
    "\n",
    "You can use the dot operator to retrieve the value associated with a key. Quotes are optional, except if the key has special characters such as spaces. It will return the value associated thereto:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.009386062622070312 ms\n",
      "\"Sarah\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $person := {\n",
    "    \"first name\" : \"Sarah\",\n",
    "    \"age\" : 13,\n",
    "    \"gender\" : \"female\",\n",
    "    \"friends\" : [ \"Jim\", \"Mary\", \"Jennifer\"]\n",
    "}\n",
    "return $person.\"first name\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also ask for all keys in an object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.00790095329284668 ms\n",
      "{\"keys\": [\"name\", \"age\", \"gender\", \"friends\"]}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $person := {\n",
    "    \"name\" : \"Sarah\",\n",
    "    \"age\" : 13,\n",
    "    \"gender\" : \"female\",\n",
    "    \"friends\" : [ \"Jim\", \"Mary\", \"Jennifer\"]\n",
    "}\n",
    "return { \"keys\" : [ keys($person)] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arrays\n",
    "\n",
    "The [[]] operator retrieves the entry at the given position:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.00620579719543457 ms\n",
      "\"Mary\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $friends := [ \"Jim\", \"Mary\", \"Jennifer\"]\n",
    "return $friends[[1+1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to get the size of an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.006299018859863281 ms\n",
      "{\"how many friends\": 3}\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $person := {\n",
    "    \"name\" : \"Sarah\",\n",
    "    \"age\" : 13,\n",
    "    \"gender\" : \"female\",\n",
    "    \"friends\" : [ \"Jim\", \"Mary\", \"Jennifer\"]\n",
    "}\n",
    "return { \"how many friends\" : size($person.friends) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the [] operator returns all elements in an array, as a sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.0063228607177734375 ms\n",
      "\"Jim\"\n",
      "\"Mary\"\n",
      "\"Jennifer\"\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $person := {\n",
    "    \"name\" : \"Sarah\",\n",
    "    \"age\" : 13,\n",
    "    \"gender\" : \"female\",\n",
    "    \"friends\" : [ \"Jim\", \"Mary\", \"Jennifer\"]\n",
    "}\n",
    "return $person.friends[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relational Algebra\n",
    "\n",
    "Do you remember SQL's SELECT FROM WHERE statements? JSONiq inherits selection, projection and join capability from XQuery, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.01275491714477539 ms\n",
      "[{\"nb\": 1, \"state\": \"MA\", \"sold\": \"broiler\"}, {\"nb\": 1, \"state\": \"MA\", \"sold\": \"socks\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"toaster\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"toaster\"}, {\"nb\": 2, \"state\": \"MA\", \"sold\": \"socks\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"toaster\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"blender\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"blender\"}, {\"nb\": 3, \"state\": \"CA\", \"sold\": \"shirt\"}]\n"
     ]
    }
   ],
   "source": [
    "%%rumble\n",
    "let $stores :=\n",
    "[\n",
    "    { \"store number\" : 1, \"state\" : \"MA\" },\n",
    "    { \"store number\" : 2, \"state\" : \"MA\" },\n",
    "    { \"store number\" : 3, \"state\" : \"CA\" },\n",
    "    { \"store number\" : 4, \"state\" : \"CA\" }\n",
    "]\n",
    "let $sales := [\n",
    "    { \"product\" : \"broiler\", \"store number\" : 1, \"quantity\" : 20  },\n",
    "    { \"product\" : \"toaster\", \"store number\" : 2, \"quantity\" : 100 },\n",
    "    { \"product\" : \"toaster\", \"store number\" : 2, \"quantity\" : 50 },\n",
    "    { \"product\" : \"toaster\", \"store number\" : 3, \"quantity\" : 50 },\n",
    "    { \"product\" : \"blender\", \"store number\" : 3, \"quantity\" : 100 },\n",
    "    { \"product\" : \"blender\", \"store number\" : 3, \"quantity\" : 150 },\n",
    "    { \"product\" : \"socks\", \"store number\" : 1, \"quantity\" : 500 },\n",
    "    { \"product\" : \"socks\", \"store number\" : 2, \"quantity\" : 10 },\n",
    "    { \"product\" : \"shirt\", \"store number\" : 3, \"quantity\" : 10 }\n",
    "]\n",
    "let $join :=\n",
    "    for $store in $stores[], $sale in $sales[]\n",
    "    where $store.\"store number\" = $sale.\"store number\"\n",
    "    return {\n",
    "        \"nb\" : $store.\"store number\",\n",
    "        \"state\" : $store.state,\n",
    "        \"sold\" : $sale.product\n",
    "    }\n",
    "return [$join]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access datasets\n",
    "\n",
    "Rumble can read input from many file systems and many file formats. If you are using our backend, you can only use json-doc() with any URI pointing to a JSON file and navigate it as you see fit. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.07178306579589844 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'There was an error.\\n\\nCode: [FODC0002] (this code can be looked up in the documentation and specifications).\\n\\nLocation information: file:/home/ubuntu/:LINE:1:COLUMN:0:\\n\\nMalformed URI: Put any HTTP URL pointing to a JSON document here! Cause: Illegal character in path at index 3: Put any HTTP URL pointing to a JSON document here!'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%rumble\n",
    "json-doc(\"Put any HTTP URL pointing to a JSON document here!\").foo[[1]].bar.foobar[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using your own Rumble server on your cluster, you can also use any other function and scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took: 0.049546241760253906 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'There was an error.\\n\\nCode: [FODC0002] (this code can be looked up in the documentation and specifications).\\n\\nLocation information: file:/home/ubuntu/:LINE:1:COLUMN:0:\\n\\nMalformed URI: put the path to a JSON lines file here. This will only work against your own Rumble backend and Spark cluster, though. Cause: Illegal character in path at index 3: put the path to a JSON lines file here. This will only work against your own Rumble backend and Spark cluster, though.'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%rumble\n",
    "json-file(\"put the path to a JSON lines file here. This will only work against your own Rumble backend and Spark cluster, though.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}