Introduction to remote data science

.ipynb_checkpoints/L4_Deployment_Tricks-checkpoint.ipynb

@@ -0,0 +1,273 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9f307d83-f207-40a4-9d66-6ee8fd6b6c00",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "# Lesson 4: Node Deployment Tips & Tricks!\n",
+    "\n",
+    "<b><u>Instructors</b></u>: Ishan Mishra, Phil Culliton\n",
+    "<br>\n",
+    "<b><u>Special Thanks to</b></u>: Madhava Jay\n",
+    "\n",
+    "Our stack is constantly evolving and getting better and more user-friendly. For the current release version of PyGrid and Hagrid, here are a few tips and tricks I've found useful on a Linux/OS X/WSL environment when it comes to working with this infrastructure.\n",
+    "\n",
+    "If you come up with any useful tricks of your own, please do share them with us and the community!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "715eee4b-fa7f-4484-ab71-b092835dcb11",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cd624d42-0392-447a-9de4-2f3822af2446",
+   "metadata": {},
+   "source": [
+    "## Aliases\n",
+    "Here are some bash aliases I use for varous purposes, such as in order to ease the process of launching my local domain node."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "de4e0e91-01dc-48a9-96b4-b2c4d1c6c028",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# Activates the venv, and navigates to the Syft directory\n",
+    "alias syft=\"conda activate <env_name> && cd <path/to/Pysyft>\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5f59b557-5853-4fe8-a951-7c90bde50379",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# Launches the local domain node with one word\n",
+    "alias launch=\"hagrid launch <node_name> domain to docker:8081+ tail=True\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f2f6bdc8-f1cb-4b75-a97d-d3277c40d77a",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# Shuts down all local domain nodes\n",
+    "alias land=\"hagrid land all\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d782924b-3deb-4002-8a02-1cc75ef0336c",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "50fc4c9a-eb7c-490c-98f1-edd19b91f8db",
+   "metadata": {},
+   "source": [
+    "# Clearing Up Space\n",
+    "Docker has a reputation for taking up a suprisingly large amount of space as its containers are running. Here are a few ways I've found to mitigate those effects. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7ff124a8-75c0-4a0e-be47-091f4d19883e",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# This cleans up after docker\n",
+    "# WARNING: will delete all currently unused or inactive containers, networks, volumes, etc- this will result in losing any data you uploaded to  #Domain Nodes that aren't currently running\n",
+    "\n",
+    "hard_reset() {\n",
+    "        echo \"Pruning Network\"\n",
+    "        docker network prune -f\n",
+    "        echo \"Pruning Volume\"\n",
+    "        docker volume prune -f\n",
+    "        echo \"Pruning System\"\n",
+    "        docker system prune -f\n",
+    "        echo \"Pruning Containers\"\n",
+    "        docker container prune -f\n",
+    "}\n",
+    "\n",
+    "# This will delete active or unused containers, networks, but won't delete any data you've uploaded or stored.\n",
+    "\n",
+    "cleanup() {\n",
+    "        echo \"Pruning Network\"\n",
+    "        docker network prune -f\n",
+    "        echo \"Pruning System\"\n",
+    "        docker system prune -f\n",
+    "        echo \"Pruning Containers\"\n",
+    "        docker container prune -f\n",
+    "}\n",
+    "\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c530773e-0973-455d-8884-b6a83147ed6c",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dba2e61a-dc86-4dba-9e44-fb2555c11227",
+   "metadata": {},
+   "source": [
+    "# Testing and debugging\n",
+    "Here are some commands that will make it easier to work with Docker containers, while testing, debugging, or for other purposes. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fb5c3696-ce57-4874-abb2-2a0f3dcb7add",
+   "metadata": {},
+   "source": [
+    "#### <b> ctop </b>\n",
+    "ctop is a handy tool that lets you quickly monitor and manage all your containers (both active and inactive):"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e7ad64d8-83da-496c-86e6-0bcb32006762",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# To install, simply\n",
+    "sudo apt install ctop\n",
+    "\n",
+    "# An alternative version of this package can be obtained from:\n",
+    "https://github.com/bcicen/ctop\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d86de22f-1b91-41b7-8f05-2070c7b815bc",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# ctop is a quick way to monitor containers\n",
+    "ctop\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "526c5add-40c9-465f-a2ba-990112a082ac",
+   "metadata": {},
+   "source": [
+    "By hovering over a container and pressing enter, you gain the ability to:\n",
+    "\n",
+    "- <b> View: </b>\n",
+    "    - <b> Single View </b> (keybind 'o'): view details of that specific container, including ports, health, CPU/MEM usage, NET/IO, environment variables, and much more.\n",
+    "    - <b> Log View </b> (keybind 'l'): View the logs of just this container. This is in contrast to launching a local domain node with tail=True, as that method continuousy outputs all the container logs in real time.\n",
+    "- <b> Interact: </b>\n",
+    "    - <b> Stop </b> (keybind 's'): Stop the container.\n",
+    "    - <b> Pause </b> (keybind 'p'): Pause the container.\n",
+    "    - <b> Restart </b> (keybind 'r'): Restart the container.\n",
+    "    - <b> Exec shell </b> (keybind 'e'): This lets you run commands inside this shell. This is very handy if you want to run test files inside a container, for instance."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ed81829b-fe69-4dd8-8848-f4da1a060e91",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "745b115e-48ee-45e0-ac93-7bbbb7bba742",
+   "metadata": {},
+   "source": [
+    "## Miscellaneous Hagrid Tricks\n",
+    "\n",
+    "<b> Seeing the logs from Hagrid in real time: </b>\n",
+    "``` \n",
+    "# Add the \"tail=True\" parameter!\n",
+    "hagrid launch local_node tail=True\n",
+    "```\n",
+    "\n",
+    "<b> Seeing a list of commands being run in Hagrid: </b>\n",
+    "``` \n",
+    "# Use the cmd=True parameter!\n",
+    "hagrid launch local_node cmd=True\n",
+    "```\n",
+    "\n",
+    "<b> Shutting down your docker containers without losing data: </b>\n",
+    "```\n",
+    "# This is great if the \"hagrid land\" command isn't working\n",
+    "docker rm `docker ps -aq` --force\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0613e05-9a03-429b-8188-0508d4c7aa54",
+   "metadata": {},
+   "source": [
+    "<b> Resetting Domain Node Storage/Memory </b>\n",
+    "\n",
+    "<t> By deleting the db container, you can do a hard reset on the domain node's storage and memory."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30b313f7-82e0-4cf5-8500-d43fdd10b8f0",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eb4cf1af-b20a-4a3b-9a36-16e7f33edf7f",
+   "metadata": {},
+   "source": [
+    "<hr>"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}