DFSG_JournalEntry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red' size=10>Be Strong, Huy! </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_f`: Current focus\n",
    "* Purpose: Refine other functions that use `base` to mine information\n",
    "* Date modified:\n",
    "    * Tuesday, 6/4/2019:\n",
    "        * Created\n",
    "        * I was able to simplify EDUCATION_YEARS and then stopped there because I felt that I should have used tokenization rather than splitting. This could speed up the code significantly.\n",
    "        * Also, need to modify the function `education_years` because sometimes the job will say \"Bachelor's degree\" or \"Associate's degree\".\n",
    "        * Next steps:\n",
    "            * See above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_e`: Current focus\n",
    "* Purpose: Refine `base` function to fit more jobs\n",
    "* Date modified:\n",
    "    * Tuesday, 6/4/2019:\n",
    "        * Created\n",
    "        * I got quite a few jobs parsed with the new refined `base` function. Turns out that refining could just mean make a function less complicated by not going into too much detailed of the specifics of a job.\n",
    "        * Have to learn regex to obtain relevant information because it's not always `REQUIREMENTS/MINIMUM QUALIFICATIONS` and `PROCESS NOTES`.\n",
    "        * Some jobs don't even have imitized initiators, so need to manually add in.\n",
    "        * I feel that I'm wasting so much time on this, but I'm going to stick to it until the deadline.\n",
    "        * Next steps:\n",
    "            * See above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_d`:\n",
    "* Purpose: Playground to test for ideas on refining `_base`\n",
    "* Date modified:\n",
    "    * Tuesday, 6/4/2019:\n",
    "        * Created\n",
    "        * I figured out lots of things messed up; thus, a new approach was required. See notebook `Object1_e`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_c`\n",
    "* Purpose: Define functions/helper functions to fill in values for dataframe\n",
    "* Date modified:\n",
    "    * Sunday, 6/2/2019:\n",
    "        * <font size=3>DONEEEEE!!!!</font>\n",
    "        * EXAM_TYPE, I borrowed the code from Daniel Becker on Kaggle, with regular expression approach as I don't have an effective way of dealing with it.\n",
    "        * Most of these functions are very primitive, so will have to build up on them.\n",
    "        * Next steps:\n",
    "            * ~~Print out~~ the code and reorganize for a more consistent manner.\n",
    "            * Have a new kernel that just has the functions and then build a complete dataframe.\n",
    "            * Apply the final approach to two other annotated files and see if things work.\n",
    "            * Maybe I should use other kernels to improve my solution. Regex code for finding salary, etc., should be employed.\n",
    "    * Thursday, 5/30/2019:\n",
    "        * I have been working on this nonstoppingly but didn't have time to write up changes and work in progress. This is a bad habit and I'll need to fix it.\n",
    "        * Last time, I mentioned that we should write a function for each field name. I have been successful at this task and currently, there are only a few field names remained.\n",
    "        * Learned a couple of new things. For example: Use parentheses () for line continuation.\n",
    "        * I start to lose some of my hope but will pull through the competitions and see how things are.\n",
    "        * There are two biggest things I can contribute for now:\n",
    "            * <font color='red'>A clean csv file since this is still the host's biggest interest. This means a lot of checkings and labor work.</font>\n",
    "            * Learn nltk from book and kernels and apply it to my competition.\n",
    "        * This means I have to structure my time in a more consitent manner.\n",
    "        * Next steps:\n",
    "            * ~~Finish writing up functions for Systems Analyst. Only a few more to go!~~\n",
    "            * Have a new kernel that just has the functions and then build a complete dataframe\n",
    "            * Tweak the code for `_base` function since it is no longer appropriate. For example: instead of stripping off dots in 1. and then scanning for numbers, I saw that some jobs had numbers in it, like 24 units, etc.\n",
    "            * Apply the new notebook to two other annotated files, observe the results, and make changes appropriately.\n",
    "    * Sunday, 5/26/2019:\n",
    "        * Created\n",
    "        * I successful wrapped up a lot of individual lines of code into functions.\n",
    "        * The salary part really gave me a hard time, but I was so proud when getting it done.\n",
    "        * Currently, I still don't know exactly how to handle REQUIREMENT_SET_ID, but it seems that the functional approach may be able to resolve this issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_a`\n",
    "* Purpose: Build a list of key words for `EXP_JOB_CLASS_TITLE`\n",
    "* Date modified:\n",
    "    * Thursday, 5/23/2019:\n",
    "        * Created\n",
    "        * I had to manually copy/paste the list in a word-processing app (pages) and read and edit the list to have a cleaner one. Not just Bob can do it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1_b`\n",
    "* Purpose: Outline idea for building a dataframe for `SYSTEMS ANALYST 1596 102717.txt`\n",
    "* Date modified:\n",
    "    * Sunday, 5/26/2019:\n",
    "        * Created\n",
    "        * I tried different approaches and finally concluded that the best way to generate a dataframe is to build a dictionary of all fieds and have fill in each of them with functions (see `Objective1_c` for all of these functions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1`\n",
    "* Purpose: Build a csv for `SYSTEMS ANALYST 1596 102717.txt`\n",
    "* Date modified:\n",
    "    * Sunday, 5/26/2019:\n",
    "        * Created\n",
    "        * Nothing here yet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective1+`\n",
    "* Purpose: Build a csv file for `SYSTEMS ANALYST 1596 102717.txt`\n",
    "* Date modified:\n",
    "    * Sunday, 5/26/2019:\n",
    "        * Renamed from `Objective1` to `Objective1+`. I messed up here and it was a good lesson for me when trying to organize my work. I created it as `Objective 1`, which said that `Build a csv for SYSTEMS ANALYST 1596 102717.txt`, but I didn't do exactly that. I did a lot of exploratory data analyses on what I would need to do (hence the the a, b, c, etc.) Then I tried to rename the files, which quickly became a real trouble because I had no way to keep track of the changes. I thought, \"If I'm confused so much like that, how could someone understand my work?\" So I created `Objective1` and kept it there for later, and changed this notebook to `Objective 1+`.\n",
    "        * I know this is confusing, but I believe I did the best to avoid confusion. Furthermore, this won't happen anymore. From now on, I'll keep track of my work very clearly: each notebook will says exactly what it attempts to do and nothing else!\n",
    "    * Thursday, 5/23/2019:\n",
    "        * I successfully resolve `EXP_JOB_CLASS_TITLE` problem now. The idea is to build a list of titles that the city has and then we can look it up. This list is obtained by going through the `City Job Path` folder, convert each pdf file into a text file, and then build a list of key words. See Notebook `Objective1_a`.\n",
    "    * Wednesday, 5/22/2019:\n",
    "        * I was able to went all the way to `requirement_id` and `requirement_subset_id`\n",
    "        * For education year, school type, education major, experience length, and full-time/part-time, I didn't solve them yet because I thought they were quite predictable. See the notebook for ideas/thought processes.\n",
    "        * I got stuck the most with `EXP_JOB_CLASS_TITLE` because I had no clue what title the city had. For example, System Aide or Chief Clerk.\n",
    "    * Sunday, 5/19/2019\n",
    "        * Created"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Objective_0a`\n",
    "* Purpose: Inspect inconsistent naming in the provided txt files\n",
    "* Date modified:\n",
    "    * Sunday, 5/19/2019: \n",
    "        * Created\n",
    "        * See Note 1 and Note 2 in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}