{"metadata": {"kernelspec": {"display_name": "sd212", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1"}}, "nbformat": 4, "nbformat_minor": 2, "cells": [{"cell_type": "markdown", "metadata": {}, "source": ["## [name] Type your name on the next line\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# SD212 Lab 02: Enron emails\n", "\n", "[webpage]: https://usna.edu/Users/cs/SD212/lab/enron/\n", "\n", "## Instructions\n", "\n", "Be sure to **actually read through the [lab webpage][webpage]**.\n", "There is a lot of useful information and detail not in this\n", "notebook file!\n", "\n", "Remember to put `%%bash` on the first line of a code cell to make\n", "Jupyter treat it as a bash script.\n", "\n", "Every question should be answered in the *same markdown cell* as\n", "the question header itself.\n", "\n", "You need to show your work. For most questions, that means adding\n", "a code cell (with `%%bash`) right below your answer to the question,\n", "where the output from that code cell shows the answer somehow.\n", "\n", "To test that you have everything working correctly, do\n", "\n", "1.  Clear All Outputs\n", "2.  Restart\n", "3.  Run All\n", "\n", "And all of the computations and graphs should be re-generated.\n", "\n", "Good luck and have fun!"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Directory selection (determines 1% 10% or 100% dataset)\n", "\n", "The `%cd` command below is a special command to change the\n", "working directory in Jupyter. Initially we have it set to\n", "go to the 1% dataset folder. Change it to the 10% folder and\n", "then the 100% one when you are ready to \"level up\"!\n", "\n", "Note, this should determine the starting point for all future\n", "bash code blocks in this notebook."]}, {"cell_type": "code", "metadata": {}, "outputs": [], "execution_count": null, "source": ["%bookmark notebook_dir\n", "%cd /home/mids/SD212/enron.01\n", "%bookmark enron_dir"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Section 3: Initial questions"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q1]\n", "> What sources of help (if any) did you utilize to complete this lab?\n", "> Please be specific.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q2]\n", "> What did you think of the lab overall? We hope that it was\n", "> challenging but instructive, and maybe even fun. Again, if you can\n", "> be specific that is helpful!\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Section 4: Single file"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q3]\n", "> What is the name of Scott Neal's fraternity brother who plays the\n", "> accordion?\n", "\n", "> (Remember, for these problems, you need to do two things.  First,\n", "> make a new code cell below each question, starting with `%%bash`,\n", "> and put the commands there which you used to answer the question.\n", "> Second, you edit the markdown cell with the question itself, and add\n", "> your answer in text.)\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q4]\n", "> Enron employee John Arnold sent an email discussing oil prices\n", "> saying (among other things), \"Who cares if we nuke afghanistan?\".\n", "\n", "> On what date was this email sent?\n", "> Type your date as `YYYY/MM/DD`.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q5]\n", "> On January 30 2002, Mark Germann from Sacramento sent an email to\n", "> a single Enron employee urging him to donate his ill-gotten gains.\n", "\n", "> What was the LAST NAME of the employee to whom Mr. Germann sent his\n", "> letter?\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q6]\n", "> After receiving it, what did that employee do with Mr. Germann's\n", "> email?\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Section 5: Make it count"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q7]\n", "> What is the total number of emails in the dataset you are looking\n", "> at?\n", "\n", "> Hint: don't count the folders. Look at the documentation for the\n", "> `-type` flag to `find`.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q8]\n", "> How many emails were *sent* by Enron employees in the dataset during\n", "> the year 1999?\n", "\n", "> (For this question, only consider emails in subfolders named 'sent'\n", "> or 'sent_items')\n", "\n", "> Hint: remember to look at the *first* time the `Date` header appears\n", "> in the email. For that purpose, the `-m` option to grep might be\n", "> useful.\n", "\n", "> Hint 2: One oddball employee has his `sent` folder in a\n", "> sub-subfolder. Make sure you don't miss it!\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q9]\n", "> How many emails mention golf in the subject line?\n", "\n", "> Hint: You will probably want to `grep` twice: First to extract the\n", "> first `Subject: ` header line, and secondly to do a\n", "> *case-insensitive* search for \"golf\" in each of those lines.\n", "\n", "> Check out the `-i` and `-c` flags to grep.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q10]\n", "> How many emails contain profanity?\n", "\n", "> There is some flexibility in your definition of \"profanity\" here,\n", "> but try to do your best to capture the kind of words that would be\n", "> \"bleeped\" on network TV or radio, without double-counting.\n", "> [This research paper][seven] contains a classic list of such\n", "> words.\n", "\n", "> Hint: make a regular expression for profanity for grep. Use the `|`\n", "> \"alternation\" operator to allow multiple possibilities.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Section 6: Choose your own adventure"]}, {"cell_type": "code", "metadata": {}, "outputs": [], "execution_count": null, "source": ["# this should change back to your lab directory\n", "%cd -b notebook_dir"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q11]\n", "> What question are you trying to answer or investigate using this\n", "> data?\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q12]\n", "> Create a bash cell (below this question) in your notebook that goes\n", "> through the email data and creates a file `mydata.csv` which has the\n", "> info needed to analyze your question.\n", "\n", "> Briefly explain the contents of your `mydata.csv` file. What does\n", "> each row and column represent?\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q13]\n", "> Which dataset did you use to create your `mydata.csv` file?\n", "\n", "> a)  1% dataset (smallest)\n", "> b)  10% dataset\n", "> c)  100% dataset (largest)\n", "\n", "> (Write just the letter of your answer below.)\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q14]\n", "> Create a Python code cell that uses the data science libraries we\n", "> have seen such as Pandas and Plotly to read in your `mydata.csv`\n", "> file and create a graph from it. Save your graph as an image\n", "> `mygraph.png` (with 1000x600 dimensions) to turn in.\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## [Q15]\n", "> Having completed your analysis, what conclusions (if any) can you\n", "> draw about the original question? If you had time, what improvements\n", "> would you make or what would be the next thing you would\n", "> investigate?\n", "\n", "\n", "\n", "\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Example \"Choose your own adventure\": Eating time\n", "\n", "Please use this as an *example* of the kind of work we want\n", "to see from you for the last section.\n", "\n", "Here we are going to examine what times of day (hours) did\n", "people send an email that mentions \"food\".\n", "\n", "Keep in mind, you should look at something completely different\n", "for your own question! As in, if what you do is look at\n", "hours of the day when people mention \"oil\" for example, that\n", "is not going to earn a high grade."]}, {"cell_type": "code", "metadata": {}, "outputs": [], "execution_count": null, "source": ["%%bash\n", "# First part: gathering data to sampledata.csv\n", "# We will make a csv that has the filename and date for every\n", "# email that mentions \"food\".\n", "# Note that we are using a semicolon to separate columns\n", "# because there are commas in the dates!\n", "echo \"fname;time\" >'sampledata.csv'\n", "for foodfile in $(find /home/mids/SD212/enron.01 -type f -exec grep -l 'food' {} +)\n", "do\n", "    echo -n \"$foodfile;\"\n", "    grep -m1 '^Date' \"$foodfile\" | cut -d' ' -f6\n", "done >>'sampledata.csv'"]}, {"cell_type": "code", "metadata": {}, "outputs": [], "execution_count": null, "source": ["# Second part: processing the csv file using Python\n", "import pandas as pd\n", "import plotly.express as px\n", "foodies = pd.read_csv('sampledata.csv', sep=';')\n", "foodies['hour'] = pd.to_datetime(foodies['time'], format='%H:%M:%S').dt.hour\n", "px.histogram(foodies, x='hour', nbins=24, width=1000, height=600)"]}]}