{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7b8133c1-430f-495f-a7bf-8b58c5a1c797",
   "metadata": {},
   "source": [
    "# Towards Scientific Programming\n",
    "\n",
    "Today we'll look at what it takes to handle truly **huge** amounts of data, as one often would when dealing with scientific data.\n",
    "\n",
    "As we've seen, with most algorithms & data structures the limiting factor will be the amount of data.\n",
    "\n",
    "Everything is \"easy\" when the data is small.\n",
    "\n",
    "### Example O(1) Operations\n",
    "\n",
    "- add/remove on linked list\n",
    "- modify array at the end\n",
    "- array access\n",
    "\n",
    "### Example O(n) Operations\n",
    "\n",
    "- A linear search for a record in sequence.\n",
    "- Always *some* operations on a linked list/array type, depending on which is chosen.\n",
    "    - insert at start/middle of array\n",
    "    - access random element in linked list\n",
    "- Memory-move type operations (list/hashtable outgrows allocated memory)\n",
    "- String concatenation with `__add__`\n",
    "\n",
    "### Example O(log2 n) Operations\n",
    "\n",
    "- Binary search/sort.\n",
    "- Grow slower, but overhead & memory complexity can still limit large data sets.\n",
    "\n",
    "### Example O(n^2) / O(n*m) Operations\n",
    "\n",
    "- Comparing every item to every other item. (Simple checking for duplicates.)\n",
    "- Generally: Nested O(n) operations.  (for each of N files, for each of M words in file...)\n",
    "\n",
    "**And remember, to get better performance we are often spending extra memory**.  \n",
    "\n",
    "So what happens when data gets too big?\n",
    "\n",
    "## Strategies for \"big\" data.\n",
    "\n",
    "I am avoiding the term of art \"Big Data\" which often has specific connotations.  Our \"big data\" would encompass \"Big Data\" but also any data where performance starts to become an issue.\n",
    "\n",
    "### \"Premature Optimization is the Root of All Evil\"\n",
    "\n",
    "(-Donald Knuth)\n",
    "\n",
    "A halllmark of an \"intermediate\" programmer is that they have all of the tools of the experienced programmer, but they lack the wisdom to know when to use which one. \n",
    "\n",
    "A common mistake stems from attempts to optimize code that was never going to be the bottleneck.\n",
    "\n",
    "Generally, when there is a straighforward way to do something, do it first!\n",
    "\n",
    "Write your code in an organized way and you'll be able to fairly easily adapt if it turns out that you need to optimize it with one of the techniques introduced below.\n",
    "\n",
    "For example, if you need to do something 1,000,000 times.\n",
    "\n",
    "Start with a function that performs that task. And call it on a subset of that data:\n",
    "\n",
    "```\n",
    "for item in full_data_set[:10000]:\n",
    "    do_something(item)\n",
    "```\n",
    "\n",
    "If this takes 1 minute to run, then your full data set will take 100 minutes.  \n",
    "\n",
    "- Will it save you more than that to rewrite?\n",
    "- What if it needs to run every week?\n",
    "- What if it takes 1 hour?\n",
    "\n",
    "Let's say it is still too slow: what then?\n",
    "\n",
    "### Strategy 1: Better Algorithms & Data Structures\n",
    "\n",
    "Before employing the strategies below, it is generally advisable to think about what portions of your code are slow.\n",
    "\n",
    "Sometimes, for a simple enough program, or an experienced enough dev, you can reason this out, but if in doubt you should **profile** your code to determine the slowest portions and work to optimize them.  The results can be surprising, perhaps you are calling a method that does a `deepcopy` that you didn't realize was there, or using an array where a linked list would be more appropriate.\n",
    "\n",
    "- `timeit`\n",
    "- `cProfile`\n",
    "\n",
    "<https://docs.python.org/3/library/profile.html>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c9b0e0-9afe-45cc-872f-00056807852f",
   "metadata": {},
   "source": [
    "### Python's General Performance\n",
    "\n",
    "Python's flexibility comes with costs. Python has a reputation for slowness in some contexts. Other implementations of the Python interpreter attempt to overcome some of the default implementation's shortcomings (e.g., Cython, PyPy, Numba).\n",
    "\n",
    "> The relative sluggishness of Python generally manifests itself in situations where many small operations are being repeated—for instance, looping over arrays to operate on each element.\n",
    "> It turns out that the bottleneck [...] is not the operations themselves, but the type-checking and function dispatches that CPython must to at each cycle of the loop.\" [This is where compiled code has an advantage.]\n",
    "\n",
    "**-Jake VanderPlas, Python Data Science Handbook**\n",
    "\n",
    "Solution: *vectorized* functions via \"ufuncs\" that circumvent problems of this nature.\n",
    "\n",
    "These functions execute in the C-layer instead of the Python layer. They bypass type checking, dunder lookups, and other overhead, allowing the full speed of the CPU (and GPU) to be utilized.\n",
    "\n",
    "Data will still be the limiting factor, but if we can get a 10-100x speedup, that's another order of magnitude or two worth of data we can process per machine. (So when combined with parallelization, can drastically save on costs.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf94429-34c4-4f45-b912-e13aa895ed79",
   "metadata": {},
   "source": [
    "### Strategy 2: Parallelization\n",
    "\n",
    "Depending on the needs of the program, your next bet may be to paralellize the operations.\n",
    "\n",
    "This means splitting the data up into smaller pieces, and have multiple processing pipelines handle subsets of the data.\n",
    "\n",
    "This can mean multiple computers on a network, or on modern machines, multiple threads or processes on the same machine.\n",
    "\n",
    "A full exploration of this is beyond the scope of what we can cover in this class, but here are some techniques:\n",
    "\n",
    "- [`threading`](https://docs.python.org/3/library/threading.html)\n",
    "- [`subprocess`](https://docs.python.org/3/library/subprocess.html)\n",
    "- [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html)\n",
    "- `async/await` - <https://docs.python.org/3/library/asyncio-task.html>\n",
    "\n",
    "### Aside: the GIL\n",
    "\n",
    "Python has a feature known as the \"global interpreter lock\" which prevents two Python threads from modifying the same data structure at the same time.\n",
    "\n",
    "This means that two threads can't accidentally remove the same item from a list, or set the same key in a dictionary to two different values.  It comes at the expense of performance, since you lose the ability to use these data structures concurrently whatsover.\n",
    "\n",
    "Some languages give much more finely grained control, which comes with more responsibility & room for error.\n",
    "\n",
    "For this reason, most Python programs prefer subprocesses to subthreads.  (If you take parallel programming or an operating systems course you will likely explore the differences.)\n",
    "\n",
    "Python 3.13 supports an experimental GIL-free mode, something that's been considered unlikely for 20+ years. This could be the biggest change to the language in decades."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8214e0f8-0467-477f-909d-cf447107c6d3",
   "metadata": {},
   "source": [
    "### Strategy 3: Vectorization\n",
    "\n",
    "Another approach that can help in tandem with parallelization, and which can be easier to reach for without refactoring your code, is to take advantage of modern hardware's support for vectorization.\n",
    "\n",
    "Vectorization takes advantage of the fact that it is often cheap to do the same operation to adjacent memory locations.\n",
    "\n",
    "With a large array, instead of adding +1 to each item, the array could instead be merged with [+1, +1, +1, +1, +1, +1, +1, +1...] incrementing many items simultaneously."
   ]
  },
  {
   "attachments": {
    "b5041abc-5968-4f8e-bfbf-a51ec77029d1.png": {
     "image/png": "[encoded data, 469696 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "392cb87a-9b22-42ef-87b9-9541b8211dbf",
   "metadata": {},
   "source": [
    "## Arrays\n",
    "\n",
    "Python's dynamic typing makes it extremely flexible, but this flexibility comes at a cost.\n",
    "\n",
    "Python's built in types are often cleverly disguised C structures containing the data associated with the object as well as header information.\n",
    "\n",
    "Every Python `object` has a header:\n",
    "\n",
    "- `ob_refcnt`: reference count used for garbage collection\n",
    "- `ob_type`: type of the object (how to interpret underlying bytes)\n",
    "- `obj_size`: size of data in bytes\n",
    "\n",
    "Lists (for example) are extremely flexible and can hold any `object`, we've said these things are in adjacent memory, but how does that work if the items are a different size?\n",
    "\n",
    "![list.png](attachment:b5041abc-5968-4f8e-bfbf-a51ec77029d1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66626152-828d-4c09-90a5-d743ffee63d6",
   "metadata": {},
   "source": [
    "Lookups are still O(1), but there is an extra level of indirection, as we've discussed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742de111-ffbb-4e38-8f3d-44e1c821ffb2",
   "metadata": {},
   "source": [
    "### `array` module\n",
    "\n",
    "Allows us to create dense, **homogenous** arrays without indirection. \n",
    "\n",
    "Unlike `list` (and virtually everything else) we actually need to declare a type when doing so.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cb64b2a7-dce3-4b5c-a84b-0e66f43addc9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "array('i', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "array('f', [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0])\n"
     ]
    }
   ],
   "source": [
    "import array\n",
    "\n",
    "# array of 10 integers\n",
    "int_ar = array.array(\"i\", range(10))\n",
    "float_ar = array.array(\"f\", range(10))\n",
    "print(int_ar)\n",
    "print(float_ar)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c01677a9-b77c-4af9-811d-da4f262fdc4a",
   "metadata": {},
   "source": [
    "## NumPy Arrays\n",
    "\n",
    "Python's `array` object provides efficient storage on array-based data, **NumPy** adds efficient operations on that data as well as a nicer interface.\n",
    "\n",
    "NumPy will upcast data if there is data of different types in a single array.  For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fb348a4c-48f5-4c49-9f25-c28385ceef16",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3 4 5]\n",
      "[1. 2. 3. 4. 5.]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "print(np.array([1, 2, 3, 4, 5]))  \n",
    "print(np.array([1, 2, 3, 4.0, 5]))  # one float will make the entire array floats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b8670310-312a-478a-ab1a-6cdd24efaeee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 2. 3. 4. 5.]\n"
     ]
    }
   ],
   "source": [
    "# you can also specify the type via `dtype`\n",
    "print(np.array([1, 2, 3, 4, 5], dtype=\"float32\"))  \n",
    "# all types: https://numpy.org/doc/stable/user/basics.types.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914be27e-cf15-445b-943c-4567f1c4a85b",
   "metadata": {},
   "source": [
    "NumPy arrays are multidimensional.  The arrays we've seen are just a special case with one axis.\n",
    "\n",
    "- **axes**: In NumPy dimensions are usually called axes.\n",
    "- **rank**: Number of axes in a given array.\n",
    "- **length**: number of elements in a given axis.\n",
    "- **shape**: size of array in each axis, given as a tuple.\n",
    "- **size**: total number of elements in entire array (all axes).\n",
    "- **itemsize**: Size in bytes of each element in array.\n",
    "- **data**: Underlying buffer used to store data, generally not accessed directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "606b9779-7ba5-4c22-8015-5f1f8c457411",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0\n"
     ]
    }
   ],
   "source": [
    "a = np.array([[1, 2, 3], [4, 5, 6]], dtype=\"float32\")\n",
    "print(a[0, 1])  # Note: multidimensional access, different from m[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "915954c5-843b-4da3-9e60-9d648fa0c068",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 4.], dtype=float32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:, 0]  # entire row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "75be9779-b3bc-43e3-9769-5a648f0df1c9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 4.], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:, 0] # entire column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ad351e9f-efec-42e8-a420-34ef58488089",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rank = 2\n",
      "shape = (2, 3)\n",
      "size = 6\n",
      "itemsize = 4\n",
      "data = <memory at 0x108169080>\n"
     ]
    }
   ],
   "source": [
    "print('rank =', len(a.shape))\n",
    "print('shape =', a.shape)\n",
    "print('size =', a.size)\n",
    "print('itemsize =', a.itemsize)\n",
    "print('data =', a.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e6b85b-75bc-41e3-b28d-32a41b74f3a3",
   "metadata": {},
   "source": [
    "### Creating numpy arrays\n",
    "- `numpy.zeros / numpy.ones`  -- create array with given shape filled with 0 or 1\n",
    "- `numpy.full` -- create array with given value\n",
    "- `numpy.arange` -- similar to range, step by a given value\n",
    "- `numpy.linspace` -- array of values between two endpoints, evenly spaced\n",
    "- `numpy.random.random` -- random values between 0 and 1\n",
    "- `numpy.random.normal` -- normally distributed random values centered on 0 w/ std.dev 1\n",
    "- `numpy.eye` -- identity matrix of a given size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6d2f82f9-ea0e-4b37-b081-156c47cd839a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.zeros((4, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f8493cc3-c05c-445b-9b50-1bb04e17f207",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1., 1., 1.],\n",
       "        [1., 1., 1.],\n",
       "        [1., 1., 1.]],\n",
       "\n",
       "       [[1., 1., 1.],\n",
       "        [1., 1., 1.],\n",
       "        [1., 1., 1.]],\n",
       "\n",
       "       [[1., 1., 1.],\n",
       "        [1., 1., 1.],\n",
       "        [1., 1., 1.]]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.ones((3, 3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e1ae268f-a7b3-49b9-b7c1-c4ac6a8498d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265],\n",
       "       [3.14159265, 3.14159265, 3.14159265, 3.14159265]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.full((7, 4), np.pi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4fbfe17b-0f2f-4105-aaac-6f2d6df7d5e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.37578793, 0.90677584, 0.70716772],\n",
       "       [0.4000077 , 0.88159431, 0.97547473],\n",
       "       [0.87264922, 0.22962945, 0.89475618]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.random((3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3cefbe84-b615-4d0b-9bec-d8cbe785e96f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0., 0.],\n",
       "       [0., 0., 0., 0., 1., 0.],\n",
       "       [0., 0., 0., 0., 0., 1.]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.eye(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d038d867-2c4b-48a1-98b9-52e7c9e8772e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0. , 1.5, 3. , 4.5, 6. ])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linspace(0, 6, 5)   # divides space [0,6] into 5 equally-sized increments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e28b82d7-49e5-4ed2-9a5a-38cb50708559",
   "metadata": {},
   "source": [
    "### Reshaping NumPy Arrays\n",
    "\n",
    "Allows us to reinterpret the existing memory as a different shape.  Must be the same total size.\n",
    "\n",
    "`reshape` - reshape to new dimensions\n",
    "\n",
    "`ravel` - flatten to rank 1\n",
    "\n",
    "`T` - transpose (note: property, not method)\n",
    "\n",
    "`resize` - reshape in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c5a68743-ad95-4cf3-b4ca-0cc56f239925",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 2 3]\n",
      " [4 5 6 7]]\n"
     ]
    }
   ],
   "source": [
    "ta = np.array([range(4), range(4, 8)])\n",
    "print(ta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f6f5e273-9c0b-4dac-943e-f91cd398fa26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1],\n",
       "       [2, 3],\n",
       "       [4, 5],\n",
       "       [6, 7]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ta.reshape(4, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c2a366f7-7204-4768-b439-8c694c82055f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ta.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "31425a7d-f14b-409d-b4f9-2581e2a6ad93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 4],\n",
       "       [1, 5],\n",
       "       [2, 6],\n",
       "       [3, 7]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ta.T  # note this is a property"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fc80d1ad-cc4e-4bee-b6bf-d782e3107ad3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ta.resize(4, 2)   # modifies ta in-place (no return value, but ta is modified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7adeb79b-1b73-4532-b034-12ea0280901f",
   "metadata": {},
   "source": [
    "### pandas\n",
    "\n",
    "For about a decade, one of the most popular libraries in Python was `pandas`.  `pandas` build on top of NumPy arrays, introducing 2D data types called `DataFrame`.  \n",
    "\n",
    "You can think of a DataFrame as a cross between a list of dictionaries and an spreadsheet.\n",
    "\n",
    "DataFrame objects consist of columns of similar data. They can take advantage of memory locality to be vectorized. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d277db45-b290-425a-88a0-e2d260daada4",
   "metadata": {},
   "source": [
    "### polars\n",
    "\n",
    "- <https://github.com/pola-rs/polars/blob/main/README.md>\n",
    "- <https://blog.jetbrains.com/pycharm/2024/07/polars-vs-pandas/>\n",
    "\n",
    "A few years ago, `polars` was released, and is becoming the preferred DataFrame library.  It provides a mostly-compatible `DataFrame` type with a few key advantages:\n",
    "\n",
    "- Implemented in Rust, which is a memory-safe language with low overhead, comparable to C.\n",
    "- Uses the 'Arrow' memory structure for columnar data.  This is an evolution of Pandas' work on column-based data formats, allowing high-speed performance & cross-language sharing of memory.\n",
    "- `pandas` took ~10 years to reach a stable interface and accumulated a lot of \"baggage\" -- methods that can't easily be fixed/changed for backwards compatibility reasons, `polars` started with a fresh take on the API & is generally easier to learn as a result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb390bd2-00e6-452a-9ff6-a9bc8d39d8b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Each of the below functions will search the file for all bills sponsored by TARGET_NAME.\n",
    "\"\"\"\n",
    "\n",
    "import csv\n",
    "\n",
    "\n",
    "def find_bills_pure_python(bills_file, legislators_file, sponsorships_file, target_name):\n",
    "    # Step 1: Load data into memory\n",
    "    with open(legislators_file, \"r\") as f:\n",
    "        legislators = list(csv.DictReader(f))\n",
    "    with open(sponsorships_file, \"r\") as f:\n",
    "        sponsorships = list(csv.DictReader(f))\n",
    "    with open(bills_file, \"r\") as f:\n",
    "        bills = list(csv.DictReader(f))\n",
    "\n",
    "    # Filter for the target name and get `person_id`\n",
    "    person_ids = {leg[\"person_id\"] for leg in legislators if leg[\"name\"] == target_name}\n",
    "\n",
    "    # Find all `identifier`s for the person_id\n",
    "    sponsored_bills = {spons[\"identifier\"] for spons in sponsorships if spons[\"person_id\"] in person_ids}\n",
    "\n",
    "    # Get the titles of the sponsored bills\n",
    "    result = [bill[\"title\"] for bill in bills if bill[\"identifier\"] in sponsored_bills]\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6037f4be-9df6-43a5-bc0e-28540fa9cea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# same implementation in Polars\n",
    "import polars as pl\n",
    "\n",
    "def find_bills_polars(bills_file, legislators_file, sponsorships_file, target_name):\n",
    "    # Polars has a way to load CSV files directly into DataFrames\n",
    "    legislators = pl.read_csv(legislators_file)\n",
    "    sponsorships = pl.read_csv(sponsorships_file)\n",
    "    bills = pl.read_csv(bills_file)\n",
    "\n",
    "    # Filter for the target name and get `person_id`\n",
    "    person_ids = legislators.filter(pl.col(\"name\") == target_name).select(\"person_id\")\n",
    "\n",
    "    # Find all `identifier`s for the person_id\n",
    "    sponsored_bills = sponsorships.join(person_ids, on=\"person_id\", how=\"inner\").select(\"identifier\")\n",
    "\n",
    "    # Get the titles of the sponsored bills\n",
    "    result = bills.join(sponsored_bills, on=\"identifier\", how=\"inner\").select(\"title\")\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "19fea8cf-93ee-422b-b8c1-93f868c1a1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# and if you're curious, here's pandas\n",
    "def find_bills_pandas(bills_file, legislators_file, sponsorships_file, target_name):\n",
    "    legislators = pd.read_csv(legislators_file)\n",
    "    sponsorships = pd.read_csv(sponsorships_file)\n",
    "    bills = pd.read_csv(bills_file)\n",
    "\n",
    "    # Filter for the target name and get `person_id`\n",
    "    person_ids = legislators.loc[legislators[\"name\"] == target_name, \"person_id\"]\n",
    "\n",
    "    # Find all `identifier`s for the person_id\n",
    "    sponsored_bills = sponsorships.loc[sponsorships[\"person_id\"].isin(person_ids), \"identifier\"]\n",
    "\n",
    "    # Get the titles of the sponsored bills\n",
    "    result = bills.loc[bills[\"identifier\"].isin(sponsored_bills), \"title\"]\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6aa5a845-a38f-41c0-8be4-a2a10c184511",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 2)\n",
      "┌────────────────┬─────────────────────┐\n",
      "│ Implementation ┆ Execution Time (ms) │\n",
      "│ ---            ┆ ---                 │\n",
      "│ str            ┆ f64                 │\n",
      "╞════════════════╪═════════════════════╡\n",
      "│ Pure Python    ┆ 238.297975          │\n",
      "│ Pandas         ┆ 86.9603             │\n",
      "│ Polars         ┆ 10.461535           │\n",
      "└────────────────┴─────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import timeit\n",
    "import polars as pl\n",
    "import altair as alt\n",
    "\n",
    "BILLS_FILE = \"bills.csv\"\n",
    "LEGISLATORS_FILE = \"legislators.csv\"\n",
    "SPONSORSHIPS_FILE = \"sponsorships.csv\"\n",
    "TARGET_NAME = \"Mike Quigley\"\n",
    "\n",
    "def avg_time(func, *args, iterations=10):\n",
    "    return timeit.timeit(lambda: func(*args), number=iterations) / iterations\n",
    "\n",
    "# time each function\n",
    "ITERATIONS = 20\n",
    "python_time = avg_time(find_bills_pure_python, BILLS_FILE, LEGISLATORS_FILE, SPONSORSHIPS_FILE, TARGET_NAME, iterations=ITERATIONS)\n",
    "pandas_time = avg_time(find_bills_pandas, BILLS_FILE, LEGISLATORS_FILE, SPONSORSHIPS_FILE, TARGET_NAME, iterations=ITERATIONS)\n",
    "polars_time = avg_time(find_bills_polars, BILLS_FILE, LEGISLATORS_FILE, SPONSORSHIPS_FILE, TARGET_NAME, iterations=ITERATIONS)\n",
    "\n",
    "# Store results in a Polars DataFrame\n",
    "results = pl.DataFrame({\n",
    "    \"Implementation\": [\"Pure Python\", \"Pandas\", \"Polars\"],\n",
    "    \"Execution Time (ms)\": [python_time * 1000, pandas_time * 1000, polars_time * 1000]\n",
    "})\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "17c75df4-4d7b-45b5-be21-56fd80f81ff6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-7b3877bef0154bc3afbd3485f825bc8f.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-7b3877bef0154bc3afbd3485f825bc8f.vega-embed details,\n",
       "  #altair-viz-7b3877bef0154bc3afbd3485f825bc8f.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-7b3877bef0154bc3afbd3485f825bc8f\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-7b3877bef0154bc3afbd3485f825bc8f\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-7b3877bef0154bc3afbd3485f825bc8f\");\n",
       "    }\n",
       "\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.20.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      let deps = [\"vega-embed\"];\n",
       "      require(deps, displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.20.1\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-c85f1199b30bf7141927b86fd4883333\"}, \"mark\": {\"type\": \"bar\"}, \"encoding\": {\"color\": {\"field\": \"Implementation\", \"legend\": null, \"type\": \"nominal\"}, \"x\": {\"field\": \"Execution Time (ms)\", \"title\": \"Execution Time (ms)\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"Implementation\", \"sort\": null, \"type\": \"nominal\"}}, \"height\": 300, \"title\": {\"text\": \"Function Timing\", \"subtitle\": \"Averaged over 20 Runs - Lower is Better\"}, \"width\": 500, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.20.1.json\", \"datasets\": {\"data-c85f1199b30bf7141927b86fd4883333\": [{\"Implementation\": \"Pure Python\", \"Execution Time (ms)\": 238.29797500002314}, {\"Implementation\": \"Pandas\", \"Execution Time (ms)\": 86.96029999991879}, {\"Implementation\": \"Polars\", \"Execution Time (ms)\": 10.46153540009982}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "chart = alt.Chart(results).mark_bar().encode(\n",
    "    y=alt.Y(\"Implementation\", sort=None),\n",
    "    x=alt.X(\"Execution Time (ms)\", title=\"Execution Time (ms)\"),\n",
    "    color=alt.Color(\"Implementation\", legend=None)\n",
    ").properties(\n",
    "    title=alt.Title(\"Function Timing\", subtitle=f\"Averaged over {ITERATIONS} Runs - Lower is Better\"),\n",
    "    width=500,\n",
    "    height=300\n",
    ")\n",
    "chart.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d7ec6a-2645-48f8-8a4e-68bf778c3cef",
   "metadata": {},
   "source": [
    "## Improving Performance\n",
    "\n",
    "1) Understand what is slow.  Use `timeit` or `cProfile` to profile your code.\n",
    "\n",
    "- https://docs.python.org/3/library/timeit.html\n",
    "- https://docs.python.org/3/library/profile.html\n",
    "\n",
    "_\"Premature optimization is the root of all evil.\"_\n",
    "\n",
    "2) Can the **critical path** be done in a different way?  (Minimize operations, use appropriate data structures, etc.)\n",
    "\n",
    "3) Can it be vectorized?  (Use NumPy, ufuncs, etc.)\n",
    "\n",
    "- https://numpy.org/doc/stable/reference/ufuncs.html\n",
    "\n",
    "4) Can it be parallelized?  (Use `multiprocessing`, `asyncio`, etc.)\n",
    "\n",
    "- https://docs.python.org/3/library/multiprocessing.html\n",
    "- https://docs.python.org/3/library/asyncio.html\n",
    "\n",
    "5) Consider using a bridge to a faster language (Cython, PyO3, CFFI, etc.)\n",
    "\n",
    "- Cython - https://cython.org/\n",
    "- PyO3 - https://pyo3.rs/\n",
    "- CFFI - https://cffi.readthedocs.io/en/latest/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11a581ed-b77f-475f-b2af-220552f018ce",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
