299 lines
11 KiB
Plaintext
299 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b4869e5b-a600-4285-8879-5053a13d6e6f",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Python Ecosystem\n",
|
|
"\n",
|
|
"One of the reason's for Python's early success was its \"batteries included\" philosophy.\n",
|
|
"\n",
|
|
"If you remember being a kid and getting a new toy, only to have to purchase (or charge) its batteries before playing-- the analogy here is that Python comes with everything you need to play (be productive) out of the box.\n",
|
|
"\n",
|
|
"We've seen this with some built in modules:\n",
|
|
"\n",
|
|
"- datetime\n",
|
|
"- math\n",
|
|
"- csv\n",
|
|
"- json\n",
|
|
"- functools\n",
|
|
"- itertools\n",
|
|
"- collections\n",
|
|
"- abc\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e4385a75-4d33-4de2-a840-006dc06fdc7b",
|
|
"metadata": {},
|
|
"source": [
|
|
"The standard library contains tools for common (and less-common) file types, data structures, dealing with data from the web, a simple UI toolkit, and much more:\n",
|
|
"\n",
|
|
"<https://docs.python.org/3/library/index.html>\n",
|
|
"\n",
|
|
"These modules are written in Python or C, and distributed with Python itself. \n",
|
|
"\n",
|
|
"The standard library is great for many things but in practice it has a few shortcomings:\n",
|
|
"\n",
|
|
"- can't possibly include *everything*\n",
|
|
"- typically only one way of doing things, and if an alternate approach is needed, an alternate library should be (example: streaming JSON)\n",
|
|
"- once a package is in standard library it is in effect *frozen* due to Python's strict backwards compatibility rules"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b3758d4b-2b85-4d69-80c1-160d16bd4ba3",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Beyond the Standard Library\n",
|
|
"\n",
|
|
"It is helpful to have a wider ecosystem that all Python developers can share-- enter the Python Package Index or PyPI (typically pronounced pie-pee-eye to avoid confusing with PyPy pronounced pie-pie).\n",
|
|
"\n",
|
|
"https://pypi.org\n",
|
|
"\n",
|
|
"Over half a million projects released on PyPI, with billions of downloads a day.\n",
|
|
"\n",
|
|
"We've seen packages that come from PyPI:\n",
|
|
"\n",
|
|
"- rich\n",
|
|
"- httpx\n",
|
|
"- pandas\n",
|
|
"- polars\n",
|
|
"- networkx\n",
|
|
"- matplotlib\n",
|
|
"- altair\n",
|
|
"- Pillow\n",
|
|
"- Flask\n",
|
|
"- Django"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4a25d8b-d3c0-462b-a87a-ee34b98e8663",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Important Note: Anyone can publish to PyPI\n",
|
|
"\n",
|
|
"Installing a package means trusting its authors to some extent. \n",
|
|
"\n",
|
|
"Packages I ask you to install are packages I trust, and which have established reputations & teams.\n",
|
|
"\n",
|
|
"If you find a small package you should be mindful of the fact that letting someone run code on your system means letting that person do whatever their code does. \n",
|
|
"\n",
|
|
"If in doubt:\n",
|
|
"- look for signs of activity on GitHub/etc.: a popular library with dozens of tutorials is one thing -- an obscure library only one person used may also be fine, but worth a bit of vetting\n",
|
|
"- look at the source code!\n",
|
|
"- ask someone! (James, TAs, etc.)\n",
|
|
"\n",
|
|
"### Licensing\n",
|
|
"\n",
|
|
"Code that is published & open source comes with a license, a set of rules saying what you may and may not do with it. Typically this prohibits redistribution of the work without the license, but in some cases may mean that your own work needs to be open source to use it. **Using open source code without following the license is plagarism/theft and can come with serious consequences here and in any workplace since your employer would carry the legal burden.**\n",
|
|
"\n",
|
|
"Make sure that the code that you are using is under a license that allows you to use it in the environment that you are in. That isn't much of an issue here in class, but in companies you may not be able to use certain licenses. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "debb1e93-d3ce-4b54-ae45-0b470ba39242",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Python's Worst Flaw\n",
|
|
"\n",
|
|
"A key part of Python's general philosophy is that there should be an obvious way to do things. It is generally acknowledged that we have fallen short when it comes to package management. \n",
|
|
"\n",
|
|
"The situation is improving, and you have already been using the latest & greatest tool. But as you venture out into the wider world of packages, you'll notice some rough edges for sure.\n",
|
|
"\n",
|
|
"When you look at instructions, some libraries will tell you to:\n",
|
|
"\n",
|
|
"- `pip install matplotlib`\n",
|
|
"\n",
|
|
"You may also have used `conda` before, and done something similar.\n",
|
|
"\n",
|
|
"In this class, we've been using `uv`, and you may also encounter:\n",
|
|
"\n",
|
|
"- `pdm`\n",
|
|
"- `poetry`\n",
|
|
"- `pip-compile`\n",
|
|
"- `pipx`\n",
|
|
"- `pip-tools`\n",
|
|
"\n",
|
|
"And probably a dozen other solutions to installing & managing packages.\n",
|
|
"\n",
|
|
"*What's going on?*\n",
|
|
"\n",
|
|
"### How Python Packages Work\n",
|
|
"\n",
|
|
"Recall that python packages are just directories. When you install a package you are getting something that resembles:\n",
|
|
"\n",
|
|
"```\n",
|
|
"baking-pkg\n",
|
|
"├── baking\n",
|
|
"│ ├── __init__.py\n",
|
|
"│ ├── cli.py\n",
|
|
"│ ├── ingredients.py\n",
|
|
"│ ├── sizes.py\n",
|
|
"│ ├── units.py\n",
|
|
"│ └── utils.py\n",
|
|
"├── LICENSE\n",
|
|
"├── README.md\n",
|
|
"└── tests\n",
|
|
" ├── test_baking.py\n",
|
|
" ├── test_units.py\n",
|
|
" └── test_utils.py\n",
|
|
"```\n",
|
|
"\n",
|
|
"If I gave you these files and you put them on your desktop, you would find that you could only use them if you were in the same directory. That is if you put your own files in `baking-pkg` you would be able to `import baking` but otherwise it would fail.\n",
|
|
"\n",
|
|
"When you type an `import` statement, Python uses an **environment variable** a variable set at the operating system level, not specific to Python named `PYTHONPATH` to look up what directories it should search.\n",
|
|
"\n",
|
|
"This variable contains a list of directories, which might resemble:\n",
|
|
"\n",
|
|
"- /home/user/james/.poetry/global/\n",
|
|
"- /usr/lib/python3.10\n",
|
|
"- /usr/lib/python3.10/lib-dynload\n",
|
|
"- /usr/local/lib/python3.10/dist-packages\n",
|
|
"- /usr/lib/python3/dist-packages\n",
|
|
"- /home/user/james/.pipx/ruff/\n",
|
|
"- /home/user/james/.pipx/jupyter/\n",
|
|
"\n",
|
|
"These directories are searched in order, so if I have a `math` library in `/home/user/james/.poetry/global/` it will supercede the built in math library in `/usr/lib/python3.10/`.\n",
|
|
"\n",
|
|
"This means that can only have one library of a given name that is importable."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dbf0e9aa-3c9a-457a-a2c6-6fdde95ca2cb",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Version Conflicts\n",
|
|
"\n",
|
|
"In practice though, you might need to have multiple copies of a library installed on your system, imagine the following:\n",
|
|
"\n",
|
|
"\n",
|
|
"**Project 1 (MPCS 51042)**\n",
|
|
"- polars==1.14\n",
|
|
"- seaborn==2.0\n",
|
|
" - matplotlib==3.0\n",
|
|
"\n",
|
|
"\n",
|
|
"**Project 2 (MPCS 52000)**\n",
|
|
"- polars==0.44\n",
|
|
"- matplotlib==2.0\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"**Project 3 (Work)**\n",
|
|
"- altair==5.0\n",
|
|
"- customlibrary\n",
|
|
" - matplotlib==2.4\n",
|
|
" \n",
|
|
"Your system would need 3 versions of `matplotlib` to be able to correctly run the code in question."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5e99e59c-f19f-4312-b842-6862eacc64a7",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Installing Packages (the wrong way)\n",
|
|
"\n",
|
|
"If you use `pip` on its own, each time you install a set of packages, it will uninstall conflicts & replace the version with the latest.\n",
|
|
"\n",
|
|
"By default `pip` installs to the `/usr/lib/python3.10/dist-packages/` directory (or equivalent), but wherever it installs on your PYTHONPATH would have the same issue. We can only have one of a package installed at a time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9a69a927-dffb-46e3-9e86-1e09fbe013bd",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Virtualenvs\n",
|
|
"\n",
|
|
"For this reason, Python has the concept of `virtualenvs`. These are directories of Python packages that are meant to only be added to PYTHONPATH when a given project is being used.\n",
|
|
"\n",
|
|
"If we installed all of the packages for our three projects to three different directories:\n",
|
|
"\n",
|
|
"```\n",
|
|
".\n",
|
|
"├── proj1-venv\n",
|
|
"│ ├── matplotlib # v3.0\n",
|
|
"│ ├── polars\n",
|
|
"│ └── seaborn\n",
|
|
"├── proj2-venv\n",
|
|
"│ ├── matplotlib # v2.0\n",
|
|
"│ └── polars\n",
|
|
"└── proj3-venv\n",
|
|
" ├── altair\n",
|
|
" ├── customlibrary\n",
|
|
" └── matplotlib # v2.4\n",
|
|
"```\n",
|
|
"\n",
|
|
"Now we can add the correct `venv` directory to our PYTHONPATH before running the appropriate command.\n",
|
|
"\n",
|
|
"This is what `venv` does for us:\n",
|
|
"<https://docs.python.org/3/library/venv.html>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9abe42e4-3a2a-417d-996b-e25fe0db2ce6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## `uv` (and other package managers)\n",
|
|
"\n",
|
|
"In practice, managing a venv can be tedious and error-prone, which is why a suite of tools exists to help manage them for you.\n",
|
|
"\n",
|
|
"In this class we have opted for `uv`, among the newest of these tools which (IMO) is the easiest to use yet.\n",
|
|
"\n",
|
|
"`uv` demo:\n",
|
|
"- uv creates `.venv`\n",
|
|
"- `uv add` updates lockfile and venv\n",
|
|
"- `uv run` ensures that the correct venv is activated before Python starts\n",
|
|
"\n",
|
|
"https://mpcs51042.netlify.app/guides/uv/"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "85c23bfc-d1b5-4941-acf2-f415fa2f7a9d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Recommendation: Always start new projects with `uv init` (or similar!)\n",
|
|
"\n",
|
|
"Never use `pip` or `venv` directly!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1be81171-4c6c-432d-8a7e-d39d4bf601ef",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|