Stop using Pip, use Poetry Instead!

Why you should switch your Python package installation method!

Python and Poetry — Credit: Michelle Duong

Alright I admit this headline is a little click-baity, since technically Poetry uses pip to install packages anyways, regardless, today I want to talk to you about Poetry (the Python kind, not the literature kind), and why you should be using it instead of other systems.

This can be broken down into three major components:

1. Ease of use
2. Dependency resolver
3. Goodies (this will be the most lengthy)

Ease of use

Poetry is super simple to use and install! If you haven’t heard about it before, here is the website. We have two installation methods (assuming you are on a “-nix”-based system:

1. Utilize curl and pipe the install:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

1. Utilize pip (the irony):

pip install poetry

Each of these methods has their benefits. First of all, on your system, I would (as do the docs/devs) highly recommend using method 1. However, method 2 shines in containers which we will talk about later in “goodies”.

Now that we have poetry installed and it is added to our PATH, assuming you used method 1, we can invoke it by simply typing poetry --version and we should see the version we have installed.

* n.b. If your version is < 1.0 there may be breaking changes so please update.

With all that out of the way, poetry usage is as straightforward as pip. We can simply poetry install <package-name> to install our packages. We can also poetry add <package-name> if we want to add packages and then poetry install to install them.

Poetry stores our packages in a pyproject.toml file that we can look at to see our dependencies, add authorship, and a lot of other useful information. It also has a poetry.lock file which contains our locked dependency versions (should we choose to lock them).

Not impressed yet? Fair enough, let’s keep going.

Dependency Resolver

Poetry’s dependency resolver far surpasses pip and conda in stability. Period. Performance can be slow, but especially given caching of dependencies on builds, I find this not to be an issue, or rather, the stability is worth the price of a few seconds.

Now let’s assume you have a python project already and you want to start using poetry. That is great! Looking at your project you can see you have three major dependencies: pydantic, fastapi, and uvicorn. (Nice API you have there :)). If you navigate to your project directory, with poetry installed, and run poetry init you will be put through a series of interactive prompts to declare you dependencies! These prompts include production vs development dependencies, specific versions of a package, and more. Below is a screenshot of a sample where I wanted to install pydantic:

Furthermore, you can interactively declare your development dependencies upon project initialization as well. Or, later on you can add them using poetry add <package-name> --dev

Poetry will take your package, version constraints, and resolve it against the other packages you have installed and save it in a nice .toml file. Enough said? I think so. Moving on.

Poetry Goodies

Poetry comes with a few great benefits and “goodies” as I will call them. Let’s talk about my five favorite:

1. Interactive dependency prompt

This was talked about in the section above, but is worth repeating. While you can declare dependencies individually, or add them into the pyproject.toml file, the interactive dependency system is fantastic and a welcomed new approach for me, but may not be preferred for everyone.

2. Project initialization

You can create an entirely new python project using poetry. Simply run poetry new <project-name>. This was great for getting started as it includes a project folder, tests, etc. See below for source directly from the Poetry documentation:

Great stuff!

3. Built in virtual environment

Poetry comes with a built in virtual environment for all of your projects. In your project’s directory, any standard command can be prefaced by poetry run <command> to run inside the poetry environment. For example, poetry run uvicorn main:app --reload to start my FastAPI unicorn server on hot reload. See here for more information on environment management in poetry. This practice effectively eliminates the need for conda environments (sort of, see below) and virtual environments using virtualenv or venv. I should also note that while poetry uses pyproject.toml and poetry.lock files to manage dependencies, these can be exported to requirements.txt (which some systems require for deployment) using poetry export --output requirements.txt

4. Publishing Packages

Poetry completely revolutionizes the python packaging process. Instead of worrying about fine details of publishing onto PyPI, Poetry handles all of this for us and publishes to PyPI by default, but can also publish to other/private repositories. After configuring username and password, a simple poetry publish --build command will build your package and publish it to PyPI! Amazing! See here for more packaging configuration details.

5. Container friendliness

This one may not apply to everyone, and may be controversial since some people have their preferred container workflow, BUT, if you use containers, especially for data science work (which you should), Poetry can help. Containers already isolate our python version and give us a clean environment, so we don’t have to worry too much about virtual environments. However, I find install packages using poetry exceptionally better than standard pip installs (let alone conda inside a container).

When using poetry inside containers I recommend installing from pip (I know, I said not to, but I prefer this over curl-ing and piping something into my production container). I do recommend locking the poetry version: pip install poetry=1.1.4 that way you have stability there. I then disable the virtualenv creation (since we don’t need it inside containers) by running poetry config virtualenvs.create false after the poetry install. Then the container can copy our two poetry files (poetry.lock and pyproject.toml) and run poetry install to install our locked dependencies just as we had them! Furthermore, we now no longer need to preface our run commands with poetry run since we disabled the virtual environment!

Wrapping Up

I hope this guide was helpful to you and I hope you start using Poetry soon! I also wanted to take a second and share my general setup for new python projects since I think it might be helpful.

• Environment/python manager: conda (Miniconda)

This is for two reasons, one I had trouble installing Pyenv and never really got back around to it, but the second is sometimes, especially for data science work, it is nice to have access to the conda ecosystem. Mostly though, I use conda to isolate python versions.

*Edit: Since migrating to an M1 MacBook Air I have started using the universal binary Python 3.9 for all of my projects so I only use conda for specifically conda-related package installations.

*Edit 2: numpy doesn’t compile on M1 yet so went to using Conda’s x86 python3.9 for now (regarding projects that require numpy, otherwise still using the universal 3.9 build

• Dependency manger/resolver: poetry
• Preferred python version: 3.9 (type hints, dict improvements)

By using conda to install my python versions, I know poetry will use the active conda environment python which makes switching a little simpler, and then I leave the dependencies to poetry itself and run most commands through poetry run.

Hope this was helpful! Have a great day!