Python Poetry – All in the Same Package

27.01.22

For a Python developer, it is common to search a package library (usually Python Package Index, PyPi) for ready-made modules. If you need a simple string library or a whole machine learning system, PyPi most likely has it.

Loading libraries for Python is easy with a tool called pip. All you need is a library name. Over time this can be a problem, if your project grows and matures to a point, where you should start managing version numbers and dependencies. This will get quickly quite complex, even impossible, if you are working on multiple projects that require different versions of packages or even different versions of Python, not to speak of packaging the final product for customers…

At first glance, Poetry looks only like a tool for packaging and publishing libraries to PyPi. However, under the hood there is an excellent system that can be utilized on any Python project. Especially, if the project requires a tight control over libraries and versions, for example because of regulations of a target industry, can Poetry easily define versions and repositories explicitly and in reproducible way.

Poetry offers the following features in a single package:

defining and managing dependencies
defining private library repositories
isolated environment for Python and libraries
user-friendly command line tools
tools for building and publishing libraries

In this article we will learn about items 1, 3 and 4 by using an example project. Items 2 and 5 have a short introduction below.

Item 2: It might be required to have a private repository for a development project because of the security or regulatory reasons. And it’s possible that using public repositories is prohibited altogether. With Poetry it’s possible to configure private repository access, for example to Artifactory, and even prevent using any libraries from PyPi. This will ensure that a development project is using only approved libraries and there is no way to accidentally download anything from the Internet.

Item 5: Poetry offers two commands, build and publish. These will package and publish a Poetry project either to PyPi repository or a user-defined repository. These commands are there to simplify library development work by combining all routine operations together. Like in Item 2, publish command can be configured to upload libraries to a private repository, so the command will work also inside organizations private infrastructure.

Installation and use

Short installation instructions:

Install Python 3.9
Install Poetry using the official installer that can be downloaded from here
Follow the instructions by the installer and add Poetry to the PATH environment variable

The most important file of a Poetry project is pyproject.toml. It defines all the basic information about the project and has definitions for allowed Python versions and defines all library dependencies. Also, this file allows extending Poetry by defining you own commands.

Before we start building our own example pyproject.toml file, I must point out one important limitation. Poetry is designed as a Python-agnostic tool. The design philosophy is that regardless of the installed version of the Python interpreter, Poetry just works. Unfortunately, this might lead to think that Poetry also will take care of the installation of different Python versions, like it does with libraries. This is not the case. Poetry leaves installation of different Python versions to the user. If there are several Python versions installed on the system, Poetry can automatically manage linking to the proper version inside the project.

Initializing Poetry for a new project can be done in one of two ways: either by creating a new project template or adding a configuration file to pre-existing project.

Running the command poetry new ExampleProject, Poetry will create a new folder, ExampleProject, which has a default folder structure and a configuration file. This is the fastest way to create a template that is suitable for a publishable Python library. Poetry will prepare everything automatically for you, including library version numbers and unit test templates.

Alternatively, you can move to the folder that already has a Python project. In this folder, running the command poetry init will start a wizard that asks few questions about the project and then creates a pyproject.toml file. This configuration file is the only thing that will be created.

Project: init command

Example Project

In this section we will go through step-by-step creating a pyproject.toml file. You can create the file either by hand or by running the poetry init command.

The first part of the project file, tool.poetry, will define at least the following project information: name, version, a short description and author names.

[tool.poetry]
name = "PoetryDemo"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

Next, in the tool.poetry.dependencies section, we will define Python version for the project. In this section we will later add a list of all dependencies that we normally would install with pip command.

[tool.poetry.dependencies]
python = "^3.9"

Finally, we will define all the requirements for Poetry itself:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

If you want to test a self-made project file, there is the poetry check command that will check that the pyproject.toml file is correctly made.

Let’s add Robot Framework library to the project. You can either add it using the command add or by adding the line by hand to the pyproject.toml file:

[tool.poetry.dependencies]
python = "^3.9"
robotframework = "^4.1.3"

If you modify the project file by hand, you must apply the changes by using the poetry install command.

If you are using the Poetry command: poetry add robotframework@^4.1.3, the install command will be executed automatically.

The most important command to run first, when Poetry project is used for example in a CI/CD pipeline, is the install command. It is a safe command to be executed every time automatically. The command checks for poetry.lock file and based on that, loads all dependencies always using the same versions. If the poetry.lock file is not yet created, or it is deleted, then the install command will read the pyproject.toml file. Based on that file, Poetry will create a new poetry.lock file while installing the libraries.

Now we have a usable environment ready for running automated Robot tests. If you want to see, how the world looks like inside this environment, you can jump in using the command poetry shell. The command starts a shell that has all project-specific Python tools, libraries and custom commands available.

Inside the shell, you can try the following commands to get to know the environment:

robot --version
pip list
which python (works on Linux and MacOS)
where python (works on Windows)

To exit the shell, use the command exit.

The shell command in Poetry is basically the same thing as “venv” in Python. The most important difference is that you can easily run commands from the outside of the virtual environment. This fits well for example in CI/CD environments.

In the CI/CD pipeline, it’s enough to define the following steps:

poetry install
poetry run robot -d robot-logs robot-tests

Poetry takes care of everything and the end results, test logs, will be in the robot-logs folder.

Scripting

Running single commands sequentially in larger projects is not very feasible, so Poetry provides a possibility to expand itself with Python scripting.

Let’s create a new file to the project folder: scripts.py:

import os
import shutil
def run_tests():
    shutil.rmtree("robot-logs", ignore_errors=True)
    os.system("robot -d robot-logs robot-tests")

Now, add a new section to the pyproject.toml file, called tool.poetry.scripts. We will define a new command, robottests, under this section:

[tool.poetry.scripts]
robottests = "scripts:run_tests"

When you run the poetry install command, Poetry will create a new command inside the virtual environment. This command will run the run_tests() function from the scripts.py file. This command can be executed either directly inside the shell virtual environment by calling robottests or from the outside of the environment by running poetry run robottests. Instructing users to always use the command robottests, we eliminate problems that might be caused by all the different ways of using the test environment.

Attention! If you created the project using the poetry init command, you have to add the following line to the [tool.poetry] section:

packages = [{include = "scripts"}]

This will add scripts folder to a module list used by Poetry, ensuring that the robottests = "scripts:run_tests" line will find the right Python file. When creating a project with the poetry new command, the whole project is defined as a library and the scripts folder will automatically be part of the module search path.

Finally, few words about the virtual environment itself. Poetry does a good job hiding the mechanics under the hood, so the file hierarchy of the virtual environment is not inside the project folder. This makes using a version control system easy. If, however, you want to know, where the Poetry environments are stored, you can ask to see the location by using the command poetry env list --full-path.

You can evaluate the suitability of Poetry in development work or with CI/CD pipelines by expanding the previous example. Extending Poetry with your own commands will open doors for building more complex environments that can be driven by using few easy-to-remember commands. And the whole package is kept together regardless of the platform, thanks to Poetry.