Here we will learn how to upload a Python open-source package to PyPI or the Python Packaging Index in this post. You must first be conscious of the PyPI for that.
It’s a Python programming language software repository. With just a few simple commands, you can download and install all the software and libraries developed by the Python community. It is a platform used to distribute their software by package authors.
Preparing Your Package for Publication
Here we are going to assume that you have a package ready for publishing. To upload this package to PyPI, you need to follow these steps:
1. Naming
This is the first and the most difficult step. You need to find a good, unique name for the package. As there are more than 150,000 packages uploaded on PyPI, it is quite possible that your first choice would be already taken. Research and brainstorm a few names till you find one that you like and suit the package.
To check if a name has been taken or not, you can use the PyPI search. If a package exists with the same name, its name will come up. Make sure that the name of the package is a bit descriptive so that others can know exactly what the package is about. Let’s name our package package1. You need to use this name while installing the package using pip.
$ pip install package1
You can use different names for importing the package and uploading it on PyPI. It is encouraged to use similar names for both, so that it is easy for users.
Download Detailed Brochure and Get Complimentary access to Live Online Demo Class with Industry Expert.
2. Configuring
The next step is to configure your package. Before you can upload your package to PyPI, it is important that you provide some information about the package.
Usually, this information is presented in the form of a setup.py file. To simplify this information to help users understand better, there are several initiatives. However, the only completely supported way of providing package’s information is through a setup.py file. This file should be in your package’s top folder. Typically, a setup.py file looks like this:
import pathlib from setuptools import setup # The directory containing this file HERE = pathlib.Path(__file__).parent # The text of the README file README = (HERE / "README.md").read_text() # This call to setup() does all the work setup( name="package1", version="1.0.0", description="description about package1", long_description=README, long_description_content_type="text/markdown", url="github url", author="author of the package", author_email="email address of the author", license="MIT", classifiers=[ "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", ], packages=["package1"], include_package_data=True, install_requires=["feedparser", "html2text"], entry_points={ "console_scripts": [ "package1.__main__:main", ] }, )
The most important and absolutely essential parameters required for calling the setup() are:
- Name: package’s name as you want on the PyPI
- Version: package’s current version
- Packages: All the packages and subpackages present in the source code.
There might be many sub-packages present in a complicated project. For simplifying this process, there is a find_packages() function used in setuptools which can discover all the sub-packages present in the source code. Here is how it is done:
from setuptools import find_packages, setup setup( ... packages=find_packages(exclude=("tests",)), ... )
To find packages on PyPI, all you need is its name, version, and packages. However, if you add some information about the package, it will be easier to find it. All this information comes from README.md and setup.py. The setup()’s last two parameters are:
⇒ install_requires – This is used for listing all the third party libraries dependencies present in the package.
⇒ entry_points – This is used for creating the scripts that will be calling a function from inside your package.
3. Documenting
You need to add some documentation before you can release your package on PyPI. If your package is simple, all you will need is a README file as documentation. However, in the case of large packages like API reference or galleries, the documentation can be as large as a web page filled with tutorials.
A good README file contains description of the project and how you can install and use the package. In the long_description argument of the setup(), you need to include your README for displaying your README on PyPI.
For package documentation, PyPI supports reStructuredText and Markdown. You need to use the long_description_content_type parameter of the setup() for telling the PyPT the format you are using. The valid values include text/plain, text/markdown, text/x-rst.
Now, if you are working on a big, complicated project, your documentation will be large enough to not fit in a single README file. In such cases you need to use the url parameter for linking the documentation. You can use websites like Read the Docs and GitHub for this purpose.
4. Versioning
Every package needs a version and for every version you are allowed to make a single upload. So, for updating the package on PyPI, you have to also increase the number of the version. This promotes reproducibility as two systems with the same package’s version should have the same behavior.
For deciding your version number, there are a number of different schemes available. You should use a simple versioning scheme for a simple project. For a complicated project, you can use recommendations of PEP 440.
Semantic versioning is one such default versioning scheme that you can use. In this, three numerical components are used for the version number names as MAJOR, MINOR, and PATCH respectively. There are some rules you need to follow for increasing each component.
⇒ For incompatible API changes, increase the MAJOR version.
⇒ For added functionality in a backwards-compatible manner, increase the MINOR version.
⇒ For backwards-compatible bug fixes, increase the PATCH version.
There are different files inside the project where you might have to specify the package’s version. Bumpversion is a tool for keeping sure that there is a consistency in the version number. For installing Bumpversion from PyPI:
$ pip install bumpversion
For increasing the MINOR version, you need to do something like this:
$ bumpversion –current-version 1.0.0 minor setup.py package1/__init__.py
This will change the version from 1.0.0 to 1.1.0.
5. Adding files
In some cases, you might have to add files other than source code files to your package. These may be binaries, data files, configuration files, and documentation.
You need to use a manifest file for telling the setup() to include the files. In some projects, the setup() is used for creating a manifest that has the README and all the code files. For making changes to the manifest, you need to create a template with the name MANIFEST.in. All the rules explaining what should and should not be included is present in this file.
include package1/*.txt
This will include all the text files present in the package1 directory.
Apart from MANIFEST.in, you also have to tell setup() to copy the non-code files. For this, you need to set the argument, include_package_data, to True. This argument is used to control if non-code files will be copied during the installation of the package.
setup( ... include_package_data=True, ... )
Publishing to PyPI
Now that your package is ready for uploading, you need to know how you can actually upload the package to PyPI. The first step in this is to create an account on PyPI. Also register on TestPyPI. You can use this for trying out all the steps involved in publishing a package without making a mess.
For uploading the package to PyPI, you need to use a tool named Twine. For installing Twine through Pip, use the following:
$ pip install twine
1. Building the Package
On PyPI, the packages are wrapped as distribution packages, not plain source code. Python wheels and source archives are some of the common formats used for distribution packages.
Your source code as well as all the supporting files are wrapped into a single tar file. This is known as source archive. You can call wheel a zip archive that has your code.
The difference between source archive and wheel is that when also contains all the ready-to-use extensions. For creating a wheel and source archive for your package, you need to use the following command:
$ Python setup.py sdist bdist_wheel
Two files will be created in a new directory with the name dist, a wheel and a source archive:
Package1/ │ └── dist/ ├── package1-1.0.0-py3-none-any.whl └── package1-1.0.0.tar.gz You can use the –help-commands option for listing all the command line arguments available. Here is an example: $ Python setup.py --help-commands Standard commands: build build everything needed to install build_py "build" pure Python modules (copy to build directory) build_ext build C/C++ and Cython extensions (compile/link to build directory) < ... many more commands ...> For getting information regarding a single, specific command, you can use, Python setup.py sdist --help.
2. Testing the Package
The first step is checking whether all the files are present in the newly created distribution package. All the supporting files and sub-packages must also be included. You can even unzip the wheel like a zip file.
Newer versions of Twine can be used for checking whether the package description will be properly rendered on PyPI or not. Here is how you can do it:
$ twine check dist/*
Checking distribution dist/package1-1.0.0-py3-none-any.whl: Passed
Checking distribution dist/package1-1.0.0.tar.gz: Passed
It lets you know whether you have the wrong content type or not. It can’t catch all the issues that you might face.
3. Uploading the Package
You will be using the Twine tool for uploading the package. You should use the TestPyPI first for making sure that everything is working as expected. Try the following command:
$ twine upload –repository-url https://test.pypi.org/legacy/ dist/*
You will have to provide your username and password.
Congratulations! Your package has now been uploaded and published on PyPI.
Once you have uploaded the package to PyPI, you can use pip to install it as well. Use the following command:
$ pip install your-package-name
Other Useful Tools
When you are creating and uploading Python packages, there are a few tools that you should know about:
1. Virtual Environments
When you are working on different projects and on different versions of Python, its own dependencies and requirements, you need virtual environments.
There are a number of guides that you can use to understand how virtual environments work. Here are a few examples
(i) Managing Python Dependencies With Pip and Virtual Environments
(ii) Pipenv: A Guide to the New Python Packaging Tool
(iii) Python Virtual Environments: A Primer
Before you upload your package to PyPI, you need to use the minimal virtual environment for testing your package and for making sure that your setup.py file contains all the necessary dependencies.
2. Cookiecutter
Cookiecutter is a great tool for getting started with your project. First, it will ask a few questions on the basis of the template for setting up your project. There are a number of templates to choose from.
To make this work, the first step is to make sure that you have properly installed Cookiecutter. You can directly install it from PyPI using the following command:
$ pip install cookiecutter
After you have answered all the questions, your project is set up by the Cookiecutter. Here is an example showing how the template creates the directories and files:
package1/ │ ├── package1/ │ └── __init__.py │ ├── tests/ │ ├── __init__.py │ └── test_sample.py │ ├── README.rst ├── setup.py └── tox.ini
You can find a list of all the cookiecutters available and tutorials on using them for creating your own template in the Cookiecutter’s documentation.
3. Flit
When it comes to packaging in Python, the whole process can get quite messy. Many developers have criticized the usage of setup.py, an executable file, for configuring the information which is not considered ideal.
There is an alternative defined by PEP 518 which is pyproject.toml. The TOML file format with a simple configuration. It is flexible, can be used easily, stems away from a standard and is not overly complicated. Even though it has been out for a few years, many standard tools do not yet fully support the pyproject.toml configuration.
However, there are a few tools that use the pyproject.toml for publishing the package to PyPI. Flit is one such tool. It cannot support uploading advanced packages like the ones with C extensions but it can publish simple packages of Python easily.
For the pip installation of flit, you can use the following command:
$ flit init Module name [reader]: Author []: Name of author Author email []: Email address of author Home page []: Address of github repository
Choose a license (see http://choosealicense.com/ for more info)
- MIT – simple and permissive
- Apache – explicitly grants patent rights
- GPL – ensures that code based on this is shared with the same terms
- Skip – choose a license later
Enter 1-4 [1]:
Written pyproject.toml; edit that file to add optional extra info.
The flit int command that you used is for creating the pyproject.toml file on the basis of the answers to the questions that you provided. In some cases, a little editing might be required in the file before it is ready to be used. For example, for the Python1 project, the Flit’s pyproject.toml file will look something like this:
[build-system] requires = ["flit"] build-backend = "flit.buildapi" [tool.flit.metadata] module = "package1" dist-name = "package1" description-file = "README.md" author = "Author’s name" author-email = "Author’s email address" home-page = "Address of the github repository" classifiers = [ "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", ] requires-Python = ">=3.7" requires = ["feedparser", "html2text"] [tool.flit.scripts] Package1 = "package1.__main__:main"
As you can see, there are a lot of different items than the setup.py file. Also, description and version is missing as Flit is capable of figuring this out itself as the docstring and __version__ defines in __init__.py file. To understand everything about the pyproject.toml file, you can go through the Flit’s documentation.
You can also use Flit for building and publishing your package to PyPI. For building purposes, you need to use the following command:
$ flit build
This results in the creation of a wheel and a source archive similar to one built in Python setup.py sdist bdist_wheel. For uploading the package to the PyPI platform, you have to use Twine or you can use Flit directly using the command:
$ flit publish
The publish command is used for building the package and uploading the files to PyPI. You will be asked to provide your username and password.
To know more about Flit, you can try one of the following:
⇒ 2 minute lightning talk from EuroSciPy 2017
⇒ Brett Cannon’s tutorial on packaging up your Python code for PyPI
4. Poetry
This is another tool used for building and uploading Python packages. It’s working is quite similar to that of Flit’s. The first step is installing Poetry. You can just try pip install poetry for the job. However, some suggest that for avoiding dependency conflicts, you should use a custom installation script. You will be able to understand more about this in the documentation.
Once you have installed poetry, you can start the process with the init command
With Poetry installed, you start using it with an init command:
$ poetry init
This command will guide you through creating your pyproject.toml config.
Package name [code]: package1
Version [0.1.0]: 1.0.0
Description []: Description about package1
…
This command will lead to the creation of a pyproject.toml file by using the answers to the questions that were asked about the package. However, in flit and poetry, the specifications mentioned in the pyproject.toml file are different. For example, the pyproject.toml file for poetry will look something like this:
[tool.poetry] name = "package1" version = "1.0.0" description = "Description about package1" readme = "README.md" homepage = "Address of github repository" authors = ["name and email address of the author"] license = "MIT" packages = [{include = "package1"}] include = ["package1/*.txt"] [tool.poetry.dependencies] Python = ">=3.7" feedparser = ">=5.2" html2text = ">=2018.1" [tool.poetry.scripts] Package1 = "package1.__main__:main" [build-system] requires = ["poetry>=0.12"] build-backend = "poetry.masonry.api"
All the items mentioned in this file have already been discussed for the setup.py file. The difference is that Poetry adds classifiers on the basis of version and license of Python automatically. However, you need to be very explicit about the dependencies’ version that you are using.
Like Flit, you can use Poetry for building and uploading packages to the PyPI platform. Use the following command for creating a wheel and a source archive:
$ poetry build
This will lead to the creation of two files in the dist sub-directory which you will be uploading using Twine. For publishing the package on PyPI using Poetry, try the following command:
$ poetry publish
Apart from this, Poetry uses the new command for helping you start a new project. It can be used in virtual environments. Everything you need to know is mentioned in the Poetry’s documentation.
Now you are aware of the steps required for preparing and uploading your project to PyPI so that other people can install and use it. It is a great way of helping others through your creation. Here, we discussed the steps involved in naming, configuring, building, and uploading your package to PyPI. Also, you need to find out several tools that use the pyproject.toml file instead of setup.py file for publishing packages on PyPI platform. If you want more information in this matter, you can check out the Python Packaging Authority.
Final Thoughts
Python is the most preferred language among the developers. One of the primary reasons behind this is the standard library it offers. There are several modules available that help you in working with files, file paths, sockets, JSON, XML files, parsing CSV, etc. Needless to say the packages of Python in respect to the standard library are great.
But thanks to the huge open-source community of Python, there are several other projects available other than the standard library. Most of these packages are hosted on the PyPI platform, also known as the Cheese shop. Here, you can find a program as simple as Hello World to something involving advanced machine learning libraries.
If you also want to become a Python Programmer, enroll yourself in a Python Programming Course and enrol the coding world like a pro.
