DevOps in Python: Infrastructure as Python 2nd Edition Moshe Zadka - The newest ebook version is ready, download now to explore
DevOps in Python: Infrastructure as Python 2nd Edition Moshe Zadka - The newest ebook version is ready, download now to explore
com
https://ebookmeta.com/product/devops-in-python-
infrastructure-as-python-2nd-edition-moshe-zadka-2/
OR CLICK HERE
DOWLOAD EBOOK
Apress Standard
The publisher, the authors and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the
material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
Before you can use Python, you need to install it. Some operating
systems, such as macOS and some Linux variants, have Python
preinstalled. Those versions of Python, colloquially called system
Python, often make poor defaults for people who want to develop in
Python.
The version of Python installed is often behind the latest practices.
System integrators often patch Python in ways that can lead to
surprises. For example, Debian-based Python is often missing modules
like venv and ensurepip. macOS Python links against a Mac shim
around its native SSL library. Those things mean, especially when
starting and consuming FAQs and web resources, it is better to install
Python from scratch.
This chapter covers a few ways to do so and the pros and cons of
each.
1.1 OS Packages
Volunteers have built ready-to-install packages for some of the more
popular operating systems.
The most famous is the deadsnakes PPA (Personal Package
Archives). The dead in the name refers to the fact that those packages
are already built—with the metaphor that sources are “alive.” Those
packages are built for Ubuntu and usually support all the versions of
Ubuntu that are still supported upstream. Getting those packages is
done with a simple
$ PROJECT=https://github.com/pyenv/pyenv-installer
\
PATH=raw/master/bin/pyenv-installer \
curl -L $PROJECT/PATH | bash
Of course, this comes with its own security issues and could be
replaced with a two-step process where you can inspect the shell script
before running or even use git-checkout to pin to a specific revision.
export PATH="~/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
This allows pyenv to properly intercept all the necessary
commands.
pyenv separates the notion of installed interpreters from available
interpreters. Enter the following to install a version.
or locally by using
env PYTHON_CONFIGURE_OPTS="--enable-shared
--enable-optimizations
--with-computed-gotos
--with-lto
--enable-ipv6" pyenv
install
Let’s build a version that is pretty similar to binary versions from
python.org.
1.3 Building Python from Source
The main challenge in building Python from source is that, in some
sense, it is too forgiving. It is all too easy to build it with one of the built-
in modules disabled because its dependency was not detected. This is
why it is important to know which dependencies are fragile and how to
make sure a local installation is good.
The first fragile dependency is SSL. It is disabled by default and
must be enabled in Modules/Setup.dist. Carefully follow the
instructions there about the location of the OpenSSL library. If you have
installed OpenSSL via system packages, it is usually in /usr/. If you
have installed it from source, it is usually in /usr/local.
The most important thing is to know how to test for it. When
Python is done building, run ./python.exe -c 'import _ssl'.
That .exe is not a mistake; this is how the build process calls the newly
built executable, which is renamed to Python during installation. If this
succeeds, the SSL module was built correctly.
Another extension that can fail to build is SQLite. Since it is a built-
in, many third-party packages depend on it, even if you are not using it
yourself. This means a Python installation without the SQLite module is
pretty broken. Test it by running ./python.exe -c 'import
sqlite3'.
In a Debian-based system (such as Ubuntu), libsqlite3-dev is
required for this to succeed. In a Red Hat-based system (such as Fedora
or CentOS), libsqlite3-dev is required for this to succeed.
Next, check for _ctypes with ./python.exe -c 'import
_ctypes'. If this fails, likely, the libffi headers are not installed.
Finally, remember to run the built-in regression test suite after
building from source. This ensures that there have been no silly
mistakes while building the package.
1.4 PyPy
The usual implementation of Python is sometimes known as CPython to
distinguish it from the language proper. The most popular alternative
implementation is PyPy, a Python-based JIT implementation of Python
in Python. Because it has a dynamic JIT (just-in-time) compilation to
assembly, it can sometimes achieve phenomenal speed-ups (three times
or even ten times) over regular Python.
There are sometimes challenges in using PyPy. Many tools and
packages are tested only with CPython. However, sometimes spending
the effort to check if PyPy is compatible with the environment is worth
it if performance matters.
There are a few subtleties in installing Python from source. While it
is theoretically possible to translate using CPython, in practice, the
optimizations in PyPy mean that translating using PyPy works on more
reasonable machines. Even when installing from source, it is better to
first install a binary version to bootstrap.
The bootstrapping version should be PyPy, not PyPy3. PyPy is
written in the Python 2 dialect, which is one of the only cases where
worrying about the deprecation is irrelevant since PyPy is a Python 2
dialect interpreter. PyPy3 is the Python 3 dialect implementation,
which is usually better in production as most packages are slowly
dropping support for Python 2.
The latest PyPy3 supports 3.5 features of Python, as well as f-
strings. However, the latest async features, added in Python 3.6, do not
work.
1.5 Anaconda
The closest to a system Python that is still reasonable for use as a
development platform is Anaconda, a metadistribution. It is, in essence,
an operating system on top of the operating system. Anaconda has its
grounding in the scientific computing community, and so its Python
comes with easy-to-install modules for many scientific applications.
Many of these modules are non-trivial to install from PyPI, requiring a
complicated build environment.
It is possible to install multiple Anaconda environments on the
same machine. This is handy when needing different Python versions
or different versions of PyPI modules.
To bootstrap Anaconda, you can use the bash installer available at
https://conda.io/miniconda.html. The installer also modifies
~/.bash_profile to add the path to conda, the installer.
conda environments are created using conda create --name
<name> and activated using source conda activate <name>.
There is no easy way to use inactivated environments. It is possible to
create a conda environment while installing packages: conda create
--name some-name python. You can specify the version using = –
conda create --name some-name python=3.5. It is also
possible to install more packages into a conda environment, using
conda install package[=version], after the environment has
been activated. Anaconda has a lot of prebuilt Python packages,
especially ones that are non-trivial to build locally. This makes it a good
choice if those packages are important to your use case.
1.6 Summary
Running a Python program requires an interpreter installed on the
system. Depending on the operating system and the versions, there are
several different ways to install Python. Using the system Python is a
problematic option. On macOS and Unix systems, using pyenv is almost
always the preferred option. On Windows, using the prepackaged
installers from Python.org is often a good idea.
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2022
M. Zadka, DevOps in Python
https://doi.org/10.1007/978-1-4842-7996-0_2
2. Packaging
Moshe Zadka1
(1) Belmont, CA, USA
$ source /home/name/venvs/my-special-env/bin/activate
The last command tests that glom has been properly installed. Glom is
a package to handle deeply-nested data, and called with no arguments,
outputs an empty Python dictionary. This makes it handy for quickly
testing whether a new virtual environment can install new packages
properly.
Internally, pip is also treated as a third-party package. Upgrading pip
itself is done with pip install --upgrade pip.
Depending on how Python was installed, its real environment might or
might not be modifiable by the user. Many instructions in various README
files and blogs might encourage using sudo pip install. This is
almost always the wrong thing to do; it installs the packages in the global
environment.
The pip install command downloads and installs all dependencies.
However, it can fail to downgrade incompatible packages. It is always
possible to install explicit versions: pip install package-name==
<version> installs this precise version. This is also a good way for local
testing to get explicitly non-general-availability packages, such as release
candidates, beta, or similar.
If wheel is installed, pip builds, and usually caches, wheels for
packages. This is especially useful when dealing with a high virtual
environment churn since installing a cached wheel is a fast operation. This
is also highly useful when dealing with native or binary packages that need
to be compiled with a C compiler. A wheel cache eliminates the need to
build it again.
pip does allow uninstalling with pip uninstall <package>. This
command, by default, requires manual confirmation. Except for exotic
circumstances, this command is not used. If an unintended package has
snuck in, the usual response is to destroy the environment and rebuild it.
For similar reasons, pip install --upgrade <package> is not
often needed; the common response is to destroy and re-create the
environment. There is one situation where it is a good idea.
pip install supports a requirements file: pip install --
requirements or pip install -r. The requirements file simply has
one package per line. This is no different from specifying packages on the
command line. However, requirement files often specify strict
dependencies. A requirements file can be generated from an environment
with pip freeze.
Like most individual packages or wheels, installing anything that is not
strict and closed under requirements requires pip to decide which
dependencies to install. The general problem of dependency resolution
does not have an efficient and complete solution. Different strategies are
possible to approach such a solution.
The way pip resolves dependencies is by using backtracking. This
means that it optimistically tries to download the latest possible
requirements recursively. If a dependency conflict is found, it backtracks;
try a different option.
As an example, consider three packages.
top
middle
base
There are two base versions: 1.0 and 2.0. The package dependencies
are setup.cfg files.
The following is for the top.
[metadata]
name = top
version = 1.0
[options]
install_requires =
base
middle
[metadata]
name = middle
version = 1.0
[options]
install_requires =
base<2.0
The base package has two versions: 1.0 and 2.0. It does not have any
dependencies.
Because top depends directly on base, pre-backtracking versions of
pip get the latest and then have a failed resolution.
[build-system]
requires = [
"setuptools"
]
build-backend = "setuptools.build_meta"
Many systems can be used to build valid distributions. The
setuptools system, which used to be the only possibility, is now one of
several. However, it is still the most popular one.
Most of the rest of the data can be found in the project section.
[project]
name = "awesome_package"
version = "0.0.3"
description = "A pretty awesome package"
readme = "README.rst"
authors = [{name = "My Name",
email = "[email protected]"}]
dependencies = ["httpx"]
import tomlkit
import datetime
import os
import pathlib
import zoneinfo
now =
datetime.datetime.now(tz=zoneinfo.ZoneInfo("UTC"))
prefix=f"{now.year}.{now.month}."
pyproject = pathlib.Path("pyproject.toml")
data = tomlkit.loads(pyproject.read_text())
current = data["project"].get("version", "")
if current.startswith(prefix):
serial = int(current.split(".")[-1]) + 1
else:
serial = 0
version = prefix + str(serial)
data["project"]["version"] = version
pyproject.write_text(tomlkit.dumps(data))
Some utilities keep the version synchronized between several files; for
example, pyproject.toml and example_package/__init__.py.
The best way to use these utilities is by not needing to do it.
If example_package/__init__.py wants to expose the version
number, the best way is to calculate it using importlib.metadata.
# example_package/__init__.py
from importlib import metada
__version__ =
metadata.distribution("example_package").version
del metadata # Keep top-level namespace clean
[project.scripts]
example-command = "example_package.commands:main"
# example_package/commands.py
def main():
print("an example")
$ example-command
an example
— Vai niin, senkin veitikka, sitäkö sinä tahdoit!… Sinä luulit voivasi
sanoa minun pudonneen, senkin salakavala lurjus!
Loppu.
*** END OF THE PROJECT GUTENBERG EBOOK IHMISPETO:
SIVEYSROMAANI ***
Updated editions will replace the previous one—the old editions will
be renamed.
1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside the
United States, check the laws of your country in addition to the
terms of this agreement before downloading, copying, displaying,
performing, distributing or creating derivative works based on this
work or any other Project Gutenberg™ work. The Foundation makes
no representations concerning the copyright status of any work in
any country other than the United States.
1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if you
provide access to or distribute copies of a Project Gutenberg™ work
in a format other than “Plain Vanilla ASCII” or other format used in
the official version posted on the official Project Gutenberg™ website
(www.gutenberg.org), you must, at no additional cost, fee or
expense to the user, provide a copy, a means of exporting a copy, or
a means of obtaining a copy upon request, of the work in its original
“Plain Vanilla ASCII” or other form. Any alternate format must
include the full Project Gutenberg™ License as specified in
paragraph 1.E.1.
• You pay a royalty fee of 20% of the gross profits you derive
from the use of Project Gutenberg™ works calculated using the
method you already use to calculate your applicable taxes. The
fee is owed to the owner of the Project Gutenberg™ trademark,
but he has agreed to donate royalties under this paragraph to
the Project Gutenberg Literary Archive Foundation. Royalty
payments must be paid within 60 days following each date on
which you prepare (or are legally required to prepare) your
periodic tax returns. Royalty payments should be clearly marked
as such and sent to the Project Gutenberg Literary Archive
Foundation at the address specified in Section 4, “Information
about donations to the Project Gutenberg Literary Archive
Foundation.”
• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.
1.F.
1.F.4. Except for the limited right of replacement or refund set forth
in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO
OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.
Please check the Project Gutenberg web pages for current donation
methods and addresses. Donations are accepted in a number of
other ways including checks, online payments and credit card
donations. To donate, please visit: www.gutenberg.org/donate.
Most people start at our website which has the main PG search
facility: www.gutenberg.org.