Chapter 1

DEVELOP PYTHON PROJECTS

1.1. Python types and protocols
1.2. Reminder about iteration with for and while
1.3. Reminder about function definition
1.4. Type annotations (aka type hints)
1.5. Exceptions
1.6. Virtual environments (venv)

Chapter 2 (optional, if appliable)

OBJECT-ORIENTED PROGRAMMING (O.O.P.)

2.1. The self object
2.2. The constructor
2.3. Inheritance
2.4. Magic methods (aka dunder methods)
2.5. Summary of OOP terminology

Chapter 3

STRUCTURE AND DISTRIBUTE PYTHON CODE

3.1. Python Enhancement Proposals (PEPs)
3.2. Decorators
3.3. Context manager: the with statement
3.4. Structure code as Python packages
3.5. The Python Package Index (PyPI.org)
3.6. Test Python packages
3.7. Distribute Python packages

Chapter 4

MANAGE IOs OF PYTHON CODE

4.1. Logging
4.2. Manipulate file/directory paths with pathlib
4.3. Use json and csv formats
4.4. Emit HTTP requests from Python with requests
4.5. Call external commands with subprocess
4.6. Multithreading, multiprocessing
4.7. Charset and encoding

Optional topics for day 3 (if appliable)

• Advanced object-oriented programming: class methods, static methods, properties, multiple inheritance, the MRO, metaclasses
• Python decorators
• Asynchronous programming: coroutines, tasks, and futures
• Python interpreters and compiling strategies
• Complexity, iterators and generators

CHAPTER 1

DEVELOP PYTHON PROJECTS

With reminders

Python types and protocols

Primitive types

Beware with floats

Python floats are IEEE754 floats with mathematically incorrect rounding precision:

0.1 + 0.1 + 0.1 - 0.3 == 0 # This is False print(0.1 + 0.1 + 0.1 - 0.3) # Returns 5.551115123125783e-17 but not 0

Also, they are not able to handle large differences of precisions:

1e-10 + 1e10 == 1e10 # This is True

When you deal with float number and if precision counts, use the decimal module!

from decimal import Decimal Decimal("1e-10") + Decimal("1e10") == Decimal("1e10") # This is False

Beware not to initialize Decimal with float since the precision is already lost: Decimal(0.1) will show Decimal('0.10000000000000000555111512312578270215')

Reminder about basic collections

Collections allow to store data in structures.

General purpose built-in containers are tuple, string, list, and dict.

Other containers exist in module collections.

The tuple

The tuple is the Python type for an ordered sequence of elements (an array).

Selection of an element uses the [ ] operator with an integer index starting at 0:

Tuple can be unpacked:

The string

The str type is an ordered sequence of characters.

Tuples and strings are immutable. Definition: An object is said immutable when its value canot be updated after the initial assignment. The opposite is mutable.

Demonstration: put the first letter of these sequences in lower case:

The list

A list is a mutable sequence of objects using integer indexes:

If needed, pop(i) and insert(value, i) operate at index i, but...

... list is fast to operate only at the right side!

Need a left-and-right efficient collection? Use deque or compare efficiency

The double-ended queue (deque)

A deque is great to append or remove elements at both extremities:

Deques perform great for appendleft() and popleft() while lists have poor performances for the equivalent operations insert(0, value) and pop(0).

The dictionary

The dictionary is a key-value pair container, mutable and ordered. Keys are unique.

With Python 3.7 and below, dictionaries are unordered (see OrderedDict if needed)

Python typing and protocols

Python typing is dynamic. Type is inferred from the value Runtime type

Runtime type of v can be introspected with type(v)

But pythonistas rely on duck typing: To state about the suitability of an obj object, the runtime type type(obj) is less important than the methods it declares

Example: As soon as method __iter__ exist in class C, then C is considered an iterable, no matters what type(C) returns.

Python has built-in protocols:

• Iterable: __iter__
• Iterator: __iter__ and __next__
• Sequence __getitem__ and __len__ (used by [] for instance)
• Container: __contains__ (used by in for instance)
• Callable: __call__
• Hashable: __hash__
• Context Manager: __enter__ and __exit__
• ...

You can use them in type annotations.

Reminder about iteration with for and while

Iteration on list i (or tuple t)

Iteration on string s

Iteration with a while loop

Iteration on dict d

List-comprehensions and dict-comprehensions

A comprehension is an inline notation to build a new sequence (list, dict, set).
Here is a list-comprehension:

You may optionally filter the selected values with an if statement:

Dict-comprehensions also work:

Reminder about function definition

No return statement is equivalent to return None

The star * is the flag that means 0 or n values. They are received in a list:

A named parameter is passed to a function via its name instead of its position:

The double star ** in the flag that means 0 or n named parameters. They are received as a dictionary:

One can return 2 values or more:

Call to compute() returns a tuple:

This tuple can also be unpacked:

The star usually means 0 or N element(s)

Type annotations (aka type hints)

Any Python variable can optionnally be associated to a type hint:

Inconsistent types and values are NOT noticed by the interpreter.

Annotations are ONLY intended to an (optional) type checker, such as PyCharm.

To specify more complex annotations, import them from typing:

• Any: every type
• Union[X, Y, Z]: one among several types (e.g. int, float or str)
• Callable[[X], Y]: function that takes X in input and returns Y
• Optional[X]: either X or NoneType
• ForwardRef(X): forward reference to X, used to circumvent circular imports

Data containers can also be fully typed, e.g. list[list[int]], dict[str: float]

Exceptions

An exception is an error.

The mechanism of exceptions allows to trigger, propagate and fix errors according to a specific mechanism.

An exception can be:

• raised with the raise keyword: it is triggered
• catched with the except keyword: it is catched and a fix is provided

When it is catched, a fix (workaround) is executed:

For instance, when value = a / b fails because b = 0, you might want to pursue the execution with value = 0.

Propagation of exceptions

At runtime, if an exception is raised but not catched, it is automatically propagated to the calling function.

If it is not catched there either, it goes up again ... and again ...

If it reaches the top of the interpreter without being catched, the interpreter exits.

Propagation is one of the main benefits of exceptions that allows to find the right balance between:

• no error management
• all functions calls individually tested for errors

Compared to a custom handling of errors, exceptions have the following benefits:

• they propagate automatically: allowing to provide general workarounds for large parts of code
• they are typed and all exception types are hierarchized

Common exception types

• ValueError: value error (e.g. square root of a negative number)
• TypeError: type error (e.g. adding an int with a str)
• IndexError: access to an index exceeding the list size
• KeyError: access to a dictionary key that does not exist
• NameError: access to an undeclared variable name or function name
• IOError: I/O error (e.g. corrupted data, unexpected end of file...)
• FileNotFoundError: file not found
• RuntimeError: error happening at runtime (parent of all of the above)
• SyntaxError: bad syntax (indentation, unexpected keywords, ...)
• KeyboardInterrupt: received SIGINT signal (Ctrl +C)

The try/except block

The basic syntax to catch an exception:

What happens at runtime:

Other uses of exceptions

Virtual environments (venv)

Context: All installed packages go into the site-packages directory of the interpreter.

The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories.

Learn more

For each new project you create/clone, create it its own dedicated virtual environment:

Then, every time you work on this project, activate its environment first:

Your terminal must prefix the prompt with the name of the env:

And quit the venv every time you stop working on the project:

In an activated venv, every call to the interpreter and every package installation will target the isolated virtual environment:

will run the Python version targeted by the venv

will install the latest numpy version into the venv

In practice, your IDE can handle venv creation, activation and deactivation automatically for you when you create or open/close a project.

PEP 668

You can no longer use pip to install packages outside a venv.

You can override this behaviour by passing --break-system-packages.

CHAPTER 2

OBJECT-ORIENTED PROGRAMMING (O.O.P.)

Python is multi-paradigm:

• Imperative: instructions create state changes
• Object-oriented: instructions are grouped with their data in objects/classes
• Functional: instructions are math function evaluations

All 3 paradigms are popular in the Python community, and often mixed all together.

Here is a program to handle the sales of an apartment:

Note: because of the scope of variables, global variables would be required here

In classic programming, these are variables...

... and these are functions:

However, functions usually manipulate on data stored in variables. So functions are linked to variables.

In Object-Oriented Programming, variables and functions are grouped into a single entity named a class that behaves as a data type:

Note: this intermediary explanation is not yet a valid Python code snippet

Object-Oriented Programming introduced specific vocabulary:

Types are called classes:

Functions are called methods:

Variables are called attributes:

Since the declaration of a class defines a new type (here, Apartment), the program can declare several independant apartments:

In this statement:

• Apartment is a class
• apartment_dupont is an object (an instance of a class)
• Apartment() is the constructor (the method creating an object out of a class)

This statement is a method call on object apartment_dupont.

Method calls can create side effects to the object (modifications of its attributes).

Like regular functions, methods can take parameters in input. Here, an integer, 15.

The self object

• self is the name designating the instanciated object
• self is implicitly passed as the first argument for each method call
• self can be read as "this object"

In other languages like Java or C++, self is named this.

The constructor

The constructor is the specific method that instanciates an object out of a class. It is always named __init__.

Here is now a valid Python syntax for our class.

This is the class declaration:

This is the class instanciation:

The constructor, like any other method, can accept input parameters:

While attributes are accessed using the prefix self. from inside the class...

...they can be accessed from outside the class, using object name as the prefix:

However some attributes may have a protected or private scope:

class Foo: def __init__(self): self.public = 0 self._protected = 0 self.__private = 0 # Name mangling applies here

Protected attributes are not enforced but private ones rely on name mangling:

Inheritance

A furnished apartment is the same as an Apartment. But with additional furniture.

The super() function allows to call the same method in the parent class.

Note: Former Pythons require a longer syntax: super(CurrentClassName, self)

Magic methods (aka dunder methods)

• apart1 + apart2 → Apartment.__add__(self, other) → Addition
• apart1 * apart2 → Apartment.__mul__(self, other) → Multiplication
• apart1 == apart2 → Apartment.__eq__(self, other) → Equality test
• str(apart) → Apartment.__str__(self) → Readable string
• repr(apart) → Apartment.__repr__(self) → Unique string

Magic methods reading or altering attributes:

• getattr(apart, "price") → Apartment.__getattr__(self, name)
• setattr(apart, "price", 10) → Apartment.__setattr__(self, name, val)
• delattr(apart, "price") → Apartment.__delattr__(self, name)

This is why Python's duck typing does not rely on runtime types.

Summary of OOP terminology

• A class is a type owning attributes and methods
• An object is an instance of a class
• Instanciating a class consists into building an object from this class
• The constructor is the method initializing the object: __init__()
• An attribute is a variable from a class (or from an object)
• A method is a function from a class (or from an object)
• A (child) class may inherit from another (parent)
• A method from a child class may override the same name in the parent

CHAPTER 2

STRUCTURE AND DISTRIBUTE PYTHON CODE

Python Enhancement Proposals (PEPs)

The PEPs rule the Python language. In their lifetime the PEPs are proposed, debated, rejected/accepted and implemented.

They are usually not very user-friendly but allow to understand some implementation choices and the implementation of the interpreter.

PEP 0 (Python PEPs)

PEP 0 is the index of all PEPs, on peps.python.org

PEP 8 (style guide for Python langage)

PEP 8 is the style guide for Python code: Indentation, line length, blank lines...

PEP 257 (docstrings)

PEP 257 tells how to document your code with docstrings (triple quotes).

Docstrings can document a function, class, variable, whole file, package...

Example docstring on a function definition:

Docstring format is RST : ReStructuredText, but other formats coexist.

After typing """ Your IDE may autocomplete the docstring with a sketch.

Decorators

The role of a decorator is to alter the behaviour of the function that follows with no need to modify the implementation to the function itself.

It can be seen as adding "options" to a function, in the form of a wrapper code.

In that case the call of function() will be equivalent to decorator(function()).

Decorators may take parameters in input.

Learn more

Example 1: @classmethod is a decorator that passes the class type cls passed as the first parameter to the following function.

Example 2: Web frameworks usually use decorators to associate a function e.g. get_bookings_list() to:

• an endpoint e.g. /bookings/list
• a HTTP method e.g. GET

Here is how Flask works:

To define your own decorator, you need to write a function returning a function:

The functools module is a set of decorators intended to alter functions.

Context manager: the with statement

The keyword with is a context manager that protects a resource to make sure it is actually teared down after allocation in any case.

What if an exception occurs during the processing of f? It wouldn't be closed.

The context manager ensures that the resource is closed in any case:

The standard library is compatible with context managers for files, locks, and synchronisation primitives. But you may also create your own:

The contextlib module provides other tools to manage contexts.

Managers can also be decorators:

See full example with __enter__ and __exit__ in the official doc

Structure code as Python packages

Difference between modules and packages

A module is a Python file, e.g. some/folder/mymodule.py. The module name uses the dotted notation to mirror the file hierarchy: some.folder.mymodule

Either the module is made to be:

• executed from a shell: it is a script: python some/folder/mymodule.py
• imported from another module: import mymodule (need to be installed in sys.path, see p55)

A Python package is a folder containing modules and optional sub-packages: some is a package, folder is a sub-package.

Scripts : the shebangs

On UNIX OSes a shebang is a header of a Python script that tells the system shell which interpreter is to be called to execute this Python module.

Invoke the env command to fetch the suitable interpreter for python3 with:

Direct call to the interpreter is possible but NOT recommended, since it will force the interpreter by ignoring any virtual environment you could be in:

The Windows shell ignores shebangs, but you should provide them anyway.

Regular structure of packages

• Packages and sub-packages allow to bring a hierarchy to your code
• The package's hierarchy is inherited from the files-and-folders hierarchy
• Modules hold resources that can be imported later on, e.g.:
  • Constants
  • Classes
  • Functions...
    

• All packages and sub-packages must contain an __init__.py file each
• In general __init__.py is empty but may contain code to be executed at import time

Then the package or subpackages can be imported:

Specific resources can also be imported:

Relative imports (Imports internal to a package)

Relative import from the same folder:

Relative import from a parent folder:

• Do not put any slash such as import ../my_math
• Relative imports can fetch . (same dir), .. (parent), ... (parent of parent)
• Relative imports are forbidden when run from a module outside a package
• Using absolute imports instead of relatives could result in name collisions

Execution of packages (only) stores Python bytecode in __pycache__ folder

As a developer, you can ignore them, the interpreter handles compiling by itself.

This is the compiling behaviour of CPython, the most popular interpreter, in C.

Other interpreters, eg Pypy, implement more efficient compiling: JIT compiling.

The Python path

The interpreter seeks for absolute import statements in the Python path sys.path.

This is a regular Python list and it can be modified at runtime (with append) to add paths to your libs.

The Python Package Index (PyPI.org)

pypi.org is a global server that allows to find, install and share Python projects.

pypi.org is operated by the Python Packaging Authority (PyPA): a working group from the Python Software Foundation (PSF).

The command-line tool Package Installer for Python (pip) can be used to install packages by their name, e.g. bottle. It can install from various sources (Link to code repos, ZIP file, local server...) and seeks on PyPI if no source is given:

Non-installable Python projects usually have a file requirements.txt at their root

pip has the following options:

• pip install -r requirements.txt to install all dependencies form the file
• pip freeze > requirements.txt to create a file of frozen versions

installable packages have no such file but specify dependencies elsewhere (e.g. in pyproject.toml for installable packages using setuptools).

PyPI Security warning

PyPI packages caught stealing credit card numbers & Discord tokens

Perform sanity checks before installing a package

• Is the package still maintained and documented?

• Does the developer consider bugs and improvements?

• Is the package developer reliable?

• If not opensource, is the development of this package likely to continue?

PyPI Typosquatting warning

pip install -r requirements.txt # pip install requirements.txt pip install rabbitmq # pip install rabitmq pip install matplotlib # pip install matploltib

Where to find/host Python documentation?

Doc of built-in packages

All builtin packages are documented in 8 languages on:
docs.python.org

builtin = anything that comes with the interpreter itself

Doc of non-built-in packages

Non-builtin packages (installed via pip) have their doc on their own website.

non-builtin = what is installed on top of the interpreter (eg. with pip)

Search that package on pypi.org to easily find its doc

Test Python packages

• Packages pytest and unittest are frequently used to test Python apps
• unittest relies on the regular test framework:
  • Setup: Prepare every prerequisite for the test
  • Call: call the tested function with input parameters setuped before
  • Assertion: an assert is a ground truth that must be true
  • Tear down: Cleanup everything that has been created for this test
• pytest is a light test framework
• On top of these, tox allows to run tests in multiple environments (e.g. Python versions)

Test files are sometimes placed in a tests/ directory, file names are prefixed with test_*.py and test function names are also prefixed with test_

Naming tests according to these conventions will allow auto-discovery of tests by the test tool: it will go through all directories and subdirectories looking for tests to execute.

Then just type pytest and the test report will be printed in the terminal!

Distribute Python packages

setuptools simplifies the package distribution. Learn more

You need a pyproject.toml file that tells:

• The package name and version number
• The list of deps on other packages from PyPI, git repos, ...
• The entry points (executables, commands, ...)
• How to build the package (using hatching, setuptools...)

setuptools replaces distutils, deprecated in 3.10.

setup.py is now discouraged in place of pyproject.toml

The TOML file then offers distribution tools:

• Install the package in the current environement:

• Build distribution:
  • sdist : Source distribution
  • bdist_wheel : Binary distribution

Learn more about package distribution: PyPA docs

Remarks about binary distribution bdist_*

• Binary format at platform-dependant (OS, arch, Python implementation, ABI)
• .egg files are just zip files containing sdist or bdist, you can unzip them
• Several binary formats exist: wheel, egg... Nowadays, wheel is preferred
• wheel files are named this way: my_math-3.0.4-py3-none-any.whl where:
  • my_math is your package name
  • 3.0.4 is your package version
  • py3 is the Python implementation tag
  • none is the ABI tag (the C API for Python)
  • any is the platform (x86_64, arm, macos...)

Learn more about package distribution: Python docs

Uploading your package distribution on PyPI

Once sdist and/or bdist are available, several pipelines exist to share your project.

Nowadays, uploading to PyPI with twine is the preferred option:

1. Create an account on PyPI or in the sandbox TestPyPI if you're just testing
2. pip install twine
3. twine upload dist/* --repository testpypi

Drop the --repository argument to upload to the regular PyPI.
Parameter --repository can also target your own mirror server. Learn more.

CHAPTER 3

MANAGE IOs OF PYTHON CODE

Logging

Python has a module dedicated to logging. It classes each log entry in levels of criticity: debug, info, warning, error and allows to filter them. Learn more

Step 1: Produce log entries

The logging library uses modular approach to organize logs in a big app. Usually every module has its own logger named as itself:

When a message is posted to logger L:

1. L decides whether to handle the event based on the level/filters
2. Handlers of L get notified and react if their own level/filter match
3. L's parent is notified, if appropriate

Step 2: Consume log entries

Both the logger & handler must accept the minimum level so the entry is printed

The simple config is a quick way to activate a stream handler for all loggers. But the output will be fussy since logs from all modules will be printed, imports too:

Manipulate file/directory paths with pathlib

Found in legacy code running on Windows:

Problems:

• Error-prone manipulation of the double-backslash
• Not cross-platform: \ is only Windows-compatible
• Does not offer path-related features: get filename, get extension, get parent...

The solution : The built-in pathlib library

Then you can manipulate that path:

User's personal data folder must be retrieved using pathlib to be cross-platform:

os.path is getting old, prefer the use of the equivalent pathlib instead

Use json and csv formats

Manipulate JSON data as strings (e.g. network payload data) or files:
json documentation

Manipulate CSV files (mostly for datascience):
csv documentation

Emit HTTP requests from Python with requests

requests allows to emit synchronous HTTP requests to servers.

This is NOT a built-in module, install it from PyPI.org with pip install requests and read the requests documentation

Call external commands with subprocess

This is a built-in module, read the subprocess documentation

Run a command

• capture_output=True captures the standard output and error to Python
• text=True decodes the output bytes into a decoded string

Advanced usecases (e.g. long-running commands) use Popen instead of run()

if the external process is your own Python code, use multiprocessing.
os.system is deprecated, use subprocess instead

Multithreading, multiprocessing

Definitions:

• Multithreading: Split work into several threads within the same process and CPU.
• Multiprocessing: Split work into several processes dispatched to several CPUs.

The bottleneck of Python Multithreading: the reference counter

The interpreter holds a counter counting how many references point to a literal.

If the counter reaches 0, the literal is destroyed. This is how Python frees memory.

The Python Global Interpreter Lock (GIL)

Several implementations of the Python interpreter exist:

• CPython (By far the most popular)
• Jython, IronPython, PyPy...

The GIL is a mutex that protects the reference counters of CPython objects.

However it prevents multiple threads from executing Python bytecodes at once. It offers poor performance for multi-threaded programs, if they are CPU-bound.

The GIL was a regular debate: so far too hard for low benefits

But hardware multiplied CPUs so Python 3.13 introduced --disable-gil

Because of the GIL, multiprocessing is way more efficient that multithreading.

Should I use multiprocessing or multithreading?

It depends of the nature of your application:

• CPU-bound code: Because of the GIL, only multiprocessing will be able to use all available CPUs
• IO-bound code: Both will work, but multithreading may be more handy. Also, asynchronous programming can be a solution since it's specialy tailored to optimize IOs.

Multithreading example

Multiprocessing example

Inter-Process Communication (IPC)

Python offers the following IPC primitives:

• multiprocessing.Lock and threading.Lock (mutex, authorizes 1)
• multiprocessing.Semaphore and threading.Semaphore (authorizes up to n)
• multiprocessing.Pipe (1-to-1 FIFO)
• multiprocessing.Queue (n-to-n FIFO)
• multiprocessing.Event and threading.Event

Java's thread-safe collections offer some level of safety but do not solve all risks:

High-level multiprocessing API

multiprocessing has higher-level tools compatible with with:

• Pool(n) : represents a Pool of n process
• Manager : manages the pool, e.g. by sharing variables between processes:
• manager.dict() for a shared dict, or manager.list() or manager.int()...

Unpack all args: pool.starmap(f, ((a, b), (c, d)) f(a, b), f(c, d)
Apply same args: pool.apply(f, (a, b)) f(a, b), f(a, b)

Charset and encoding

All text assets (Python string or .py file, .json file, .txt file…) are encoded using a charset, a correspondance table between Bytes Actual character

• e.g. in utf-8: 0xC3A9 é
• e.g. in latin-1: 0xE9 é

You MUST know the encoding of a text asset in order to read it.

If you do not, it can be guessed but there is a chance to make mistakes.

This is what happens if you do not specify explicit encodings and rely on default parameters of file reading libraries.

If the guess is wrong it may result in e.g. hétérogène instead of hétérogène.

Rule of thumb for encoding and decoding

• I RECEIVE data coming IN the interpreter (from stdin, the network, a file...):

  • If it is a str: it has already been decoded by reading functions
    (Prey that they used the right charset )
  • If it is a bytes: decode-it with the charset declared by the source e.g.
    data.decode("utf-8") if the source sends UTF-8 strings

• My Python code must operate only on Unicode (Python type str)

• I SEND data OUT of the interpreter (to stdout, to the network, to a file...):

  • If it is a bytes: it has already been encoded by writing functions
    (Prey that they used the right charset )
  • If it is a str: encode-it with the charset declared by the recipient e.g.
    data.encode("utf-8") if the recipient expects UTF-8 strings

What is Unicode?

Unicode is NOT a charset, this is the global table of all world code points.
e.g. U+1F601:

Encodings (ASCII, latin-1, UTF-8...) may be able to code Unicode in whole or in part.

ASCII and latin-1 can only code a subset of Unicode (resp 128 & 256 code points).

UTF-8, UTF-16 and UTF-32 can code all Unicode code points.

The difference between UTF-8, 16 and 32 is about how they code characters:

• UTF-8 uses a variable number of bytes
• UTF-16 uses a variable number of bytes
• UTF-32 uses a fixed number of 4 bytes

In average, UTF-16 is more efficient for Asian texts compared to UTF-8. But UTF-8 is more widely recommanded as a global standard.

• The Python interpreter holds:
  • decoded unicode strings in type str
  • encoded strings in type bytes
• encode() and decode() methods on these objects can convert them between bytes and str
• stdout and stderr in your terminal are outputs, they have a charset
• stdin in your terminal is an input, it has a charset
• Your .py file itself is an input, it has a charset