Additional features¶

This section discusses various features that did not fit in naturally in one of the previous sections.

Dataclasses¶

In Python 3.7, a new dataclasses module has been added to the standard library. This module allows defining and customizing simple boilerplate-free classes. They can be defined using the @dataclasses.dataclass decorator:

from dataclasses import dataclass, field

@dataclass
class Application:
    name: str
    plugins: List[str] = field(default_factory=list)

test = Application("Testing...")  # OK
bad = Application("Testing...", "with plugin")  # Error: List[str] expected

Mypy will detect special methods (such as __lt__) depending on the flags used to define dataclasses. For example:

from dataclasses import dataclass

@dataclass(order=True)
class OrderedPoint:
    x: int
    y: int

@dataclass(order=False)
class UnorderedPoint:
    x: int
    y: int

OrderedPoint(1, 2) < OrderedPoint(3, 4)  # OK
UnorderedPoint(1, 2) < UnorderedPoint(3, 4)  # Error: Unsupported operand types

Dataclasses can be generic and can be used in any other way a normal class can be used:

from dataclasses import dataclass
from typing import Generic, TypeVar

T = TypeVar('T')

@dataclass
class BoxedData(Generic[T]):
    data: T
    label: str

def unbox(bd: BoxedData[T]) -> T:
    ...

val = unbox(BoxedData(42, "<important>"))  # OK, inferred type is int

For more information see official docs and PEP 557.

Caveats/Known Issues¶

Some functions in the dataclasses module, such as replace() and asdict(), have imprecise (too permissive) types. This will be fixed in future releases.

Mypy does not yet recognize aliases of dataclasses.dataclass, and will probably never recognize dynamically computed decorators. The following examples do not work:

from dataclasses import dataclass

dataclass_alias = dataclass
def dataclass_wrapper(cls):
  return dataclass(cls)

@dataclass_alias
class AliasDecorated:
  """
  Mypy doesn't recognize this as a dataclass because it is decorated by an
  alias of `dataclass` rather than by `dataclass` itself.
  """
  attribute: int

@dataclass_wrapper
class DynamicallyDecorated:
  """
  Mypy doesn't recognize this as a dataclass because it is decorated by a
  function returning `dataclass` rather than by `dataclass` itself.
  """
  attribute: int

AliasDecorated(attribute=1) # error: Unexpected keyword argument
DynamicallyDecorated(attribute=1) # error: Unexpected keyword argument

The attrs package¶

attrs is a package that lets you define classes without writing boilerplate code. Mypy can detect uses of the package and will generate the necessary method definitions for decorated classes using the type annotations it finds. Type annotations can be added as follows:

import attr

@attr.s
class A:
    one: int = attr.ib()          # Variable annotation (Python 3.6+)
    two = attr.ib()  # type: int  # Type comment
    three = attr.ib(type=int)     # type= argument

If you’re using auto_attribs=True you must use variable annotations.

import attr

@attr.s(auto_attribs=True)
class A:
    one: int
    two: int = 7
    three: int = attr.ib(8)

Typeshed has a couple of “white lie” annotations to make type checking easier. attr.ib and attr.Factory actually return objects, but the annotation says these return the types that they expect to be assigned to. That enables this to work:

import attr
from typing import Dict

@attr.s(auto_attribs=True)
class A:
    one: int = attr.ib(8)
    two: Dict[str, str] = attr.Factory(dict)
    bad: str = attr.ib(16)   # Error: can't assign int to str

Caveats/Known Issues¶

The detection of attr classes and attributes works by function name only. This means that if you have your own helper functions that, for example, return attr.ib() mypy will not see them.
All boolean arguments that mypy cares about must be literal True or False. e.g the following will not work:
```
import attr
YES = True
@attr.s(init=YES)
class A:
    ...
```
Currently, converter only supports named functions. If mypy finds something else it will complain about not understanding the argument and the type annotation in __init__ will be replaced by Any.
Validator decorators and default decorators are not type-checked against the attribute they are setting/validating.
Method definitions added by mypy currently overwrite any existing method definitions.

Using a remote cache to speed up mypy runs¶

Mypy performs type checking incrementally, reusing results from previous runs to speed up successive runs. If you are type checking a large codebase, mypy can still be sometimes slower than desirable. For example, if you create a new branch based on a much more recent commit than the target of the previous mypy run, mypy may have to process almost every file, as a large fraction of source files may have changed. This can also happen after you’ve rebased a local branch.

Mypy supports using a remote cache to improve performance in cases such as the above. In a large codebase, remote caching can sometimes speed up mypy runs by a factor of 10, or more.

Mypy doesn’t include all components needed to set this up – generally you will have to perform some simple integration with your Continuous Integration (CI) or build system to configure mypy to use a remote cache. This discussion assumes you have a CI system set up for the mypy build you want to speed up, and that you are using a central git repository. Generalizing to different environments should not be difficult.

Here are the main components needed:

A shared repository for storing mypy cache files for all landed commits.
CI build that uploads mypy incremental cache files to the shared repository for each commit for which the CI build runs.
A wrapper script around mypy that developers use to run mypy with remote caching enabled.

Below we discuss each of these components in some detail.

Shared repository for cache files¶

You need a repository that allows you to upload mypy cache files from your CI build and make the cache files available for download based on a commit id. A simple approach would be to produce an archive of the .mypy_cache directory (which contains the mypy cache data) as a downloadable build artifact from your CI build (depending on the capabilities of your CI system). Alternatively, you could upload the data to a web server or to S3, for example.

Continuous Integration build¶

The CI build would run a regular mypy build and create an archive containing the .mypy_cache directory produced by the build. Finally, it will produce the cache as a build artifact or upload it to a repository where it is accessible by the mypy wrapper script.

Your CI script might work like this:

Run mypy normally. This will generate cache data under the .mypy_cache directory.
Create a tarball from the .mypy_cache directory.
Determine the current git master branch commit id (say, using git rev-parse HEAD).
Upload the tarball to the shared repository with a name derived from the commit id.

Mypy wrapper script¶

The wrapper script is used by developers to run mypy locally during development instead of invoking mypy directly. The wrapper first populates the local .mypy_cache directory from the shared repository and then runs a normal incremental build.

The wrapper script needs some logic to determine the most recent central repository commit (by convention, the origin/master branch for git) the local development branch is based on. In a typical git setup you can do it like this:

git merge-base HEAD origin/master

The next step is to download the cache data (contents of the .mypy_cache directory) from the shared repository based on the commit id of the merge base produced by the git command above. The script will decompress the data so that mypy will start with a fresh .mypy_cache. Finally, the script runs mypy normally. And that’s all!

Caching with mypy daemon¶

You can also use remote caching with the mypy daemon. The remote cache will significantly speed up the first dmypy check run after starting or restarting the daemon.

The mypy daemon requires extra fine-grained dependency data in the cache files which aren’t included by default. To use caching with the mypy daemon, use the --cache-fine-grained option in your CI build:

$ mypy --cache-fine-grained <args...>

This flag adds extra information for the daemon to the cache. In order to use this extra information, you will also need to use the --use-fine-grained-cache option with dmypy start or dmypy restart. Example:

$ dmypy start -- --use-fine-grained-cache <options...>

Now your first dmypy check run should be much faster, as it can use cache information to avoid processing the whole program.

Extended Callable types¶

Note

This feature is deprecated. You can use callback protocols as a replacement.

As an experimental mypy extension, you can specify Callable types that support keyword arguments, optional arguments, and more. When you specify the arguments of a Callable, you can choose to supply just the type of a nameless positional argument, or an “argument specifier” representing a more complicated form of argument. This allows one to more closely emulate the full range of possibilities given by the def statement in Python.

As an example, here’s a complicated function definition and the corresponding Callable:

from typing import Callable
from mypy_extensions import (Arg, DefaultArg, NamedArg,
                             DefaultNamedArg, VarArg, KwArg)

def func(__a: int,  # This convention is for nameless arguments
         b: int,
         c: int = 0,
         *args: int,
         d: int,
         e: int = 0,
         **kwargs: int) -> int:
    ...

F = Callable[[int,  # Or Arg(int)
              Arg(int, 'b'),
              DefaultArg(int, 'c'),
              VarArg(int),
              NamedArg(int, 'd'),
              DefaultNamedArg(int, 'e'),
              KwArg(int)],
             int]

f: F = func

Argument specifiers are special function calls that can specify the following aspects of an argument:

its type (the only thing that the basic format supports)
its name (if it has one)
whether it may be omitted
whether it may or must be passed using a keyword
whether it is a *args argument (representing the remaining positional arguments)
whether it is a **kwargs argument (representing the remaining keyword arguments)

The following functions are available in mypy_extensions for this purpose:

def Arg(type=Any, name=None):
    # A normal, mandatory, positional argument.
    # If the name is specified it may be passed as a keyword.

def DefaultArg(type=Any, name=None):
    # An optional positional argument (i.e. with a default value).
    # If the name is specified it may be passed as a keyword.

def NamedArg(type=Any, name=None):
    # A mandatory keyword-only argument.

def DefaultNamedArg(type=Any, name=None):
    # An optional keyword-only argument (i.e. with a default value).

def VarArg(type=Any):
    # A *args-style variadic positional argument.
    # A single VarArg() specifier represents all remaining
    # positional arguments.

def KwArg(type=Any):
    # A **kwargs-style variadic keyword argument.
    # A single KwArg() specifier represents all remaining
    # keyword arguments.

In all cases, the type argument defaults to Any, and if the name argument is omitted the argument has no name (the name is required for NamedArg and DefaultNamedArg). A basic Callable such as

MyFunc = Callable[[int, str, int], float]

is equivalent to the following:

MyFunc = Callable[[Arg(int), Arg(str), Arg(int)], float]

A Callable with unspecified argument types, such as

MyOtherFunc = Callable[..., int]

is (roughly) equivalent to

MyOtherFunc = Callable[[VarArg(), KwArg()], int]

Note

Each of the functions above currently just returns its type argument at runtime, so the information contained in the argument specifiers is not available at runtime. This limitation is necessary for backwards compatibility with the existing typing.py module as present in the Python 3.5+ standard library and distributed via PyPI.

Additional features¶

Dataclasses¶

Caveats/Known Issues¶

The attrs package¶

Caveats/Known Issues¶

Using a remote cache to speed up mypy runs¶

Shared repository for cache files¶

Continuous Integration build¶

Mypy wrapper script¶

Caching with mypy daemon¶

Refinements¶

Extended Callable types¶