pyre Setup, along with bazel

pyre is a Python checker developed within Meta, that checks for all errors in your python code. It does a great job. With it, we can typically fix many superficial problems (e.g. type mismatch) before a miserable runtime crashes after running the program for hours. I'm trying out its open source version, and configuring it with VScode and Bazel.

Like many python projects' configurations, the challenge is to make sure the paths are set correctly. With bazel, there are many python files hanging around:

1. my own code – which is the only target for pyre

2. virtual environment, where we do the daily dev, and where we installed lots of libraries to use.

3. bazel external output – the dependencies that bazel pulls in.

4. bazel compilation results, e.g. grpc, which are auto-gen code by bazel. These are not targets for pyre, but they are the requirements for many of our libraries, and we also hope pyre can recognize them, so as to reduce its false alarms.

My overall design is to:

1. Only scan my own code directories (1) for errors

2. Let pyre make use of the virtualenv for checking correctness when calling external libraries. This is leveraging (2) instead of (3). I found (3) hard to use.

3. Give up on (4).

So, going into implementing it.

My targeting directory structure is as follows:

.venv
my_code/
my_code/.pyre_configuration
my_code/bazel-out/...
my_code/bazel-bin/...

Having .venv installed outside of the code directory, and installing pyre there will let us leverage virtualenv, while keeping it outside of scanning.

First, installing pyre. The commands are quite simple:

cd /path/to/my_code
source ../.venv/bin/activate
pip install pyre-check typeshed-client
pyre init

Second, configure pyre. The above commands initialize a .pyre_configuration file. Below is my configuration. You may also find lots of other examples on github.

{
  "site_package_search_strategy": "all",
  "source_directories": [
    "."
  ],
  "exclude": [
    ".*/bazel-.*"
  ]
}

The only tricky part is to make sure pyre finds all the python dependencies, and provides useful diagnosis. The above lines configures the right behavior in my case. Let me explain these lines one by one:

1. site_package_search_strategy being set to "all" means we will also include packages not declared in the package files. I'm planning to use all the site-packages within the dev virtualenv, therefore setting it to all will surpress many false alarms.

2. source_directories being set to current directory, is consistent with my bazel setup.

3. the exclude statement ensures the bazel build output are not included in

Python interpreter needs to be outside of the current directory.

Third, get going with pyre. Now running pyre will now start complaining about many errors. Adding # pyre-strict in a python file will allow pyre to be very aggressive in reporting errors – which is actually rather helpful. After installing the VSCode pyre plugin from Meta, these errors will be highlighted in the files.

The above config does have some drawbacks:

1. The python packages used in pyre might be inconsistent with the actual version used in bazel builds.

2. Intermediate libraries generated by bazel (e.g. grpc) cannot be detected. Pyre will report errors.

Still seeking for better solutions.