Developer Guide¶
Do not hang around in Pythran code base without your developer guide! It is the compass that will guide you in the code jungle!
Disclaimer¶
This document is a never ending work-in-progress draft. Please contribute!
Configuration¶
Pythran can be configured with a rc file. An example is found in pythran/pythran.cfg. Look at it! To customize it:
$> cp pythran/pythran.cfg ~/.pythranrc
In particular, you may want to add -g -O0
to the cxxflags
.
Coding Style¶
All Python code must be conform to the PEP 8, and the flake8
command must not
yield any message when run on our database. Additionally, avoid backslashes,
and try to make your code as concise as possible.
$> flake8 pythran/.py pythran//.py –exclude=”pythran/tests/test.py,__init__.py”
C++ code use spaces (no tabs) and a tab width of 4.
File Hierarchy¶
Listing the top level directory yields the following entries:
- setup.py
The files that describes what gets installed, that holds
PyPI
entries and such.- docs/
If you’re reading this document, you know what it’s all about!
MANUAL
is the user documentation andDEVGUIDE
is the developer documentation.Use
make
from this directory to produce the static website.- LICENSE
Boring but important stuff.
- MANIFEST.in
Describe additional stuff to package there.
- README.rst
Quick introduction and description of _pythran_.
- pythran/
The source of all things.
- pythran/tests/
The source of all issues.
- pythran/pythonic/
Where C++ back-end lies.
Validation¶
pythran
uses the unittest
module and the pytest package to manage test cases.
All requirements are listed in pythran/tests/requirements.txt
.
The whole validation suite is run through the command:
$> python -m pytest pythran/tests
To run it faster we use the pytest
extension xdist, the test suite will run using all
available cores. Otherwise it might run very slowly, something like four
hours on a decent laptop :’(.
Note that it is possible to use the pytest
module to pass a subset of the
test suite:
$> pytest -n 8 pythran/tests/test_list.py
runs all the tests found in pythran/tests/test_list.py
.
There are two kinds of tests in pythran
:
unit tests that test a specific feature of the implementation. Such tests are listed as method of a class deriving from
test_env.TestEnv
and must call therun_test(function_to_translate, *effective_parameters, **name_to_signature)
method [1]. It translatesfunction_to_translate
into a native function using the type annotations given in thename_to_signature
dictionary, runs both the python and the native version witheffective_parameters
as arguments and asserts the results are the same.test cases that are just plain python modules to be converted in native module by
pythran
. It is used to test complex situations, codes or benchmarks found on the web etc. They are just translated, not run. These test cases lie inpythran/tests/cases/
and are listed inpythran/tests/test_cases.py
.
C++ runtime¶
The C++ code generated by pythran
relies on a specific back-end,
pythonic
. It is a set of headers that mimics Python’s intrinsics and
collections behavior in C++. It lies in pythran/pythonic/
. There is one
directory per module, e.g. pythran/pythonic/numpy
for the numpy
module,
and one file per function, e.g. pythran/pythonic/numpy/ones.hpp
for the
numpy.ones
function. Type definitions are stored in the seperate
pythran/pythonic/types
directory, one header per type. Each function header
must be #includ
-able independently, i.e. it itself includes all the type
and function definition it needs. This helps keeping compilation time low.
All Pythran functions and types live in the pythonic
namespace. Each extra
module defines a new namespace, like pythonic::math
or
pythonic::random
, and each type is defined in the pythonic::types
namespace. The DECLARE_FUNCTOR
and DEFINE_FUNCTOR
macros from
pythonic/utils/functor.hpp
is commonly used to convert functions into
functors and put them into the mandatory functor
namespace.
The pythonic runtime can be used without Python support, so it is important to
protect all Python-specific stuff inside ENABLE_PYTHON_MODULE
guard.
All methods are represented by functions in Pythran. The associated
pseudo-modules are prefixed and suffixed by a double underscore __
, as in
pythran/pythonic/__list__
.
Benchmarking and Testing¶
Stand-alone algorithms are put into pythran/tests/cases
. They must be valid
Pythran input (including spec annotations). To be taken into account by the
validation suite, they must be listed in pythran/tests/test_cases.py
. To be
taken into account by the benchmarking suite, they must have a line starting
with the #runas
directive. Check pythran/tests/matmul.py
for a complete
example.
To run the benchmark suite, one can rely on:
$> python setup.py bench --mode=<mode>
where <mode> is one among:
- python
Uses the interpreter used to run
setup.py
.- pythran
Uses the Pythran compiler.
- pythran+omp
Uses the Pythran compiler in OpenMP mode.
All measurements are made using the timeit
module. The number of iterations
is customizable through the --nb-iter
switch.
How to¶
- Add support for a new module:
- Provide its C++ implementation in
pythran/pythonic++/<mymodule>
. pythran/pythonic++/math/*.hpp
andpythran/pythonic++/__list__/*.hpp
are good example to referer to.
- Provide its C++ implementation in
- Provide its description in
pythran/tables.py
. Each function, method or variable must be listed there with the appropriate description.
- Provide its description in
- Provide its test suite in
pythran/tests/
under the name test_my_module.py
. One test case per function, method or variable is great.
- Provide its test suite in
- Add a new analysis:
Subclass one of
ModuleAnalysis
,FunctionAnalysis
orNodeAnalysis
.List analysis required by yours in the parent constructor, they will be built automatically and stored in the attribute with the corresponding uncameled name.
Write your analysis as a regular
ast.NodeVisitor
. The analysis result must be stored inself.result
.Use it either from another pass’s constructor, or through the
passmanager.gather
function.
- Push changes into the holy trunk:
Use the
github
interface and the pull/push requests features- Make your dev available on the web and asks for a merge on the IRC
channel
#pythran
onirc.oftc.net
(via your browser: https://webchat.oftc.net)