Jul 25, 2015

[Python] Trying out PEP0484 Type Hinting, ABCMeta, and PyCharm

After my wife gave birth to our first-born daughter, I started a new Python project and named it after this baby girl. This was a brand new project and did not require me to think about backward compatibility, so I decided to use later version of Python and challenge to employ some ideas that I liked while learning Java: type safety, private properties, explicit overriding with @Override, and organized inheritance. These are things that make me feel safer and comfortable.
Python itself does not actually support them, but I found some can be achieved in certain level with help of modules and IDE.

Type Hinting

I just realized that PEP 0484 was accepted on May. This kind of type hinting was already available with docstrings, but it employed PEP 3107 style annotations to achieve it, so it is now officially a part of Python 3.5 or later. Python 2.7 or 3.2+ can still use backported version of typing module to implement it. I knew, in terms of readability, there was a constructive discussion introduced in Type hinting in Python: focus on readability, but I decided to give this a try.
Utilizing this type hinting, however, included another challenge for me. Since jedi-vim did not support this yet, I had to switch from Vim to PyCharm. PyCharm's support is still limited, but basic features are covered. With the help of IdeaVim plugin, switching cost was much lower than I had ever thought.

Installing typing module

Installing typing was a bit tricky. Regular pip command, `pip install typing`, somehow broke and ended up installing empty ver.0.0, so I had to explicitly set version with `pip install -lv typing==3.5.0b1`. This worked O.K.
With the nature of PEP0484, type checking is not done on runtime, so the type hinting is mostly for static check by other third party modules or IDEs. Therefore I found it really important to know IDEs' current support level. The worst case scenario is that you think your IDE supports PEP 0484 perfectly, you mis-define or miscall the function, but your IDE does not actually support correctly so you end up with no warnings. Here are some limitations I have found so far.

Some limitation with PyCharm

Type aliases

Using type aliases to declare complex type at one place or to give a more meaningful type name seems to be a pretty good idea, but this support is limited with current version of PyCharm.
See the capture below. PyCharm correctly gives warnings when the argument declaration has no type alias or has simple alias with built-in types; but gives no warning for type alias with customized types provided by typing module.


Just like other warnings, you can click on the yellow marked part to see the detailed description.


The tricky thing is that you want to use type alias for complex declaration to avoid miscalling or mis-declaration, but this ironically leads to a bug since the use of type aliases eliminates the chance to see the proper warnings. I stored all complex declarations in types.py and used them in other modules, then I found out it did not work with PyCharm. We must keep this in mind.
It will be supported with PyCharm ver.5.0, though. Let us look forward to it.

Abstract collection types

Although PEP0484 recommends to use abstract collection types for arguments as below, such declaration results in warnings.
Note: Dict , List , Set and FrozenSet are mainly useful for annotating return values. For arguments, prefer the abstract collection types defined below, e.g. Mapping , Sequence or AbstractSet .

I guess this is not supported yet just like the previous one, but as long as you get warnings you have chance to know something is happening and have option to ignore it. So I am not as worried as the previous type alias ignorance.

@overload

With type hinting, function signature became more detailed comparing to *args and **kwargs, so I became more interested in what typing module's @overload decorator would do for us.
It did not take so long to find that this decorator can only be used in stub files with .pyi extension, and the actual declaration and implementation stay in .py just like usual. In other words, even though typing module provide us this decorator, we can not declare same-named functions with different signatures.
Let us look at the signature below. This tells us that __getitem__ receives int or slice as an argument, and returns int or bytes as return value. BUT you can not be sure which of int or bytes is returned when you give int as argument.
def __getitem__(self, a: Union[int, slice]) -> Union[int, bytes]:
    if isinstance(a, int):
        # Do something
        pass
    else:
        # Do something 
        pass

In such case, in stub file, we can declare separately with @overload decorator like below:
@overload
def __getitem__(self, i: int) -> int: ...

@overload
def __getitem__(self, s: slice) -> bytes: ...

This looks more obvious that int argument returns int while slice argument returns bytes. Since I was not going to need such use cases and preparing separate .pyi file seemed a little pain, I did not use this, anyway.

My feedback about type hinting

As I employed type hinting with typing module, I became more cautious about giving arbitrary arguments. Especially for public methods, I stopped using dict, *args, **kwargs. Instead I started to define a JavaBean-like classes to hold those values, use name mangling strategy to provide private-like properties, and set @property for getter methods. As a happy side-effect, I enjoy stronger code completion where PyCharm support type hinting. Now I feel more comfortable and safe.

Structured/Organized inheritance

As I learned Java for several days, I started to like the idea of interface, default modifier, abstract class, @Override annotation, and etc... Those things seem to make me more comfortable in terms of safer design. It is a bit tiring to think about design in detail, though.
I found ABCMeta and Type hinting can support some of them.

ABCMeta

With a help of ABCMeta, we can create a class with abstract methods and abstract properties that obligate inheriting class to override, and can create default methods that can be overridden. This is a bit like Java's interface that has methods with abstract or default modifiers.

@abstractmethod and @abstractproperty

Method and property with these decorators obligate inheriting class to override. I am really comfortable with PyCharm's assistance giving me the option to override the abstract methods.


It automatically declare the function with the same signature, which I think is pretty neat. So when I want to define behavior, I like to use ABCMeta with @abstractmethod.

@Override

In Java, we have @Override annotation, but Python does not provide @override decorator. I missed it at first, but I found it somewhat O.K. since PyCharm gives us warnings for wrongful overriding. Why *somewhat*? It is now safer that PyCharm gives us warnings for mis-overriding, but we still miss the explicitness to indicate the declaring method is overriding the existing one or we are extending the class to declare a new method.

My feedback about abc module

After I started using abc.ABCMeta, I feel more comfortable about declaring behavior in an Java's interface-like class. It is still more casual, but it is nice that we can obligate inheriting class to behave in a certain way by overriding designated abstract methods.

Conclusion

When I work on my personal projects, it is not uncommon that projects are left for relatively a long time. Then I look into the implementation for maintenance purpose, but I do not remember much about them. So I feel more comfortable when things are organized and are explicitly declared.
With that said, things I tried this time are pretty neat, and I am looking forward to having next version of PyCharm.

Further reading

Posted by at
Labels: ,