I was wrapping a client rpc which was calling functions on a server rpc. The code looked like this.
The are over 20 functions on the RPC server, I didn’t feel like typing out all of the code for these functions. Also what if I added more rpc functions on the server ?. The solution in this case was metaclasses.
Note this demo uses python 3, the syntax for metaclasses with slightly different.
Understanding python classes and objects
To understand how to use metaclasses, one must understand how python classes and objects work behind the scenes. A great way to explore this is with ipython or ipython notebooks which is part of the juypter project.
Simply put classes and objects are nothing more then dictionaries. You can verify this yourself using your python interpretor.
Output from running above code.
Inside all python classes, there is a variable called __dict__. If we print it out, we see that type mappingproxy which if we look inside the python docs says that it is type of dictionary which can not be modified.
Inside the dict variable we see two values:
- ‘a_class_variable’: ‘a’
- ‘a_function’: <function Foo.a_function at 0x7fb358f8d6a8>
If we could modify Foo.__dict__ we could add methods and variables to class Foo at runtime. So we would want to do something like this.
But the mappingproxy object will not allow itself to be modified. So how can we do this ? With a metaclass.
What is a metaclass
In simple english a metaclass is a class which inherits off class type and allows one to modify the class’s mappingproxy. Here is a simple example.
And here is the output from the code.
Simple Explanation what is going on
The metaclass of a class in python 3 is set like this class Bar(object,metaclass = BarMetaClass). All Metaclasses are inherited off class type. All class type classes contain two constuctors which will be called when the type class is being created. These two constructors are.
- def __new__(cls, name, bases, dct) is called before the class is created. It must return the parent metaclass new method which in this case is type.__new__
- def __init__(cls, name, bases, dct): is called after the class is created. Somewhere in the function it much called the parent’s init function which in this case is type.__init__
Both of these functions container 4 arguments.
- cls : this refers to the metaclass itself. It is equivalent to self in a regular object method. So if the metaclass had other methods they could be called like this cls.somemethod(cls,arg). Remember metaclasses are only called when the class is being created. So in the above example, class Bar can not access the methods of class BarMetaClass.
- name : The name of the class being created. So in the above example the name is “Bar”.
- base : which classes it inherits off, in this case “Bar” inherits off “object”
- dct : this equivalent to Bar.dict except it is a regular python dictionary. This means we can modify it.
Since cls,name,base and dct are normal python structures, that means we can modify them before we call the parent’s new function. Before when we wattempted to modify Foo.dict python would not allow use to do this. Using a metaclass this is possible. It done in the new method of the metaclass. Here is a simple example.
When we run the above example we get the following output.
We can see that we have added a class variable and a class method to foo. Ofcourse in this case we could have done this in the class defintion of Foo, there was no need to use a metaclass, but it is just an example to show how it is done.
Solving our orignal problem
In our orignal problem we needed a client class which we needed to add methods to. Each method was basically the same except for the name of the rpc call. So we need a class or a function which creates a function which is similar to this
We notice that the “rpc_list_users” is the only thing changing. So my solution is the make a class like this.
There are other possible ways to do this. In this case we have created what I believe is known as a closure. We then put our function creater to use like this. wrapFunctionNames contains a list of rpc methods in the serve, our metaclass creates the required functions on the client.
The output of the above code looks like this.
This is just a simple example of how to use a metaclass to create class method at runtime. Once you understand the basics of metaclasses, then the more advanced tutorials start to make sense. In the above example we could have queried the server for a list of rpc functions it provides. Then we could have created the appropriate functions on the client using a metaclass. This is the magic that many python database libraries use. The query the database and then generate the classes at runtime using metaclasses.
The best way to learn about metaclasses is to use a good python console such as ipython or jupyter notebooks (formerly ipython notebooks). The advantage is that you can modify things at runtime and experiment with them.
All of the code for this post is in an ipython notebook located here.