I was wrapping a client rpc which was calling functions on a server rpc. The code looked like this.

class RpcClient(object): 
    def rpc_list_users(self,*args,**kwArgs):
        result = self.call("rpc_list_users",args,kwArgs)
        return result

    def rpc_get_user(self,*args,**kwArgs):
        result = self.call("rpc_get_user",args,kwArgs)
        return result

The are over 20 functions on the RPC server, I didn’t feel like typing out all of the code for these functions. Also what if I added more rpc functions on the server ?. The solution in this case was metaclasses.

Note this demo uses python 3, the syntax for metaclasses with slightly different.

Understanding python classes and objects

To understand how to use metaclasses, one must understand how python classes and objects work behind the scenes. A great way to explore this is with ipython or ipython notebooks which is part of the juypter project.

Simply put classes and objects are nothing more then dictionaries. You can verify this yourself using your python interpretor.

class Foo(object):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)
        
pp("Foo.__dict__")
pp(Foo.__dict__)
pp("pretty Foo")
pp("Pretty Foo.__dict__")
pp(dict(Foo.__dict__)) #convert it to a dictionary so pp dumps it out nice

Output from running above code.

'Foo.__dict__'
mappingproxy({'__doc__': None, '__weakref__': <attribute '__weakref__' of 'Foo' objects>, 'a_class_variable': 'a', 'a_function': <function Foo.a_function at 0x7fb358f8d6a8>, '__dict__': <attribute '__dict__' of 'Foo' objects>, '__module__': '__main__'})
'pretty Foo'
'Pretty Foo.__dict__'
{'__dict__': <attribute '__dict__' of 'Foo' objects>,
 '__doc__': None,
 '__module__': '__main__',
 '__weakref__': <attribute '__weakref__' of 'Foo' objects>,
 'a_class_variable': 'a',
 'a_function': <function Foo.a_function at 0x7fb358f8d6a8>}

Inside all python classes, there is a variable called __dict__. If we print it out, we see that type mappingproxy which if we look inside the python docs says that it is type of dictionary which can not be modified.

Inside the dict variable we see two values:

• ‘a_class_variable’: ‘a’
• ‘a_function’: <function Foo.a_function at 0x7fb358f8d6a8>

If we could modify Foo.__dict__ we could add methods and variables to class Foo at runtime. So we would want to do something like this.

def my_function(self):
    pass

Foo.__dict__["my_function"] = my_function
Foo.__dict__["my_variable"] = 99 

But the mappingproxy object will not allow itself to be modified. So how can we do this ? With a metaclass.

What is a metaclass

In simple english a metaclass is a class which inherits off class type and allows one to modify the class’s mappingproxy. Here is a simple example.

class BarMetaClass(type):
    def __new__(cls, name,bases, dct):
        #this is called before the class has been created
        pp("__new__ \n cls => %r \n name => %r \n bases => %r \n dct => %r" % (cls,name,bases,dct))
        #the parent __new__ metaclass has to be returned
        return type.__new__(cls,name, bases, dct)
    
    def __init__(cls, name,bases, dct):
        #the parent __init__ metaclass method has to be called
        type.__init__(cls,name, bases, dct)
        #this is called after the class has been created
        pp("__init__ \n cls => %r \n name => %r \n bases => %r \n dct => %r" % (cls,name,bases,dct))
        

class Bar(object,metaclass = BarMetaClass):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)

And here is the output from the code.

('__new__ \n'
 " cls => <class '__main__.BarMetaClass'> \n"
 " name => 'Bar' \n"
 " bases => (<class 'object'>,) \n"
 " dct => {'__qualname__': 'Bar', 'a_class_variable': 'a', '__module__': "
 "'__main__', 'a_function': <function Bar.a_function at 0x7fb358f43598>}")
('__init__ \n'
 " cls => <class '__main__.Bar'> \n"
 " name => 'Bar' \n"
 " bases => (<class 'object'>,) \n"
 " dct => {'__qualname__': 'Bar', 'a_class_variable': 'a', '__module__': "
 "'__main__', 'a_function': <function Bar.a_function at 0x7fb358f43598>}")

Simple Explanation what is going on

The metaclass of a class in python 3 is set like this class Bar(object,metaclass = BarMetaClass). All Metaclasses are inherited off class type. All class type classes contain two constuctors which will be called when the type class is being created. These two constructors are.

• def __new__(cls, name, bases, dct) is called before the class is created. It must return the parent metaclass new method which in this case is type.__new__
• def __init__(cls, name, bases, dct): is called after the class is created. Somewhere in the function it much called the parent’s init function which in this case is type.__init__

Both of these functions container 4 arguments.

• cls : this refers to the metaclass itself. It is equivalent to self in a regular object method. So if the metaclass had other methods they could be called like this cls.somemethod(cls,arg). Remember metaclasses are only called when the class is being created. So in the above example, class Bar can not access the methods of class BarMetaClass.
• name : The name of the class being created. So in the above example the name is “Bar”.
• base : which classes it inherits off, in this case “Bar” inherits off “object”
• dct : this equivalent to Bar.dict except it is a regular python dictionary. This means we can modify it.

Since cls,name,base and dct are normal python structures, that means we can modify them before we call the parent’s new function. Before when we wattempted to modify Foo.dict python would not allow use to do this. Using a metaclass this is possible. It done in the new method of the metaclass. Here is a simple example.

def my_function(self):
    return "from my function"

class FooMetaClass(type):
    def __new__(cls, name,bases, dct):
        dct["my_function"] = my_function
        dct["my_variable"] = "my_variable"
        #the parent __new__ metaclass has to be returned
        return type.__new__(cls,name, bases, dct)        

class Foo(object,metaclass = FooMetaClass):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)
        
pp("testing out foo")
pp("Foo.my_variable => %r" % Foo.my_variable)
foo = Foo()
pp("foo.my_function() => %r" % foo.my_function())content

When we run the above example we get the following output.

'testing out foo'
"Foo.my_variable => 'my_variable'"
"foo.my_function() => 'from my function'"

We can see that we have added a class variable and a class method to foo. Ofcourse in this case we could have done this in the class defintion of Foo, there was no need to use a metaclass, but it is just an example to show how it is done.

Solving our orignal problem

In our orignal problem we needed a client class which we needed to add methods to. Each method was basically the same except for the name of the rpc call. So we need a class or a function which creates a function which is similar to this

def rpc_list_users(self,*args,**kwArgs):
    result = await self.call("rpc_list_users",args,kwArgs)
    return result

We notice that the “rpc_list_users” is the only thing changing. So my solution is the make a class like this.

def CreateRpcFunction(name):
    def wrappedFunction(self,*args,**kwArgs):
        self.call(name,*args,**kwArgs)
        return name
    return wrappedFunction

There are other possible ways to do this. In this case we have created what I believe is known as a closure. We then put our function creater to use like this. wrapFunctionNames contains a list of rpc methods in the serve, our metaclass creates the required functions on the client.

wrapFunctionNames = ["rpc_list_users","rpc_get_user","rpc_set_user"]

class RpcClientMeta(type):
    def __new__(cls,name,bases, dct):
        for name in wrapFunctionNames :
            dct[name] = CreateRpcFunction(name)
        return type.__new__(cls,name,bases, dct)
        

class RpcClient(object,metaclass = RpcClientMeta):
        
    def call(self,name,*args,**kwArgs):
        """
        simulates calling the rpc on the server.
        """
        pp("called name %r args %r kwArgs %r" % (name,args,kwArgs))
        return name
        
rpcClient = RpcClient()
pp("dump rpc functions")
pp([name for name in dir(rpcClient) if name.startswith("rpc_")])

pp("rpcClient.rpc_get_user() => %r" % rpcClient.rpc_get_user())
pp("rpcClient.rpc_list_user() => %r" % rpcClient.rpc_list_users(startsWith = "Fred"))

The output of the above code looks like this.

'dump rpc functions'
['rpc_get_user', 'rpc_list_users', 'rpc_set_user']
"called name 'rpc_get_user' args () kwArgs {}"
"rpcClient.rpc_get_user() => 'rpc_get_user'"
"called name 'rpc_list_users' args () kwArgs {'startsWith': 'Fred'}"
"rpcClient.rpc_list_user() => 'rpc_list_users'"

Summary

This is just a simple example of how to use a metaclass to create class method at runtime. Once you understand the basics of metaclasses, then the more advanced tutorials start to make sense. In the above example we could have queried the server for a list of rpc functions it provides. Then we could have created the appropriate functions on the client using a metaclass. This is the magic that many python database libraries use. The query the database and then generate the classes at runtime using metaclasses.

The best way to learn about metaclasses is to use a good python console such as ipython or jupyter notebooks (formerly ipython notebooks). The advantage is that you can modify things at runtime and experiment with them.

All of the code for this post is in an ipython notebook located here.