Python: Decorating with class through descriptors

Update: If you find this article helpful, you may want to read the follow-up.

As a fairly new Python developer, my first attempt at decorators hit a snag: my simple class-based decorator failed when decorating a method. I got around the immediate problem by rewriting the decorator as a function. Yet the episode left me wondering if there were some way to fix the class-based decorator to work when applied to methods. I’ve found what seems like an elegant solution, and picked up a better understanding of decorators and descriptors in the process.

Here’s an example that illustrates the original problem. DebugTrace is the decorator class:

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Running this with Python 2.6 or 3.1 results in an error:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
Traceback (most recent call last):
  File "./DecoratorExample.py", line 31, in <module>
    greet()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
  File "./DecoratorExample.py", line 27, in greet
    g.hello()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
TypeError: hello() takes exactly 1 argument (0 given)

The output explains the problem. DebugTrace was instantiated only twice: once for Greeter.hello, and once for greet. That’s a reminder that decoration occurs during compile time, not run time. Accordingly, DebugTrace‘s reference to Greeter.hello represents an unbound function—it doesn’t reference any Greeter instances. So no ‘self’ argument was passed into the call to Greeter.hello; hence the TypeError.

All object-oriented languages that are worth knowing (and many that aren’t) allow container objects to redirect access to their contained objects. But of the languages that I’ve used, Python is unique in allowing object attributes to redirect calls that attempt to access them. Objects that implement this capability are called descriptors1. As we’ll soon see, function objects, which Python uses to implement methods, are descriptors.

A full description of descriptors would be too long to fit here. Here are the most important points for this post:

  1. When the interpreter reads a class attribute, and the attribute value is an object that has a __get__ method, then the return value of that __get__ method is used as the attribute’s value.
  2. Methods are class attributes.
  3. Any callable object can serve as a method.

Here’s a good guide to descriptors. There’s also a recent post by Guido van Rossum, Python’s BDFL, that provides good background material on the feature.

To see where descriptors come into play, let’s look at the calling sequences for different versions of Greeter.hello. Here’s the rough sequence before Greeter.hello was decorated:

  • The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
  • The value of the Greeter.hello attribute (a function object) has a __get__ method, making it a descriptor, so that __get__ method is invoked. It’s passed a reference to the Greeter instance (obj) through which Greeter.hello was called.
  • The function object’s __get__ method creates and returns another callable object, which we’ll refer to as a (bound) method object. The method object references both the Greeter.hello function and obj.
  • The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

When Greeter.hello is decorated with the DebugTracer class as shown above, a call to Greeter.hello runs more or less like this:

  • The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
  • The value of the Greeter.hello attribute is an instance of DebugTrace. This isn’t a function object, and it doesn’t have a __get__ method, but it does have a __call__ method. That __call__ method is invoked with the empty argument list.
  • DebugTrace.__call__ then calls Greeter.hello with the empty argument list.
  • Since Greeter.hello was looking for a single argument (self), rather than an empty argument list, a TypeError is raised.

To fix DebugTrace, I turned it into a descriptor class, adding a __get__ method that fills the same role as a function object’s __get__ method. However, this method binds the Greeter instance to the callable DebugTrace object.

import types
# ...
    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

Compare the new calling sequence for Greeter.hello to the sequence prior to decoration:

  • The interpreter finds an attribute named hello in the Greeter class object.
  • The value of the Greeter.hello attribute is an instance of DebugTrace, which is now a descriptor. DebugTrace.__get__ is called, with obj (the Greeter instance) passed as one of the arguments.
  • DebugTrace.__get__ creates and returns a method object. The method object references both the DebugTrace instance and obj.
  • The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls DebugTrace.__call__. That in turn calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

It’s worth noting that DebugTrace.__get__ is only invoked when accessing a DebugTrace object through an object’s class dictionary. Hence its presence has no effect on functions that aren’t methods, such as greet.

You can see the full, working example here (click on the “show source” link to view) :

import types

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Executing the new version gives the desired output:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
*** Greeter 1 says hello!
Calling hello
*** Greeter 2 says hello!
Credit Where Credit Is Due Dept: I’m not the first one to discover that a class-based decorator needs a __get__ method. For example, Ian Bicking wrote about a similar technique over a year and a half ago. However, Ian’s descriptor creates a new instance of the decorator class every time the method is invoked. I think the solution that I found—binding the original decorator instance to the method object—is different enough to be worth its own post.

For what it’s worth, I ran a simple performance test comparing Ian’s and my own class-based decorators, along with a function-based decorator. It showed no significant difference in performance among them. Apparently, the interpreter already optimizes these cases, which isn’t all that surprising.


1 I found the term “descriptor” to be somewhat confusing at first. A descriptor doesn’t really describe anything other than itself. To be fair, I don’t have any better suggestions. (“Redirector” ?) Naming is often one of the hardest challenges in software design, at least when it’s done right.

, ,

3 Comments

  • Mark Roberts says:

    Just as I sat down to figure this out I spotted your post. Thanks!

  • tem says:

    Or you could just use functions to define decorators. Don’t forget how easy it is to use functools.wrap as well.

    Don’t oop in the name of oop, that’s an oops.

    • Dan Breslau says:

      Yes, you can use functions to define decorators (I even noted that in the first paragraph.) And that is often the appropriate path to take.

      But my goal in this article (and its follow-up) wasn’t to show folks how to use a sledgehammer to kill a mosquito. (At least, not only that.) After I realized that I’d need to use descriptors to make a class-based decorator work, and figured out how to do so, I felt I’d reached a better understanding of both decorators and descriptors. My real goal was to present the two concepts in a way that would help others gain similar insights.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>