{"id":640,"date":"2010-07-06T10:16:18","date_gmt":"2010-07-06T14:16:18","guid":{"rendered":"http:\/\/www.outofwhatbox.com\/blog\/?p=640"},"modified":"2010-07-09T16:48:58","modified_gmt":"2010-07-09T20:48:58","slug":"python-decorating-with-class","status":"publish","type":"post","link":"https:\/\/www.outofwhatbox.com\/blog\/2010\/07\/python-decorating-with-class\/","title":{"rendered":"Python: Decorating with class through descriptors"},"content":{"rendered":"<div class=\"oowbbtw\"><strong>Update<\/strong>: If you find this article helpful, you may want to read the <a href=\"http:\/\/www.outofwhatbox.com\/blog\/2010\/07\/python-decorator-classes-on-the-edge\/\">follow-up<\/a>.<\/div>\n<p>As a fairly new Python developer, my first attempt at <a href=\"http:\/\/www.artima.com\/weblogs\/viewpost.jsp?thread=240808\">decorators<\/a> hit a snag: my simple class-based decorator failed when decorating a method. I got around the immediate problem by rewriting the decorator as a function. Yet the episode left me wondering if there were some way to fix the class-based decorator to work when applied to methods. I&#8217;ve found what seems like an elegant solution, and picked up a better understanding of decorators and descriptors in the process.<\/p>\n<p>Here&#8217;s an example that illustrates the original problem. <code>DebugTrace<\/code> is the decorator class:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nclass DebugTrace(object):\r\n    def __init__(self, f):\r\n        print(&quot;Tracing: {0}&quot;.format(f.__name__))\r\n        self.f = f\r\n\r\n    def __call__(self, *args, **kwargs):\r\n        print(&quot;Calling: {0}&quot;.format(self.f.__name__))\r\n        return self.f(*args, **kwargs)\r\n\r\n\r\nclass Greeter(object):\r\n    instances = 0\r\n\r\n    def __init__(self):\r\n        Greeter.instances += 1\r\n        self._inst = Greeter.instances\r\n\r\n    @DebugTrace\r\n    def hello(self):\r\n        print(&quot;*** Greeter {0} says hello!&quot;.format(self._inst))\r\n\r\n\r\n@DebugTrace\r\ndef greet():\r\n    g = Greeter()\r\n    g2 = Greeter()\r\n    g.hello()\r\n    g2.hello()\r\n\r\n\r\ngreet()\r\n<\/pre>\n<p>Running this with Python 2.6 or 3.1 results in an error:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nTracing: hello\r\nTracing: greet\r\nCalling greet\r\nCalling hello\r\nTraceback (most recent call last):\r\n  File &quot;.\/DecoratorExample.py&quot;, line 31, in &lt;module&gt;\r\n    greet()\r\n  File &quot;.\/DecoratorExample.py&quot;, line 8, in __call__\r\n    return self.f(*args, **kwargs)\r\n  File &quot;.\/DecoratorExample.py&quot;, line 27, in greet\r\n    g.hello()\r\n  File &quot;.\/DecoratorExample.py&quot;, line 8, in __call__\r\n    return self.f(*args, **kwargs)\r\nTypeError: hello() takes exactly 1 argument (0 given)\r\n<\/pre>\n<p>The output explains the problem. <code>DebugTrace<\/code> was instantiated only twice: once for <code>Greeter.hello<\/code>, and once for <code>greet<\/code>. That&#8217;s a reminder that decoration occurs during compile time, not run time. Accordingly, <code>DebugTrace<\/code>&#8216;s reference to <code>Greeter.hello<\/code> represents an <em>unbound<\/em> function\u00e2\u20ac\u201dit doesn&#8217;t reference any <code>Greeter<\/code> instances. So no &#8216;self&#8217; argument was passed into the call to <code>Greeter.hello<\/code>; hence the TypeError.<\/p>\n<p>All object-oriented languages that are worth knowing (and many that aren&#8217;t) allow container objects to redirect access to their contained objects. But of the languages that I&#8217;ve used, Python is unique in allowing object <em>attributes<\/em> to redirect calls that attempt to access them. Objects that implement this capability are called <a href=\"http:\/\/docs.python.org\/reference\/datamodel.html#descriptors\"><em>descriptors<\/em><\/a><a name=\"PythonDecDescNote1\" href=\"#PythonDecDesc1\"><sup>1<\/sup><\/a>. As we&#8217;ll soon see, <em>function objects<\/em>, which Python uses to implement methods, are descriptors.<\/p>\n<div class=\"oowbnotice\">A full description of descriptors would be too long to fit here. Here are the most important points for this post:<\/p>\n<ol>\n<li>When the interpreter reads a class attribute, and the attribute value is an object that has a <code>__get__<\/code> method, then the return value of that <code>__get__<\/code> method is used as the attribute&#8217;s value.<\/li>\n<li>Methods are class attributes.<\/li>\n<li>Any callable object can serve as a method.<\/li>\n<\/ol>\n<p>Here&#8217;s a good <a href=\"http:\/\/users.rcn.com\/python\/download\/Descriptor.htm\">guide to descriptors<\/a>. There&#8217;s also a recent post by Guido van Rossum, Python&#8217;s <a href=\"http:\/\/www.artima.com\/weblogs\/viewpost.jsp?thread=235725\">BDFL<\/a>, that provides <a href=\"http:\/\/python-history.blogspot.com\/2010\/06\/inside-story-on-new-style-classes.html\">good background material<\/a> on the feature.<\/div>\n<p>To see where descriptors come into play, let&#8217;s look at the calling sequences for different versions of <code>Greeter.hello<\/code>. Here&#8217;s the rough sequence <em>before<\/em> <code>Greeter.hello<\/code> was decorated:<\/p>\n<ul>\n<li>The interpreter searches for an attribute named <code>hello<\/code> on the <code>Greeter<\/code> instance, finding it in the <code>Greeter<\/code> class object.<\/li>\n<li>The value of the <code>Greeter.hello<\/code> attribute (a function object) has a <code>__get__<\/code> method, making it a descriptor, so that <code>__get__<\/code> method is invoked. It&#8217;s passed a reference to the <code>Greeter<\/code> instance (<code>obj<\/code>) through which <code>Greeter.hello<\/code> was called.<\/li>\n<li>The function object&#8217;s <code>__get__<\/code> method creates and returns another callable object, which we&#8217;ll refer to as a (bound) <em>method object<\/em>. The method object references both the <code>Greeter.hello<\/code> function and <code>obj<\/code>.<\/li>\n<li>The interpreter invokes the method object, passing it the arguments from the call to <code>Greeter.hello<\/code> (an empty list.) The method object then calls <code>Greeter.hello<\/code>, passing <code>obj<\/code> as the first argument, followed by the (empty) argument list.<\/li>\n<\/ul>\n<p>When <code>Greeter.hello<\/code> is decorated with the <code>DebugTracer<\/code> class as shown above, a call to <code>Greeter.hello<\/code> runs more or less like this:<\/p>\n<ul>\n<li>The interpreter searches for an attribute named <code>hello<\/code> on the <code>Greeter<\/code> instance, finding it in the <code>Greeter<\/code> class object.<\/li>\n<li>The value of the <code>Greeter.hello<\/code> attribute is an instance of <code>DebugTrace<\/code>. This isn&#8217;t a function object, and it doesn&#8217;t have a <code>__get__<\/code> method, but it <em>does<\/em> have a <code>__call__<\/code> method. That <code>__call__<\/code> method is invoked with the empty argument list.<\/li>\n<li><code>DebugTrace.__call__<\/code> then calls <code>Greeter.hello<\/code> with the empty argument list.<\/li>\n<li>Since <code>Greeter.hello<\/code> was looking for a single argument (<code>self<\/code>), rather than an empty argument list, a <code>TypeError<\/code> is raised.<\/li>\n<\/ul>\n<p>To fix <code>DebugTrace<\/code>, I turned it into a descriptor class, adding a <code>__get__<\/code> method that fills the same role as a function object&#8217;s <code>__get__<\/code> method. However, this method binds the <code>Greeter<\/code> instance to the callable <code>DebugTrace<\/code> object.<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nimport types\r\n# ...\r\n    def __get__(self, obj, ownerClass=None):\r\n        # Return a wrapper that binds self as a method of obj (!)\r\n        return types.MethodType(self, obj)\r\n<\/pre>\n<p>Compare the new calling sequence for <code>Greeter.hello<\/code> to the sequence prior to decoration:<\/p>\n<ul>\n<li>The interpreter finds an attribute named <code>hello<\/code> in the <code>Greeter<\/code> class object.<\/li>\n<li>The value of the <code>Greeter.hello<\/code> attribute is an instance of <code>DebugTrace<\/code>, which is now a descriptor. <code>DebugTrace.__get__<\/code> is called, with <code>obj<\/code> (the <code>Greeter<\/code> instance) passed as one of the arguments.<\/li>\n<li><code>DebugTrace.__get__<\/code> creates and returns a method object. The method object references both the <code>DebugTrace<\/code> instance and <code>obj<\/code>.<\/li>\n<li>The interpreter invokes the method object, passing it the arguments from the call to <code>Greeter.hello<\/code> (an empty list.) The method object then calls <code>DebugTrace.__call__<\/code>. That in turn calls <code>Greeter.hello<\/code>, passing <code>obj<\/code> as the first argument, followed by the (empty) argument list.<\/li>\n<\/ul>\n<p>It&#8217;s worth noting that <code>DebugTrace.__get__<\/code> is <em>only<\/em> invoked when accessing a <code>DebugTrace<\/code> object through an object&#8217;s class dictionary. Hence its presence has no effect on functions that aren&#8217;t methods, such as <code>greet<\/code>.<a name=\"WhereILeftOff\"><\/p>\n<p><\/a>You can see the full, working example here <em>(click on the &#8220;show source&#8221; link to view)<\/em> :<\/p>\n<pre class=\"brush: python; collapse: true; title: ; notranslate\" title=\"\">\r\nimport types\r\n\r\nclass DebugTrace(object):\r\n    def __init__(self, f):\r\n        print(&quot;Tracing: {0}&quot;.format(f.__name__))\r\n        self.f = f\r\n\r\n    def __get__(self, obj, ownerClass=None):\r\n        # Return a wrapper that binds self as a method of obj (!)\r\n        return types.MethodType(self, obj)\r\n\r\n    def __call__(self, *args, **kwargs):\r\n        print(&quot;Calling: {0}&quot;.format(self.f.__name__))\r\n        return self.f(*args, **kwargs)\r\n\r\n\r\nclass Greeter(object):\r\n    instances = 0\r\n\r\n    def __init__(self):\r\n        Greeter.instances += 1\r\n        self._inst = Greeter.instances\r\n\r\n    @DebugTrace\r\n    def hello(self):\r\n        print(&quot;*** Greeter {0} says hello!&quot;.format(self._inst))\r\n\r\n\r\n@DebugTrace\r\ndef greet():\r\n    g = Greeter()\r\n    g2 = Greeter()\r\n    g.hello()\r\n    g2.hello()\r\n\r\n\r\ngreet()\r\n<\/pre>\n<p>Executing the new version gives the desired output:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nTracing: hello\r\nTracing: greet\r\nCalling greet\r\nCalling hello\r\n*** Greeter 1 says hello!\r\nCalling hello\r\n*** Greeter 2 says hello!\r\n<\/pre>\n<div class=\"oowbbtw\"><strong>Credit Where Credit Is Due Dept<\/strong>: I&#8217;m not the first one to discover that a class-based decorator needs a <code>__get__ <\/code> method. For example, <a href=\"http:\/\/blog.ianbicking.org\/2008\/10\/24\/decorators-and-descriptors\/\">Ian Bicking<\/a> wrote about a similar technique over a year and a half ago. However, Ian&#8217;s descriptor creates a new instance of the decorator class every time the method is invoked. I think the solution that I found\u00e2\u20ac\u201dbinding the original decorator instance to the method object\u00e2\u20ac\u201dis different enough to be worth its own post.<\/p>\n<p>For what it&#8217;s worth, I ran a simple performance test comparing Ian&#8217;s and my own class-based decorators, along with a function-based decorator. It showed no significant difference in performance among them. Apparently, the interpreter already optimizes these cases, which isn&#8217;t all that surprising.<\/p><\/div>\n<hr \/>\n<p><a href=\"#PythonDecDescNote1\" name=\"PythonDecDesc1\"><sup>1<\/sup><\/a> I found the term &#8220;descriptor&#8221; to be somewhat confusing at first. A descriptor doesn&#8217;t really <em>describe<\/em> anything other than itself. To be fair, I don&#8217;t have any better suggestions. (&#8220;<em>Redirector<\/em>&#8221; ?) Naming is often one of the hardest challenges in software design, at least when it&#8217;s done right.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Developing a decorator class in Python leads to an enlightening look at Python descriptors.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[30],"tags":[50,31,32],"_links":{"self":[{"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/posts\/640"}],"collection":[{"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/comments?post=640"}],"version-history":[{"count":12,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/posts\/640\/revisions"}],"predecessor-version":[{"id":657,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/posts\/640\/revisions\/657"}],"wp:attachment":[{"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/media?parent=640"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/categories?post=640"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.outofwhatbox.com\/blog\/wp-json\/wp\/v2\/tags?post=640"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}