Android: AutoCompleteTextView, SQLite, and Dependent Fields

Update: Subclassing from SimpleCursorAdapter isn’t necessary after all. Please see my follow-up post for a simpler way of using AutoCompleteTextView together with database queries.

I recently integrated an AutoCompleteTextView into a prototype Android application that I’m developing. There isn’t exactly a wealth of documentation on the Web describing AutoCompleteTextView, and what I’ve found didn’t match the scenario I had in mind. Specifically:

  • The choices came from a database, so I needed to use a CursorAdapter; and
  • The AutoCompleteTextView had some dependent fields; changes to the value in the AutoCompleteTextView needed to be reflected in those dependent fields.

For example, let’s use a AutoCompleteTextView for choosing states. When the user chooses a state, I want to update a TextView to show the capital of the chosen state.

AutoCompleteTextView, with no selectionAutoCompleteTextView, selecting Massachusetts

First the form appears with no value in the State Capital field (left); then a state is chosen from the list (right.)

AutoCompleteTextView, updated with capital of Massachusetts.
After a state has been chosen, its capital is also displayed.

Adapting SQLite to AutoCompleteTextView

The documentation for AutoCompleteTextView says that it obtains suggestions from a data adapter. This should be familiar territory for anyone who’s used ListView, which also obtains its list items from a data adapter. Not surprisingly, AutoCompleteTextView uses a ListView internally.

However, ListView and AutoCompleteTextView require different interactions with their adapters. A standard ListView will typically need to fetch the list of items once, and only once. On the other hand, AutoCompleteTextView issues a query for matching choices whenever the user enters new text into the field. (In fact, the AutoCompleteTextView won’t issue a query until the user has typed some text. Hence the CursorAdapter‘s constructor can take null for the Cursor parameter.)

SimpleCursorAdapter wasn’t designed for this kind of usage pattern; in particular, it doesn’t know how you want to query for matching choices. So, you must derive from SimpleCursorAdapter, or from its base class, CursorAdapter. Your subclass must supply two methods that are essentially stubbed out in CursorAdapter:

  • convertToString, which supplies the String value that is entered into the AutoCompleteTextView when a choice is made; and
  • runQueryOnBackgroundThread. In this method, you invoke a query for choices that match the text entered by the user, and return a Cursor that provides the results.

Let’s take a closer look at implementing the derived CursorAdapter class.

public CharSequence convertToString(Cursor cursor)

This method receives the Cursor positioned to a specific row, and returns a String label or representation for that row.

Incidentally, if you derive from SimpleCursorAdapter without overriding convertToString, the choices shown in the drop-down will look fine. However, when an item in the list is chosen, the string entered in the AutoCompleteTextView field will be gibberish like “android.database.sqlite.SQLiteCursor@43e8b260“.

(This shows that SimpleCursorAdapter bypasses convertToString when building the choice list for an AutoCompleteTextView. Instead, it uses the from and to arguments in its constructor to pull the label text directly from the database. It also demonstrates that SimpleCursorAdapter does rely on convertToString when it comes time to update the field. This inconsistency seems like a bug in SimpleCursorAdapter.)

If you derive directly from CursorAdapter without implementing convertToString, then the behavior is consistent: you get the gibberish in the list and in the field value.

Perhaps convertToString should have been declared abstract, since the base class implementation doesn’t seem to be very helpful.

public Cursor runQueryOnBackgroundThread(CharSequence constraint)

Just to clarify, “On Background Thread” means that this method is called in a background thread — it does not mean that you have to spawn that thread within this method.

This method runs a query to get choices that match the input. It’s up to the application to determine what constitutes a match. (In this example, I’m searching for states whose names begin with the given letters.)

The constraint parameter is the input that the user has typed in. It may be null if the user has typed, and then erased, input text. In this case, all results should be returned.

Note that you must not call changeCursor within this method, because changeCursor is not thread-safe. The UI will invoke changeCursor from the main thread after this method returns the new Cursor.

But wait, there’s more!

It might seem… well, simpler, to derive the new Adapter class from SimpleCursorAdapter. However, deriving from CursorAdapter requires only two additional methods to be implemented, newView and bindView, both of which are straightforward. Hence for this example, I’m using CursorAdapter as the base class. Later, I’ll describe the changes needed to use SimpleCursorAdapter as the base.

public View newView(Context context, Cursor cursor, ViewGroup parent)

newView returns a newly constructed View suitable for displaying a single item in the choice list. The usual method for doing this is to use the Inflater, which is given the resource ID of a layout that describes the new View. It’s not necessary to create your own layout file; Android provides a standard layout that’s appropriate for this purpose, identified by android.R.layout.simple_dropdown_item_1line.

Note: Many examples of newView on the web call setText on the view before returning. This unnecessarily duplicates the code in bindView. Nor is it necessary to invoke bindView explicitly: after we return the new view, bindView will be invoked to display the value.

public void bindView(View view, Context context, Cursor cursor)

This is called to display the label for the current row. If newView uses android.R.layout.simple_dropdown_item_1line for layout, then bindView will be called with a TextView as the View argument.

Updating dependent fields

To update dependent fields, start by creating a class that implements OnItemClickListener. (Here, it’s the Adapter class; it could also be your Activity, or some other class.) This interface defines a callback that’s invoked when the user makes a choice. The setOnItemClickListener method in AutoCompleteTextView associates the listener with the field.

The callback method, onItemClick, receives a position parameter that identifies the row number within the result set, starting from 0. For example, if the third state, “Arizona”, is chosen, the value of position will be 2. There’s also an AdapterView<?> parameter, whose purpose is a bit obscure. This is the ListView belonging to the AutoCompleteTextView. (This may be the only situation in which that ListView is directly exposed to the caller.)

We can then get the cursor from the AdapterView<?> parameter, and position it to the correct row, by calling getItemAtPosition. Note that getItemAtPosition returns an Object, so a cast is necessary. (It doesn’t seem to be documented anywhere that listView.getItemAtPosition will return a Cursor in this case.)

In the full example, this logic is seen in the line:

Cursor cursor = (Cursor) listView.getItemAtPosition(position)

The result would probably be the same if we replaced the above line with:

            Cursor cursor = getCursor();
            cursor.moveToPosition(position);

I went with the first approach because: 1) It’s (arguably) more in keeping with the Adapter’s contract. (There doesn’t seem to be any guarantee that changeCursor will be called by the main thread; without that call, getCursor() certainly won’t work.) And 2) it’s simpler, if by “simpler” we mean one less line of code.

The code

Now let’s have a look at the code. Here’s the Activity class, SelectState, with its nested Adapter class, ItemAutoTextAdapter. (The other class needed for this example, AutoCompleteDbAdapter, is available below.)

ZIP files are available for the full example, using either CursorAdapter and SimpleCursorAdapter as the base classes.
package org.oowb.AutoCompleteExample;

import android.app.Activity;
import android.content.Context;
import android.database.Cursor;
import android.os.Bundle;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.AdapterView;
import android.widget.AutoCompleteTextView;
import android.widget.Button;
import android.widget.CursorAdapter;
import android.widget.TextView;

/**
 * A simple Android Activity to demonstrate: 
 * 
 * 1) How to use an AutoCompleteTextView with a CursorAdapter
 * 
 * 2) How to access the cursor row for the user's choice, to obtain 
 *    additional data from that row when an item is selected.
 * 
 * @author Dan Breslau
 * 
 */
public class SelectState extends Activity {
    /**
     * Specializes CursorAdapter to supply choices to a AutoCompleteTextView.
     * Also implements OnItemClickListener to be notified when a choice is made,
     * and uses the choice to update other fields on the Activity form.
     */
    class ItemAutoTextAdapter extends CursorAdapter
            implements android.widget.AdapterView.OnItemClickListener {

        private AutoCompleteDbAdapter mDbHelper;

        /**
         * Constructor. Note that no cursor is needed when we create the
         * adapter. Instead, cursors are created on demand when completions are
         * needed for the field. (see
         * {@link ItemAutoTextAdapter#runQueryOnBackgroundThread(CharSequence)}.)
         * 
         * @param dbHelper
         *            The AutoCompleteDbAdapter in use by the outer class
         *            object.
         */
        public ItemAutoTextAdapter(AutoCompleteDbAdapter dbHelper) {
            // Call the CursorAdapter constructor with a null Cursor.
            super(SelectState.this, null);
            mDbHelper = dbHelper;
        }

        /**
         * Invoked by the AutoCompleteTextView field to get completions for the
         * current input.
         * 
         * NOTE: If this method either throws an exception or returns null, the
         * Filter class that invokes it will log an error with the traceback,
         * but otherwise ignore the problem. No choice list will be displayed.
         * Watch those error logs!
         * 
         * @param constraint
         *            The input entered thus far. The resulting query will
         *            search for states whose name begins with this string.
         * @return A Cursor that is positioned to the first row (if one exists)
         *         and managed by the activity.
         */
        @Override
        public Cursor runQueryOnBackgroundThread(CharSequence constraint) {
            if (getFilterQueryProvider() != null) {
                return getFilterQueryProvider().runQuery(constraint);
            }

            Cursor cursor = mDbHelper.getMatchingStates(
                    (constraint != null ? constraint.toString() : null));

            return cursor;
        }

        /**
         * Called by the AutoCompleteTextView field to get the text that will be
         * entered in the field after a choice has been made.
         * 
         * @param Cursor
         *            The cursor, positioned to a particular row in the list.
         * @return A String representing the row's text value. (Note that this
         *         specializes the base class return value for this method,
         *         which is {@link CharSequence}.)
         */
        @Override
        public String convertToString(Cursor cursor) {
            final int columnIndex = cursor.getColumnIndexOrThrow("state");
            final String str = cursor.getString(columnIndex);
            return str;
        }

        /**
         * Called by the ListView for the AutoCompleteTextView field to display
         * the text for a particular choice in the list.
         * 
         * @param view
         *            The TextView used by the ListView to display a particular
         *            choice.
         * @param context
         *            The context (Activity) to which this form belongs;
         *            equivalent to {@code SelectState.this}.
         * @param cursor
         *            The cursor for the list of choices, positioned to a
         *            particular row.
         */
        @Override
        public void bindView(View view, Context context, Cursor cursor) {
            final String text = convertToString(cursor);
            ((TextView) view).setText(text);
        }

        /**
         * Called by the AutoCompleteTextView field to display the text for a
         * particular choice in the list.
         * 
         * @param context
         *            The context (Activity) to which this form belongs;
         *            equivalent to {@code SelectState.this}.
         * @param cursor
         *            The cursor for the list of choices, positioned to a
         *            particular row.
         * @param parent
         *            The ListView that contains the list of choices.
         * 
         * @return A new View (really, a TextView) to hold a particular choice.
         */
        @Override
        public View newView(Context context, Cursor cursor, ViewGroup parent) {
            final LayoutInflater inflater = LayoutInflater.from(context);
            final View view =
                    inflater.inflate(android.R.layout.simple_dropdown_item_1line,
                            parent, false);
 
           return view;
        }

        /**
         * Called by the AutoCompleteTextView field when a choice has been made
         * by the user.
         * 
         * @param listView
         *            The ListView containing the choices that were displayed to
         *            the user.
         * @param view
         *            The field representing the selected choice
         * @param position
         *            The position of the choice within the list (0-based)
         * @param id
         *            The id of the row that was chosen (as provided by the _id
         *            column in the cursor.)
         */
        @Override
        public void onItemClick(AdapterView<?> listView, View view, int position, long id) {
            // Get the cursor, positioned to the corresponding row in the result set
            Cursor cursor = (Cursor) listView.getItemAtPosition(position);

            // Get the state's capital from this row in the database.
            String capital = cursor.getString(cursor.getColumnIndexOrThrow("capital"));

            // Update the parent class's TextView
            mStateCapitalView.setText(capital);
        }
    }

    private TextView mStateCapitalView;
    private AutoCompleteTextView mStateNameView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        AutoCompleteDbAdapter dbHelper = new AutoCompleteDbAdapter(this);
        setContentView(R.layout.selectstate);
        Button confirmButton = (Button) findViewById(R.id.confirm);
        confirmButton.setOnClickListener(new View.OnClickListener() {
            public void onClick(View view) {
                setResult(RESULT_OK);
                finish();
            }
        });

        mStateCapitalView = (TextView) findViewById(R.id.state_capital);
        mStateNameView = (AutoCompleteTextView) findViewById(R.id.state_name);

        // Create an ItemAutoTextAdapter for the State Name field,
        // and set it as the OnItemClickListener for that field.
        ItemAutoTextAdapter adapter = this.new ItemAutoTextAdapter(dbHelper);
        mStateNameView.setAdapter(adapter);
        mStateNameView.setOnItemClickListener(adapter);
    }
}


Here’s the AutoCompleteDbAdapter class. It’s less interesting for this example, so the code is hidden by default. Click on the “show source” link to view it.

package org.oowb.AutoCompleteExample;

import android.app.Activity;
import android.content.ContentValues;
import android.content.Context;
import android.database.Cursor;
import android.database.SQLException;
import android.database.sqlite.SQLiteDatabase;
import android.database.sqlite.SQLiteOpenHelper;
import android.util.Log;

/**
 * Simple database access helper class.
 * 
 * @author Dan Breslau
 */
public class AutoCompleteDbAdapter {
    /**
     * List of states and capitals.
     */
    private static final String[][] States = {
            { "Alabama", "Montgomery" },
            { "Alaska", "Juneau" },
            { "Arizona", "Phoenix" },
            { "Arkansas", "Little Rock" },
            { "California", "Sacramento" },
            { "Colorado", "Denver" },
            { "Confusion", "\"C\"" },
            { "Connecticut", "Hartford" },
            { "Delaware", "Dover" },
            { "Florida", "Tallahassee" },
            { "Georgia", "Atlanta" },
            { "Hawaii", "Honolulu" },
            { "Idaho", "Boise" },
            { "Illinois", "Springfield" },
            { "Indiana", "Indianapolis" },
            { "Iowa", "Des Moines" },
            { "Kansas", "Topeka" },
            { "Kentucky", "Frankfort" },
            { "Louisiana", "Baton Rouge" },
            { "Maine", "Augusta" },
            { "Maryland", "Annapolis" },
            { "Massachusetts", "Boston" },
            { "Michigan", "Lansing" },
            { "Minnesota", "St. Paul" },
            { "Mississippi", "Jackson" },
            { "Missouri", "Jefferson City" },
            { "Montana", "Helena" },
            { "Nebraska", "Lincoln" },
            { "Nevada", "Carson City" },
            { "New Hampshire", "Concord" },
            { "New Jersey", "Trenton" },
            { "New Mexico", "Santa Fe" },
            { "New York", "Albany" },
            { "North Carolina", "Raleigh" },
            { "North Dakota", "Bismarck" },
            { "Ohio", "Columbus" },
            { "Oklahoma", "Oklahoma City" },
            { "Oregon", "Salem" },
            { "Pennsylvania", "Harrisburg" },
            { "Rhode Island", "Providence" },
            { "South Carolina", "Columbia" },
            { "South Dakota", "Pierre" },
            { "Tennessee", "Nashville" },
            { "Texas", "Austin" },
            { "Utah", "Salt Lake City" },
            { "Vermont", "Montpelier" },
            { "Virginia", "Richmond" },
            { "Washington", "Olympia" },
            { "West Virginia", "Charleston" },
            { "Wisconsin", "Madison" },
            { "Wyoming", "Cheyenne" }
    };

    private static final String DATABASE_NAME = "capitals";
    private static final String TABLE_NAME = "state";
    private static final int DATABASE_VERSION = 1;

    private class DatabaseHelper extends SQLiteOpenHelper {

        DatabaseHelper(Context context) {
            super(context, DATABASE_NAME, null, DATABASE_VERSION);
        }

        @Override
        public void onCreate(SQLiteDatabase db) {
            final String DATABASE_CREATE_STATES =
                    "create table " + TABLE_NAME
                            + "(_id integer primary key autoincrement"
                            + ", state text not null"
                            + ", capital text not null)";

            db.execSQL(DATABASE_CREATE_STATES);
            populateWithData(db);
        }

        @Override
        public void onUpgrade(SQLiteDatabase db, int old, int new) {
            db.execSQL("DROP TABLE IF EXISTS " + TABLE_NAME);
            onCreate(db);
        }
    }

    private DatabaseHelper mDbHelper;
    private SQLiteDatabase mDb;
    private final Activity mActivity;

    /**
     * Constructor - takes the context to allow the database to be
     * opened/created
     * 
     * @param activity
     *            the Activity that is using the database
     */
    public AutoCompleteDbAdapter(Activity activity) {
        this.mActivity = activity;
        mDbHelper = this.new DatabaseHelper(activity);
        mDb = mDbHelper.getWritableDatabase();
    }

    /**
     * Closes the database.
     */
    public void close() {
        mDbHelper.close();
    }

    /**
     * Return a Cursor that returns all states (and their state capitals) where
     * the state name begins with the given constraint string.
     * 
     * @param constraint
     *            Specifies the first letters of the states to be listed. If
     *            null, all rows are returned.
     * @return Cursor managed and positioned to the first state, if found
     * @throws SQLException
     *             if query fails
     */
    public Cursor getMatchingStates(String constraint) throws SQLException {

        String queryString =
                "SELECT _id, state, capital FROM " + TABLE_NAME;

        if (constraint != null) {
            // Query for any rows where the state name begins with the
            // string specified in constraint.
            //
            // NOTE:
            // If wildcards are to be used in a rawQuery, they must appear
            // in the query parameters, and not in the query string proper.
            // See http://code.google.com/p/android/issues/detail?id=3153
            constraint = constraint.trim() + "%";
            queryString += " WHERE state LIKE ?";
        }
        String params[] = { constraint };

        if (constraint == null) {
            // If no parameters are used in the query,
            // the params arg must be null.
            params = null;
        }
        try {
            Cursor cursor = mDb.rawQuery(queryString, params);
            if (cursor != null) {
                this.mActivity.startManagingCursor(cursor);
                cursor.moveToFirst();
                return cursor;
            }
        }
        catch (SQLException e) {
            Log.e("AutoCompleteDbAdapter", e.toString());
            throw e;
        }

        return null;
    }

    /**
     * Populates the database with data on states and state capitals.
     * 
     * @param db
     *            The database to be populated; must have the appropriate table
     *            ("state") and columns ("state" and "values") already set up.
     */
    private void populateWithData(SQLiteDatabase db) {
        try {
            db.beginTransaction();
            ContentValues values = new ContentValues();
            // Populate the database with the state/capital city
            // pairs found in the States array.
            for (String[] s : States) {
                values.put("state", s[0]);
                values.put("capital", s[1]);

                db.insert(TABLE_NAME, "state", values);
            }
            db.setTransactionSuccessful();
        }
        finally {
            db.endTransaction();
        }
    }
}

Using SimpleCursorAdapter

As mentioned above, changing the Adapter’s base class to SimpleCursorAdapter is fairly simple. Here’s a summary of the changes needed:

  • The SimpleCursorAdapter constructor requires a resource ID for the layout used for items in the list. As before, we are using android.R.layout.simple_dropdown_item_1line for this layout.
  • The constructor for SimpleCursorAdapter also requires parameters named from and to. These are arrays indicating which column(s) in the database hold the relevant data, and which field(s) in the View will display that data.
  • When the SimpleCursorAdapter is used to support an AutoCompleteTextView, only one column/field pair is used. The column, of course, is the column that supplies the names (here, “state”.) But how do we identify the field in the to array? Help comes to us through android.R.id.text1, a resource that is used to identify the first (and only) TextField within the standard item layout (android.R.layout.simple_dropdown_item_1line.)
    This results in the following definitions for the from and to parameters. (I defined these in the outer SelectState class, because an inner class isn’t allowed to have static data.)
        final static int[] to = new int[] { android.R.id.text1 };
        final static String[] from = new String[] { "state" };
    
  • The constructor is changed to call its superclass as follows:
  • super(SelectState.this, 
            android.R.layout.simple_dropdown_item_1line, 
            null, from, to);
    
  • The newView and bindView methods can be removed.

Python: Decorator Classes On The Edge

OK, I cheated.

In yesterday’s post on writing decorator classes that decorate methods, I left out two edge cases that can’t be completely ignored: static methods and class methods.

To illustrate, I’ll start where I left off yesterday, adding a decorated class method and a decorated static method to the example:

import types

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))

    @DebugTrace
    @classmethod
    def classHello(cls, to):
        print("*** The {0} class says hello to {1}".format(cls.__name__, to))

    @DebugTrace
    @staticmethod
    def staticHello(to):
        print("*** Something says hello to " + to)


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()
    Greeter.staticHello("you")
    Greeter.classHello("everyone")

greet()

Running this gives an error:


Tracing: hello
Traceback (most recent call last):
  File "DecoratorExample.py", line 17, in <module>
    class Greeter(object):
  File "DecoratorExample.py", line 29, in Greeter
    @classmethod
  File "DecoratorExample.py", line 5, in __init__
    print("Tracing: {0}".format(f.__name__))
AttributeError: 'classmethod' object has no attribute '__name__'

Just for this example, I’ll try removing the “Tracing” print call; but still no joy:

Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Traceback (most recent call last):
  File "DecoratorExample.py", line 48, in <module>
    greet()
  File "DecoratorExample.py", line 14, in __call__
    return self.f(*args, **kwargs)
  File "DecoratorExample.py", line 45, in greet
    Greeter.staticHello("you")
  File "DecoratorExample.py", line 10, in __get__
    return types.MethodType(self, obj)
TypeError: self must not be None

The essential problem is that class methods and static methods are not callable.1 There’s an easy enough workaround: always use @staticmethod or @classmethod as the outermost (i.e., last) decorator in a sequence, as in:

    @classmethod
    @DebugTrace
    def classHello(cls, to):
        print("*** The Greeter class says hello to " + to)

    @staticmethod
    @DebugTrace
    def staticHello(to):
        print("*** Something says hello to " + to)

That produces the desired result:

Tracing: hello
Tracing: classHello
Tracing: staticHello
Tracing: greet
Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Calling: staticHello
*** Something says hello to you
Calling: classHello
*** The Greeter class says hello to everyone

But suppose we really, really need to decorate an already-decorated classmethod or staticmethod. The key lies again in the descriptor protocol.

First, we need to modify the decorator’s __init__ method. (Note that the only reason that we need to modify __init__ is to find the name of the classmethod or staticmethod that’s being decorated. If we didn’t produce the “Tracing:” output, we could leave __init__ alone.)

The new __init__ method detects whether the passed “function” has a __call__ method. If it doesn’t, then it’s reasonable to assume that it’s a classmethod or a staticmethod. Calling the object’s __get__ method returns a function object, from which we can get the function name:

    def __init__(self, f):
        self.f = f
        if hasattr(f, "__call__"):
            name = self.f.__name__
        else:
            # f is a class or static method.
            tmp = f.__get__(None, f.__class__)
            name = tmp.__name__
        print("Tracing: {0}".format(name))

In the decorator’s __get__ method, we’ll know that we’re dealing with a staticmethod or classmethod if the passed obj has the value None. If that’s the case, then we make a one-time adjustment to self.f, ensuring that it points to the underlying function.

Wait—why didn’t we do this in DebugTrace.__init__? It may seem redundant, but the call to f.__get__ that we made in DebugTrace.__init__ doesn’t count: that call didn’t specify the class that f actually belongs to. (Any class works for the purpose of getting the function’s name.) Now that we’re in DebugTrace.__get__, we know via the ownerClass parameter the class that self.f is associated with. This class may make its way into a classmethod call (e.g., the call to Greeter.classHello), so it matters that we get it right.

Note that we return self in this case. We don’t want to create a new method object for classmethods or staticmethods; just calling self.__call__ will call the method appropriately.

    def __get__(self, obj, ownerClass=None):
        if obj is None:
            f = self.f
            if not hasattr(f, "__call__"):
                self.f = f.__get__(None, ownerClass)
            return self
        else:
            # Return a wrapper that binds self as a method of obj (!)
            return types.MethodType(self, obj)
Setting self.f as above might raise thread-safety issues, especially if you don’t want to rely on the atomicity of modifying a dict in-place. Borrowing from Ian Bicking’s solution, which returns a copy of the decorator for each call to __get__, can help us dodge the concurrency bullet. We’d replace

            return self

with

            return self.__class__(self.f)

However, this results in any side effects in the decorator’s __init__ method being re-executed for every call to the decorated method. Note the additional “Tracing:” lines in the output here:

Tracing: hello
Tracing: classHello
Tracing: staticHello
Tracing: greet
Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Tracing: staticHello
Calling: staticHello
*** Something says hello to you
Tracing: classHello
Calling: classHello
*** The Greeter class says hello to everyone

Another option, of course, is to use a mutex around the statement that modifies self.f.

The decorator’s __call__ method is unchanged from yesterday’s example. As before, it simply prints out the desired trace message, then invokes self.f.

Here’s the entire decorator, as revised:

import types

class DebugTrace(object):
    def __init__(self, f):
        self.f = f
        if hasattr(f, "__call__"):
            name = self.f.__name__
        else:
            # f is a class or static method
            tmp = f.__get__(None, f.__class__)
            name = tmp.__name__
        print("Tracing: {0}".format(name))

    def __get__(self, obj, ownerClass=None):
        if obj is None:
            f = self.f
            if not hasattr(f, "__call__"):
                self.f = f.__get__(None, ownerClass)
            return self
        else:
            # Return a wrapper that binds self as a method of obj (!)
            return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))

    @DebugTrace
    @classmethod
    def classHello(cls, to):
        print("*** The {0} class says hello to {1}".format(cls.__name__, to))

    @DebugTrace
    @staticmethod
    def staticHello(to):
        print("*** Something says hello to " + to)


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()
    Greeter.staticHello("you")
    Greeter.classHello("everyone")

greet()

I’ve tested this with Python 2.6, 2.7, and 3.1.


1 Without taking a deep dive into Python’s history, I couldn’t say why they’re not callable. But it does seem that class methods and static methods were never intended to be used frequently.

Python: Decorating with class through descriptors

Update: If you find this article helpful, you may want to read the follow-up.

As a fairly new Python developer, my first attempt at decorators hit a snag: my simple class-based decorator failed when decorating a method. I got around the immediate problem by rewriting the decorator as a function. Yet the episode left me wondering if there were some way to fix the class-based decorator to work when applied to methods. I’ve found what seems like an elegant solution, and picked up a better understanding of decorators and descriptors in the process.

Here’s an example that illustrates the original problem. DebugTrace is the decorator class:

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Running this with Python 2.6 or 3.1 results in an error:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
Traceback (most recent call last):
  File "./DecoratorExample.py", line 31, in <module>
    greet()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
  File "./DecoratorExample.py", line 27, in greet
    g.hello()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
TypeError: hello() takes exactly 1 argument (0 given)

The output explains the problem. DebugTrace was instantiated only twice: once for Greeter.hello, and once for greet. That’s a reminder that decoration occurs during compile time, not run time. Accordingly, DebugTrace‘s reference to Greeter.hello represents an unbound function—it doesn’t reference any Greeter instances. So no ‘self’ argument was passed into the call to Greeter.hello; hence the TypeError.

All object-oriented languages that are worth knowing (and many that aren’t) allow container objects to redirect access to their contained objects. But of the languages that I’ve used, Python is unique in allowing object attributes to redirect calls that attempt to access them. Objects that implement this capability are called descriptors1. As we’ll soon see, function objects, which Python uses to implement methods, are descriptors.

A full description of descriptors would be too long to fit here. Here are the most important points for this post:

  1. When the interpreter reads a class attribute, and the attribute value is an object that has a __get__ method, then the return value of that __get__ method is used as the attribute’s value.
  2. Methods are class attributes.
  3. Any callable object can serve as a method.

Here’s a good guide to descriptors. There’s also a recent post by Guido van Rossum, Python’s BDFL, that provides good background material on the feature.

To see where descriptors come into play, let’s look at the calling sequences for different versions of Greeter.hello. Here’s the rough sequence before Greeter.hello was decorated:

  • The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
  • The value of the Greeter.hello attribute (a function object) has a __get__ method, making it a descriptor, so that __get__ method is invoked. It’s passed a reference to the Greeter instance (obj) through which Greeter.hello was called.
  • The function object’s __get__ method creates and returns another callable object, which we’ll refer to as a (bound) method object. The method object references both the Greeter.hello function and obj.
  • The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

When Greeter.hello is decorated with the DebugTracer class as shown above, a call to Greeter.hello runs more or less like this:

  • The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
  • The value of the Greeter.hello attribute is an instance of DebugTrace. This isn’t a function object, and it doesn’t have a __get__ method, but it does have a __call__ method. That __call__ method is invoked with the empty argument list.
  • DebugTrace.__call__ then calls Greeter.hello with the empty argument list.
  • Since Greeter.hello was looking for a single argument (self), rather than an empty argument list, a TypeError is raised.

To fix DebugTrace, I turned it into a descriptor class, adding a __get__ method that fills the same role as a function object’s __get__ method. However, this method binds the Greeter instance to the callable DebugTrace object.

import types
# ...
    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

Compare the new calling sequence for Greeter.hello to the sequence prior to decoration:

  • The interpreter finds an attribute named hello in the Greeter class object.
  • The value of the Greeter.hello attribute is an instance of DebugTrace, which is now a descriptor. DebugTrace.__get__ is called, with obj (the Greeter instance) passed as one of the arguments.
  • DebugTrace.__get__ creates and returns a method object. The method object references both the DebugTrace instance and obj.
  • The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls DebugTrace.__call__. That in turn calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

It’s worth noting that DebugTrace.__get__ is only invoked when accessing a DebugTrace object through an object’s class dictionary. Hence its presence has no effect on functions that aren’t methods, such as greet.

You can see the full, working example here (click on the “show source” link to view) :

import types

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Executing the new version gives the desired output:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
*** Greeter 1 says hello!
Calling hello
*** Greeter 2 says hello!
Credit Where Credit Is Due Dept: I’m not the first one to discover that a class-based decorator needs a __get__ method. For example, Ian Bicking wrote about a similar technique over a year and a half ago. However, Ian’s descriptor creates a new instance of the decorator class every time the method is invoked. I think the solution that I found—binding the original decorator instance to the method object—is different enough to be worth its own post.

For what it’s worth, I ran a simple performance test comparing Ian’s and my own class-based decorators, along with a function-based decorator. It showed no significant difference in performance among them. Apparently, the interpreter already optimizes these cases, which isn’t all that surprising.


1 I found the term “descriptor” to be somewhat confusing at first. A descriptor doesn’t really describe anything other than itself. To be fair, I don’t have any better suggestions. (“Redirector” ?) Naming is often one of the hardest challenges in software design, at least when it’s done right.

SyntaxHighlighter: Easier to load; faster to boot?

SyntaxHighlighter has a relatively high surface area, typically requiring two CSS files and at least two JavaScript files to be linked into a web page. Here’s a truncated example:

<html>
<head>
	<!-- Stylesheets for SyntaxHighlighter -->
<link type="text/css" rel="stylesheet" href="styles/shCore.css"/>
<link type="text/css" rel="stylesheet" href="styles/shThemeDefault.css"/>
...
</head>
<body>

<em>...Some kind of interesting page content usually goes here, 
but we're not interested in that right now...</em>

	<!-- Load the SyntaxHighlighter Core and Brush scripts. A 
               separate Brush script is required for each language. -->
<script type="text/javascript" src="scripts/shCore.js"></script>
<script type="text/javascript" src="scripts/shBrushJava.js"></script>
<script type="text/javascript" src="scripts/shBrushJScript.js"></script>
	<!-- Don't forget the XML brush if you're using html-script -->
<script type="text/javascript" src="scripts/shBrushXml.js"></script>

	<!-- Now that all the script files are loaded, begin highlighting. -->
<script type="text/javascript">
SyntaxHighlighter.config.stripBrs = true;
// ... additional configuration, as needed ...

SyntaxHighlighter.all();
</script>

</body>

Some users have wanted a simpler way to integrate SyntaxHighlighter into their sites. Last year, David Chambers wrote a script that uses the Prototype library to load the required CSS and brush files. He ran into a troubling quirk in SyntaxHighlighter: It needs any required brush files to be loaded before it starts highlighting. I responded by updating the OOWB fork of SyntaxHighlighter, so that brushes could be loaded asynchronously. But that’s as far as I’d gone towards automated loading, until now.

I recently looked into adding automated loading to SyntaxHighlighter itself, coming up with not one, but two automated solutions. I should really say one and a half, as one is still in the alpha stage and will likely stay there.

The “alpha-stage” approach comprises a new JavaScript file that loads SyntaxHighlighter and the necessary brush files through AJAX, by way of a pair of simple Java servlets. I’d hoped that this would lead to faster load times, by reducing the total number of downloads. And so it does—sometimes. For testing, I used Google App Engine as the servlet host (using the free quotas.) Performance was inconsistent, to say the least; this is probably due to the Google App Engine’s method of starting up and shutting down servlets. It seemed that only by making repeated hits on the servlets in rapid succession could I be sure of getting to a container that already had my servlets running, thus getting good load times.1 The servlets are still up there; if you’d like the Java source (to host them elsewhere), and/or the JavaScript for using them, just let me know.

I also implemented a pure JavaScript approach, which I hadn’t expected to yield better load times. Yet I’ve seen a number of cases where this is faster than the original, static way of loading SyntaxHighlighter. I’m releasing a new download so that you can try it out. I highly recommend that you test it thoroughly before putting it into real use.

To use this new method, the web page invokes SyntaxHighlighter.boot instead of SyntaxHighlighter.all. When boot is used as the entry point, SyntaxHighlighter determines which brushes are required by the web page, and issues HTTP requests for each brush file (as well as the appropriate CSS.) These requests use the age-old method of adding a tag for each brush, and a tag for each CSS file.

Using SyntaxHighlighter.boot, a web page only needs to add a couple of scripts, usually at the end of the . No CSS link elements are required.

<!--  We just need to load shCore.js near the end of the <body> element,
	then call SyntaxHighlighter.boot() with our configuration options. -->

<script type="text/javascript"
src="http://path/to/sh/directory/syntaxhighlighter/scripts/shCore.js"></script>

<script type="text/javascript">
	SyntaxHighlighter.boot(
		"http://path/to/syntaxhighlighter/root/syntaxhighlighter/",
		{theme : "Default"}, // Configuration settings
		{stripBrs : true}      // Default settings
	);
</script>

</body>

That’s it. Every page on your site can use the same set-up, regardless of which brush files it needs. (For another example, look at the HTML source for this page.)

When an HTML page includes external scripts—that is, elements with src attributes, the browser loads the script files synchronously, as it encounters them in the HTML. When the scripts are loaded dynamically, the browser can load them asynchronously; several scripts may then be loading simultaneously. This can be particularly helpful for pages that use multiple brushes. Here’s a Firebug sequence diagram of a web page with five brushes, loading in SyntaxHighlighter’s traditional way:

(Click on image for larger view)

Here’s the same page, modified to use the boot method. Notice that, unlike in the first picture, the shBrush...js files are loaded in parallel:

(Click on image for larger view)

Of course, this isn’t an entirely new discovery, but it came as a pleasant surprise nonetheless.

Availability

In addition to SyntaxHighlighter itself, I also modified Viper007Bond’s SyntaxHighlighter Evolved plugin for WordPress to use the new boot method. This simplified the code quite a bit, though it may have introduced bugs.

SyntaxHighlighter and SyntaxHighlighter Evolved are each available on the downloads page. Test before using. Note: If you’ve been using SyntaxHighlighter Evolved, you will probably need to re-set your settings if you install this plugin.

Implementation notes

In the traditional SyntaxHighlighter, all brushes must be loaded before SyntaxHighlighter.all is invoked. In this fork, “>brushes can load at any time after shCore.js has been loaded. This flexibility was key to the design of the boot method: If a brush is loaded after SyntaxHighlighter.all or SyntaxHighlighter.boot is invoked, SyntaxHighlighter looks through the page to see if there’s any input requesting the newly loaded brush.

The boot method needs to map each brush name to the script file that implements that brush. I implemented this by modifying the Perl script that builds and stages SyntaxHighlighter. The script now parses the *Brush.js files in the source tree, pulling out the names and aliases supported by each file, and writes them into an array in shCore.js. The downside of this is that testing via the boot method requires re-running the script, which builds and stages the files into a new directory, from where I can run tests. Even though the script only takes about a second, it’s crossed the boundary between having no build step in the development cycle, and having any build step in the development cycle.

When You Come To A Fork In The Road, Take It.

I had two motives for working on SyntaxHighlighter: One, to improve its display of code on my own blog and elsewhere. Two, as a training ground for learning JavaScript.

I was between jobs when I took on this project. As of this Monday, that will no longer be the case. That’s great news for me, but much as I’d like to keep the SyntaxHighlighter work moving forward, I don’t really know if or when I’ll have the time for it. On the other hand, I no longer feel like I need a private JavaScript training ground. So, I’ve pushed my repository to bitbucket.org as a fork of Alex’s repository. If he so chooses (and so far he hasn’t, which is fine), Alex is welcome to merge the fork into his branch. Perhaps more importantly, anyone else can now fork the code, and/or make contributions to this fork.

I do have some more goals in mind for SyntaxHighlighter. I’ll be writing about them in the not-too-distant future.


1Perhaps a faster host would lead to the faster load times I’d hoped for; but that would probably require payment, which makes this approach considerably less attractive.