If you’ve used Android’s Notepad tutorial (Version 3) , or modeled your own Activity on Notepadv3, then you’ve probably seen log messages that look something like this:

android.database.sqlite.DatabaseObjectNotClosedException: 
    Application did not close the cursor or database object that was opened here
at android.database.sqlite.SQLiteCompiledSql.<init>(SQLiteCompiledSql.java:62)
at android.database.sqlite.SQLiteProgram.<init>(SQLiteProgram.java:80)
at android.database.sqlite.SQLiteStatement.<init>(SQLiteStatement.java:36)
at android.database.sqlite.SQLiteDatabase.compileStatement(SQLiteDatabase.java:1145)
at android.database.sqlite.SQLiteDatabase.updateWithOnConflict(SQLiteDatabase.java:1671)
at android.database.sqlite.SQLiteDatabase.update(SQLiteDatabase.java:1622)
at com.android.demo.notepad3.NotesDbAdapter.updateNote(NotesDbAdapter.java:186)
at com.android.demo.notepad3.NoteEdit.saveState(NoteEdit.java:106)
at com.android.demo.notepad3.NoteEdit.onPause(NoteEdit.java:87)
at android.app.Activity.performPause(Activity.java:3842)
at android.app.Instrumentation.callActivityOnPause(Instrumentation.java:1190)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3335)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3305)
at android.app.ActivityThread.handlePauseActivity(ActivityThread.java:3288)
at android.app.ActivityThread.access$2500(ActivityThread.java:125)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2044)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:123)
at android.app.ActivityThread.main(ActivityThread.java:4627)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:521)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:868)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:626)
at dalvik.system.NativeStart.main(Native Method)

This is resolved easily enough by overriding the onDestroy method in each of the Activities in your application — or at least, each Activity that uses the database. In the Notepadv3 app, this means overriding onDestroy in both the Notepadv3 and NoteEdit classes. The same definition is suitable for both classes:

    public void onDestroy() {
        super.onDestroy();

        // Replace mDbHelper as needed with your database connection, or
        // whatever wraps your database connection. (See below.)
        mDbHelper.close();
    }

In the Notepadv3 tutorial, the mDbHelper field is an instance of the class NotesDbAdapter, which wraps the database connection. NotesDbAdapter comes from google with the close method already defined, but there’s no code that calls it. With this change, we’re supplying the code that calls close at the right time. (I think.)

In general: If your activity opens a database by calling either SQLiteOpenHelper.getReadableDatabase or SQLiteOpenHelper.getWritableDatabase, then you should rely on SQLiteOpenHelper.close to close that database. Your activity’s onDestroy method must result in a call to the close method for any instances of SQLiteOpenHelper, or classes derived from it, that you’ve created. (In the code above, mDbHelper.close will have that effect.) SQLiteOpenHelper.close will then close any database handle that it created for you.

It’s often the case that bulk insertions are performed in the onCreate method of an SQLiteOpenHelper, so I’ll use that context for this example. (This has implications on transaction management, which in turn affects performance, as described below.)

Suppose the onCreate method currently looks something like this:

private class DatabaseHelper extends SQLiteOpenHelper {
    @Override
    public void onCreate(SQLiteDatabase db) {
        ContentValues values = new ContentValues();
        while (moreRowsToInsert) {
            // ... create the data for this row (not shown) ...

            // Add the data for each column
            values.put("Greek", greekData);
            values.put("Ionic", ionicData);
            // ...
            values.put("Roman", romanData);

            // Insert the row into the database.
            db.insert("columnTable", null, values);
        }
    }
    //...
}

Using DatabaseUtils.InsertHelper, this would be re-written as:

import android.database.DatabaseUtils.InsertHelper;
//...
private class DatabaseHelper extends SQLiteOpenHelper {
    @Override
    public void onCreate(SQLiteDatabase db) {
        // Create a single InsertHelper to handle this set of insertions.
        InsertHelper ih = new InsertHelper(db, "columnTable");

        // Get the numeric indexes for each of the columns that we're updating
        final int greekColumn = ih.getColumnIndex("Greek");
        final int ionicColumn = ih.getColumnIndex("Ionic");
        //...
        final int romanColumn = ih.getColumnIndex("Roman");

        try {
            while (moreRowsToInsert) {
                // ... Create the data for this row (not shown) ...

                // Get the InsertHelper ready to insert a single row
                ih.prepareForInsert();

                // Add the data for each column
                ih.bind(greekColumn, greekData);
                ih.bind(ionicColumn, ionicData);
                //...
                ih.bind(romanColumn, romanData);

                // Insert the row into the database.
                ih.execute();
            }
        }
        finally {
            ih.close();  // See comment below from Stefan Anca
        }
    }
    //...
}

As this shows, using InsertHelper is barely more complicated than using SQLiteDatabase.insert. The major differences are that you need to call ih.prepareForInsert() before adding (“binding”) the column data; and you need to obtain each column’s numeric index, which we get by calling ih.getColumnIndex() prior to the loop.

After replacing SQLiteDatabase.insert with DatabaseUtils.InsertHelper, the database insertion speed went from the equivalent of about 95 rows per second to about 525 rows per second. (“Equivalent of”, because the app also spends cycles creating the data to insert. Here, the performance timings are measured using constant data, eliminating that overhead.)

Other ways to speed up insertions

In addition to that gain, two more changes then brought the insertion speed to well over 900 rows per second. Whether these tricks techniques work for you will depend on your application.

Don’t bind empty columns

In my app, the data for at least 50% of the columns is empty. By skipping the call to ih.bind() when the column data is a null or empty string, I saw a roughly 30% performance boost.

Temporarily disable database thread locking

I’m loading the database during the onCreate method of my app’s SQLiteOpenHelper. During this time, it seems safe to assume that only one thread is accessing the database, so I use SQLiteDatabase.setLockingEnabled() to temporarily (let me emphasize that: temporarily) disable thread locks within the database API. This yielded about a 35% performance gain:

    public void onCreate(SQLiteDatabase db) {
    //...
    try {
        // *Temporarily* (have I emphasized that enough?) disable
        // thread locking in the database. Be sure to re-enable locking 
        // within a finally block.
        db.setLockingEnabled(false);
        // ... load the database ...
    }
    finally {
        db.setLockingEnabled(true);
    }

Transactions and performance

A number of people have cited performance gains through use of explicit transactions in SQLite. However, SQLiteOpenHelper creates a transaction before invoking its callback methods (onCreate, onUpgrade, and onOpen), so explicit transaction control is unnecessary within those methods. (SQLiteOpenHelper will assume that the transaction was successful unless your method throws an exception.)

You would need to manage your own transactions if your insertion code is running outside of one of SQLiteOpenHelper‘s callback methods. The main APIs for this are SQLiteDatabase.beginTransaction, SQLiteDatabase.setTransactionSuccessful, and SQLiteDatabase.endTransaction.

It is possible to nest transactions, but, not surprisingly, this doesn’t seem to help performance. In fact, I saw a very slight performance degradation when using nested transactions (approximately 1%, probably below the accuracy of the measurements.) I also tried periodically closing the current transaction — the first of these being the transaction that was opened by SQLiteOpenHelper — then opening a new one. This didn’t yield much improvement, if any.

A bit of searching turned up this issue about using a Spinner in combination with a SimpleAdapter to display a CheckedTextView; it seems that a ViewBinder must be set on the SimpleAdapter for that combination to work. Although it’s not the same problem, it still made me wonder if the SimpleAdapter / Spinner combination might need a ViewBinder for displaying TextView, too.

I wasn’t able to reproduce the IllegalStateException when I ran a sample program on AndroidOS 2.2, but I thought I’d try it on 1.5 in the emulator. Sure enough, I hit the same problem that Chromium reported. Apparently, whatever issue this is has been fixed by 2.2, and perhaps in earlier releases. (I haven’t tried other releases besides 2.2 and 1.5.)

And I also found that ViewBinder once again comes to the rescue, preventing the IllegalStateException from being thrown.

Like some other interfaces in Android (I’m looking at you, CursorToStringConverter. You too, FilterQueryProvider), ViewBinder’s bark is worse than its bite. In other words, the name may be a bit intimidating, but the implementation is a breeze.

Here’s a sample program, adapted from Chromium’s example, that adds a ViewBinder to a SimpleAdapter. This works on AndroidOS 1.5 and 2.2 (and, presumably, intermediate versions as well.)

package org.oowb.HelloSpinner;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.widget.AdapterView;
import android.widget.SimpleAdapter;
import android.widget.Spinner;
import android.widget.TextView;
import android.widget.Toast;
import android.widget.AdapterView.OnItemSelectedListener;

/**
 *  Displays a list of colors in a Spinner.
 */
public class HelloSpinner extends Activity {
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        // The key to use for reading the color from the Map
        final String[] from = new String[] { "color" };

        // The type of View to use for displaying the color name.
        // android.R.id.text1 is a standard resource for displaying text.
        final int[] to = new int[] { android.R.id.text1  };

        // Create the List of strings for the spinner to display. Each string
        // is embedded within a Map, using "color" as the key.
        final List<Map<String, String>> data = 
            new ArrayList<Map<String, String>>();
        final String[] colors = getResources().getStringArray(R.array.colors);

        for (int i = 0; i < colors.length; i++) {
            data.add(addData(colors[i]));
        }

        final SimpleAdapter simpleAdapter = 
            new SimpleAdapter(this, data, 
                    android.R.layout.simple_spinner_item, from, to);
        simpleAdapter.setDropDownViewResource(
                android.R.layout.simple_spinner_dropdown_item);

        // Add a ViewBinder to display a color name in a TextView within the
        // Spinner. (This isn't needed in AndroidOS 2.2. In earlier releases,
        // when we're displaying text data within a Spinner, and no ViewBinder
        // is set in the SimpleAdapter, an IllegalStateException is thrown.)
        SimpleAdapter.ViewBinder viewBinder = new SimpleAdapter.ViewBinder() {
            
            public boolean setViewValue(View view, Object data,
                    String textRepresentation) {
                // We configured the SimpleAdapter to create TextViews (see
                // the 'to' array, above), so this cast should be safe:
                TextView textView = (TextView) view;
                textView.setText(textRepresentation);
                return true;
            }
        };
        simpleAdapter.setViewBinder(viewBinder);

        final Spinner spinner = (Spinner) findViewById(R.id.spinner);
        spinner.setAdapter(simpleAdapter);

        // Add an OnItemSelectedListener to display the selected Color
        spinner.setOnItemSelectedListener(new OnItemSelectedListener() {
            @Override
            public void onItemSelected(AdapterView<?> parent, View view,
                    int position, long id) {
                // Get the color name out of the Map
                final Map<String, String> data = 
                    (Map<String, String>) parent.getItemAtPosition(position);
                final String text = "Selected Color:-  " + data.get("color");

                Toast.makeText(parent.getContext(), text, 
                        Toast.LENGTH_LONG).show();
            }

            @Override
            public void onNothingSelected(AdapterView<?> arg0) {
                // Do nothing
            }
        });
    }

    /**
     * Convert the String that's passed in into a Map, with
     * "color" as the key, and the String as the value.
     * @param    colorName  The color to be inserted into a new Map
     * @return   the new Map
     */
    private Map<String, String> addData(String colorName) {
        Map<String, String> map = new HashMap<String, String>();
        map.put("color", colorName);
        return map;
    }
}

To complete the example, here are the layout (main.xml) and values files (arrays.xml and strings.xml):

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:padding="10dip"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content">

    <Spinner 
        android:id="@+id/spinner"
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:drawSelectorOnTop="true"
        android:prompt="@string/prompt"
    />
</LinearLayout>

<?xml version="1.0" encoding="UTF-8"?>
<resources>
    <string-array name="colors">
        <item>Red</item>
        <item>Orange</item>
        <item>Yellow</item>
        <item>Green</item>
        <item>Blue</item>
        <item>Purple</item>
        <item>Violet</item>
        <item>Octarine</item>
    </string-array>
</resources>

<?xml version="1.0" encoding="utf-8"?>
<resources>
    <string name="app_name">Color Chooser</string>
    <string name="prompt">Choose a color:</string>
</resources>

Earlier today, I published a lengthy examination of the work needed to supply an AutoCompleteTextView with data using a CursorAdapter. My intentions were pure, but my code was not. Where I’d seen a need for subclassing, it was because I hadn’t caught on to some of the handler methods provided by the SimpleCursorAdapter class.

Specifically, if we’re using a SimpleCursorAdapter, then:

• We don’t need to override the convertToString method, if instead we call adapter.setCursorToStringConverter.
• And there’s no need to override runQueryOnBackgroundThread, if instead we call adapter.setFilterQueryProvider.

This allows a significant refactoring of the SelectState class. Instead of the subclassed Adapter class, I’m now using anonymous inner classes to provide the cursor-to-name conversion and the filter query. The inner classes use essentially the same code as the methods that they’ve replaced.

The re-written SelectState follows. The full example can be downloaded as a ZIP file.

(See the earlier post for a description of the sample application’s functionality.)

package org.oowb.AutoCompleteExample;

import android.app.Activity;
import android.database.Cursor;
import android.os.Bundle;
import android.view.View;
import android.widget.AdapterView;
import android.widget.AutoCompleteTextView;
import android.widget.Button;
import android.widget.FilterQueryProvider;
import android.widget.SimpleCursorAdapter;
import android.widget.TextView;
import android.widget.AdapterView.OnItemClickListener;
import android.widget.SimpleCursorAdapter.CursorToStringConverter;

/**
 * A simple Android Activity to demonstrate:
 * 
 * 1) How to use an AutoCompleteTextView with a SimpleCursorAdapter
 * 
 * 2) How to access the cursor row for the user's choice, to obtain additional
 * data from that row when an item is selected.
 * 
 * @author Dan Breslau
 * 
 */
public class SelectState extends Activity {

    final static int[] to = new int[] { android.R.id.text1 };
    final static String[] from = new String[] { "state" };

    private TextView mStateCapitalView;
    private AutoCompleteTextView mStateNameView;
    private AutoCompleteDbAdapter mDbHelper;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        mDbHelper = new AutoCompleteDbAdapter(this);
        setContentView(R.layout.selectstate);
        Button confirmButton = (Button) findViewById(R.id.confirm);
        confirmButton.setOnClickListener(new View.OnClickListener() {
            public void onClick(View view) {
                setResult(RESULT_OK);
                finish();
            }
        });

        mStateCapitalView = (TextView) findViewById(R.id.state_capital);
        mStateNameView = (AutoCompleteTextView) findViewById(R.id.state_name);

        // Create a SimpleCursorAdapter for the State Name field.
        SimpleCursorAdapter adapter = 
            new SimpleCursorAdapter(this, 
                    android.R.layout.simple_dropdown_item_1line, null,
                    from, to);
        mStateNameView.setAdapter(adapter);

        // Set an OnItemClickListener, to update dependent fields when
        // a choice is made in the AutoCompleteTextView.
        mStateNameView.setOnItemClickListener(new OnItemClickListener() {
            public void onItemClick(AdapterView<?> listView, View view,
                        int position, long id) {
                // Get the cursor, positioned to the corresponding row in the
                // result set
                Cursor cursor = (Cursor) listView.getItemAtPosition(position);

                // Get the state's capital from this row in the database.
                String capital = 
                    cursor.getString(cursor.getColumnIndexOrThrow("capital"));

                // Update the parent class's TextView
                mStateCapitalView.setText(capital);
            }
        });

        // Set the CursorToStringConverter, to provide the labels for the
        // choices to be displayed in the AutoCompleteTextView.
        adapter.setCursorToStringConverter(new CursorToStringConverter() {
            public String convertToString(android.database.Cursor cursor) {
                // Get the label for this row out of the "state" column
                final int columnIndex = cursor.getColumnIndexOrThrow("state");
                final String str = cursor.getString(columnIndex);
                return str;
            }
        });

        // Set the FilterQueryProvider, to run queries for choices
        // that match the specified input.
        adapter.setFilterQueryProvider(new FilterQueryProvider() {
            public Cursor runQuery(CharSequence constraint) {
                // Search for states whose names begin with the specified letters.
                Cursor cursor = mDbHelper.getMatchingStates(
                        (constraint != null ? constraint.toString() : null));
                return cursor;
            }
        });
    }
}

I recently integrated an AutoCompleteTextView into a prototype Android application that I’m developing. There isn’t exactly a wealth of documentation on the Web describing AutoCompleteTextView, and what I’ve found didn’t match the scenario I had in mind. Specifically:

• The choices came from a database, so I needed to use a CursorAdapter; and
• The AutoCompleteTextView had some dependent fields; changes to the value in the AutoCompleteTextView needed to be reflected in those dependent fields.

For example, let’s use a AutoCompleteTextView for choosing states. When the user chooses a state, I want to update a TextView to show the capital of the chosen state.

First the form appears with no value in the State Capital field (left); then a state is chosen from the list (right.)

After a state has been chosen, its capital is also displayed.

Adapting SQLite to AutoCompleteTextView

The documentation for AutoCompleteTextView says that it obtains suggestions from a data adapter. This should be familiar territory for anyone who’s used ListView, which also obtains its list items from a data adapter. Not surprisingly, AutoCompleteTextView uses a ListView internally.

However, ListView and AutoCompleteTextView require different interactions with their adapters. A standard ListView will typically need to fetch the list of items once, and only once. On the other hand, AutoCompleteTextView issues a query for matching choices whenever the user enters new text into the field. (In fact, the AutoCompleteTextView won’t issue a query until the user has typed some text. Hence the CursorAdapter‘s constructor can take null for the Cursor parameter.)

SimpleCursorAdapter wasn’t designed for this kind of usage pattern; in particular, it doesn’t know how you want to query for matching choices. So, you must derive from SimpleCursorAdapter, or from its base class, CursorAdapter. Your subclass must supply two methods that are essentially stubbed out in CursorAdapter:

• convertToString, which supplies the String value that is entered into the AutoCompleteTextView when a choice is made; and
• runQueryOnBackgroundThread. In this method, you invoke a query for choices that match the text entered by the user, and return a Cursor that provides the results.

Let’s take a closer look at implementing the derived CursorAdapter class.

public CharSequence convertToString(Cursor cursor)

This method receives the Cursor positioned to a specific row, and returns a String label or representation for that row.

public Cursor runQueryOnBackgroundThread(CharSequence constraint)

Just to clarify, “On Background Thread” means that this method is called in a background thread — it does not mean that you have to spawn that thread within this method.

This method runs a query to get choices that match the input. It’s up to the application to determine what constitutes a match. (In this example, I’m searching for states whose names begin with the given letters.)

The constraint parameter is the input that the user has typed in. It may be null if the user has typed, and then erased, input text. In this case, all results should be returned.

But wait, there’s more!

It might seem… well, simpler, to derive the new Adapter class from SimpleCursorAdapter. However, deriving from CursorAdapter requires only two additional methods to be implemented, newView and bindView, both of which are straightforward. Hence for this example, I’m using CursorAdapter as the base class. Later, I’ll describe the changes needed to use SimpleCursorAdapter as the base.

public View newView(Context context, Cursor cursor, ViewGroup parent)

newView returns a newly constructed View suitable for displaying a single item in the choice list. The usual method for doing this is to use the Inflater, which is given the resource ID of a layout that describes the new View. It’s not necessary to create your own layout file; Android provides a standard layout that’s appropriate for this purpose, identified by android.R.layout.simple_dropdown_item_1line.

public void bindView(View view, Context context, Cursor cursor)

This is called to display the label for the current row. If newView uses android.R.layout.simple_dropdown_item_1line for layout, then bindView will be called with a TextView as the View argument.

Updating dependent fields

To update dependent fields, start by creating a class that implements OnItemClickListener. (Here, it’s the Adapter class; it could also be your Activity, or some other class.) This interface defines a callback that’s invoked when the user makes a choice. The setOnItemClickListener method in AutoCompleteTextView associates the listener with the field.

The callback method, onItemClick, receives a position parameter that identifies the row number within the result set, starting from 0. For example, if the third state, “Arizona”, is chosen, the value of position will be 2. There’s also an AdapterView<?> parameter, whose purpose is a bit obscure. This is the ListView belonging to the AutoCompleteTextView. (This may be the only situation in which that ListView is directly exposed to the caller.)

We can then get the cursor from the AdapterView<?> parameter, and position it to the correct row, by calling getItemAtPosition. Note that getItemAtPosition returns an Object, so a cast is necessary. (It doesn’t seem to be documented anywhere that listView.getItemAtPosition will return a Cursor in this case.)

In the full example, this logic is seen in the line:

Cursor cursor = (Cursor) listView.getItemAtPosition(position)

The result would probably be the same if we replaced the above line with:

            Cursor cursor = getCursor();
            cursor.moveToPosition(position);

I went with the first approach because: 1) It’s (arguably) more in keeping with the Adapter’s contract. (There doesn’t seem to be any guarantee that changeCursor will be called by the main thread; without that call, getCursor() certainly won’t work.) And 2) it’s simpler, if by “simpler” we mean one less line of code.

The code

Now let’s have a look at the code. Here’s the Activity class, SelectState, with its nested Adapter class, ItemAutoTextAdapter. (The other class needed for this example, AutoCompleteDbAdapter, is available below.)

package org.oowb.AutoCompleteExample;

import android.app.Activity;
import android.content.Context;
import android.database.Cursor;
import android.os.Bundle;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.AdapterView;
import android.widget.AutoCompleteTextView;
import android.widget.Button;
import android.widget.CursorAdapter;
import android.widget.TextView;

/**
 * A simple Android Activity to demonstrate: 
 * 
 * 1) How to use an AutoCompleteTextView with a CursorAdapter
 * 
 * 2) How to access the cursor row for the user's choice, to obtain 
 *    additional data from that row when an item is selected.
 * 
 * @author Dan Breslau
 * 
 */
public class SelectState extends Activity {
    /**
     * Specializes CursorAdapter to supply choices to a AutoCompleteTextView.
     * Also implements OnItemClickListener to be notified when a choice is made,
     * and uses the choice to update other fields on the Activity form.
     */
    class ItemAutoTextAdapter extends CursorAdapter
            implements android.widget.AdapterView.OnItemClickListener {

        private AutoCompleteDbAdapter mDbHelper;

        /**
         * Constructor. Note that no cursor is needed when we create the
         * adapter. Instead, cursors are created on demand when completions are
         * needed for the field. (see
         * {@link ItemAutoTextAdapter#runQueryOnBackgroundThread(CharSequence)}.)
         * 
         * @param dbHelper
         *            The AutoCompleteDbAdapter in use by the outer class
         *            object.
         */
        public ItemAutoTextAdapter(AutoCompleteDbAdapter dbHelper) {
            // Call the CursorAdapter constructor with a null Cursor.
            super(SelectState.this, null);
            mDbHelper = dbHelper;
        }

        /**
         * Invoked by the AutoCompleteTextView field to get completions for the
         * current input.
         * 
         * NOTE: If this method either throws an exception or returns null, the
         * Filter class that invokes it will log an error with the traceback,
         * but otherwise ignore the problem. No choice list will be displayed.
         * Watch those error logs!
         * 
         * @param constraint
         *            The input entered thus far. The resulting query will
         *            search for states whose name begins with this string.
         * @return A Cursor that is positioned to the first row (if one exists)
         *         and managed by the activity.
         */
        @Override
        public Cursor runQueryOnBackgroundThread(CharSequence constraint) {
            if (getFilterQueryProvider() != null) {
                return getFilterQueryProvider().runQuery(constraint);
            }

            Cursor cursor = mDbHelper.getMatchingStates(
                    (constraint != null ? constraint.toString() : null));

            return cursor;
        }

        /**
         * Called by the AutoCompleteTextView field to get the text that will be
         * entered in the field after a choice has been made.
         * 
         * @param Cursor
         *            The cursor, positioned to a particular row in the list.
         * @return A String representing the row's text value. (Note that this
         *         specializes the base class return value for this method,
         *         which is {@link CharSequence}.)
         */
        @Override
        public String convertToString(Cursor cursor) {
            final int columnIndex = cursor.getColumnIndexOrThrow("state");
            final String str = cursor.getString(columnIndex);
            return str;
        }

        /**
         * Called by the ListView for the AutoCompleteTextView field to display
         * the text for a particular choice in the list.
         * 
         * @param view
         *            The TextView used by the ListView to display a particular
         *            choice.
         * @param context
         *            The context (Activity) to which this form belongs;
         *            equivalent to {@code SelectState.this}.
         * @param cursor
         *            The cursor for the list of choices, positioned to a
         *            particular row.
         */
        @Override
        public void bindView(View view, Context context, Cursor cursor) {
            final String text = convertToString(cursor);
            ((TextView) view).setText(text);
        }

        /**
         * Called by the AutoCompleteTextView field to display the text for a
         * particular choice in the list.
         * 
         * @param context
         *            The context (Activity) to which this form belongs;
         *            equivalent to {@code SelectState.this}.
         * @param cursor
         *            The cursor for the list of choices, positioned to a
         *            particular row.
         * @param parent
         *            The ListView that contains the list of choices.
         * 
         * @return A new View (really, a TextView) to hold a particular choice.
         */
        @Override
        public View newView(Context context, Cursor cursor, ViewGroup parent) {
            final LayoutInflater inflater = LayoutInflater.from(context);
            final View view =
                    inflater.inflate(android.R.layout.simple_dropdown_item_1line,
                            parent, false);
 
           return view;
        }

        /**
         * Called by the AutoCompleteTextView field when a choice has been made
         * by the user.
         * 
         * @param listView
         *            The ListView containing the choices that were displayed to
         *            the user.
         * @param view
         *            The field representing the selected choice
         * @param position
         *            The position of the choice within the list (0-based)
         * @param id
         *            The id of the row that was chosen (as provided by the _id
         *            column in the cursor.)
         */
        @Override
        public void onItemClick(AdapterView<?> listView, View view, int position, long id) {
            // Get the cursor, positioned to the corresponding row in the result set
            Cursor cursor = (Cursor) listView.getItemAtPosition(position);

            // Get the state's capital from this row in the database.
            String capital = cursor.getString(cursor.getColumnIndexOrThrow("capital"));

            // Update the parent class's TextView
            mStateCapitalView.setText(capital);
        }
    }

    private TextView mStateCapitalView;
    private AutoCompleteTextView mStateNameView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        AutoCompleteDbAdapter dbHelper = new AutoCompleteDbAdapter(this);
        setContentView(R.layout.selectstate);
        Button confirmButton = (Button) findViewById(R.id.confirm);
        confirmButton.setOnClickListener(new View.OnClickListener() {
            public void onClick(View view) {
                setResult(RESULT_OK);
                finish();
            }
        });

        mStateCapitalView = (TextView) findViewById(R.id.state_capital);
        mStateNameView = (AutoCompleteTextView) findViewById(R.id.state_name);

        // Create an ItemAutoTextAdapter for the State Name field,
        // and set it as the OnItemClickListener for that field.
        ItemAutoTextAdapter adapter = this.new ItemAutoTextAdapter(dbHelper);
        mStateNameView.setAdapter(adapter);
        mStateNameView.setOnItemClickListener(adapter);
    }
}

Here’s the AutoCompleteDbAdapter class. It’s less interesting for this example, so the code is hidden by default. Click on the “show source” link to view it.

package org.oowb.AutoCompleteExample;

import android.app.Activity;
import android.content.ContentValues;
import android.content.Context;
import android.database.Cursor;
import android.database.SQLException;
import android.database.sqlite.SQLiteDatabase;
import android.database.sqlite.SQLiteOpenHelper;
import android.util.Log;

/**
 * Simple database access helper class.
 * 
 * @author Dan Breslau
 */
public class AutoCompleteDbAdapter {
    /**
     * List of states and capitals.
     */
    private static final String[][] States = {
            { "Alabama", "Montgomery" },
            { "Alaska", "Juneau" },
            { "Arizona", "Phoenix" },
            { "Arkansas", "Little Rock" },
            { "California", "Sacramento" },
            { "Colorado", "Denver" },
            { "Confusion", "\"C\"" },
            { "Connecticut", "Hartford" },
            { "Delaware", "Dover" },
            { "Florida", "Tallahassee" },
            { "Georgia", "Atlanta" },
            { "Hawaii", "Honolulu" },
            { "Idaho", "Boise" },
            { "Illinois", "Springfield" },
            { "Indiana", "Indianapolis" },
            { "Iowa", "Des Moines" },
            { "Kansas", "Topeka" },
            { "Kentucky", "Frankfort" },
            { "Louisiana", "Baton Rouge" },
            { "Maine", "Augusta" },
            { "Maryland", "Annapolis" },
            { "Massachusetts", "Boston" },
            { "Michigan", "Lansing" },
            { "Minnesota", "St. Paul" },
            { "Mississippi", "Jackson" },
            { "Missouri", "Jefferson City" },
            { "Montana", "Helena" },
            { "Nebraska", "Lincoln" },
            { "Nevada", "Carson City" },
            { "New Hampshire", "Concord" },
            { "New Jersey", "Trenton" },
            { "New Mexico", "Santa Fe" },
            { "New York", "Albany" },
            { "North Carolina", "Raleigh" },
            { "North Dakota", "Bismarck" },
            { "Ohio", "Columbus" },
            { "Oklahoma", "Oklahoma City" },
            { "Oregon", "Salem" },
            { "Pennsylvania", "Harrisburg" },
            { "Rhode Island", "Providence" },
            { "South Carolina", "Columbia" },
            { "South Dakota", "Pierre" },
            { "Tennessee", "Nashville" },
            { "Texas", "Austin" },
            { "Utah", "Salt Lake City" },
            { "Vermont", "Montpelier" },
            { "Virginia", "Richmond" },
            { "Washington", "Olympia" },
            { "West Virginia", "Charleston" },
            { "Wisconsin", "Madison" },
            { "Wyoming", "Cheyenne" }
    };

    private static final String DATABASE_NAME = "capitals";
    private static final String TABLE_NAME = "state";
    private static final int DATABASE_VERSION = 1;

    private class DatabaseHelper extends SQLiteOpenHelper {

        DatabaseHelper(Context context) {
            super(context, DATABASE_NAME, null, DATABASE_VERSION);
        }

        @Override
        public void onCreate(SQLiteDatabase db) {
            final String DATABASE_CREATE_STATES =
                    "create table " + TABLE_NAME
                            + "(_id integer primary key autoincrement"
                            + ", state text not null"
                            + ", capital text not null)";

            db.execSQL(DATABASE_CREATE_STATES);
            populateWithData(db);
        }

        @Override
        public void onUpgrade(SQLiteDatabase db, int old, int new) {
            db.execSQL("DROP TABLE IF EXISTS " + TABLE_NAME);
            onCreate(db);
        }
    }

    private DatabaseHelper mDbHelper;
    private SQLiteDatabase mDb;
    private final Activity mActivity;

    /**
     * Constructor - takes the context to allow the database to be
     * opened/created
     * 
     * @param activity
     *            the Activity that is using the database
     */
    public AutoCompleteDbAdapter(Activity activity) {
        this.mActivity = activity;
        mDbHelper = this.new DatabaseHelper(activity);
        mDb = mDbHelper.getWritableDatabase();
    }

    /**
     * Closes the database.
     */
    public void close() {
        mDbHelper.close();
    }

    /**
     * Return a Cursor that returns all states (and their state capitals) where
     * the state name begins with the given constraint string.
     * 
     * @param constraint
     *            Specifies the first letters of the states to be listed. If
     *            null, all rows are returned.
     * @return Cursor managed and positioned to the first state, if found
     * @throws SQLException
     *             if query fails
     */
    public Cursor getMatchingStates(String constraint) throws SQLException {

        String queryString =
                "SELECT _id, state, capital FROM " + TABLE_NAME;

        if (constraint != null) {
            // Query for any rows where the state name begins with the
            // string specified in constraint.
            //
            // NOTE:
            // If wildcards are to be used in a rawQuery, they must appear
            // in the query parameters, and not in the query string proper.
            // See http://code.google.com/p/android/issues/detail?id=3153
            constraint = constraint.trim() + "%";
            queryString += " WHERE state LIKE ?";
        }
        String params[] = { constraint };

        if (constraint == null) {
            // If no parameters are used in the query,
            // the params arg must be null.
            params = null;
        }
        try {
            Cursor cursor = mDb.rawQuery(queryString, params);
            if (cursor != null) {
                this.mActivity.startManagingCursor(cursor);
                cursor.moveToFirst();
                return cursor;
            }
        }
        catch (SQLException e) {
            Log.e("AutoCompleteDbAdapter", e.toString());
            throw e;
        }

        return null;
    }

    /**
     * Populates the database with data on states and state capitals.
     * 
     * @param db
     *            The database to be populated; must have the appropriate table
     *            ("state") and columns ("state" and "values") already set up.
     */
    private void populateWithData(SQLiteDatabase db) {
        try {
            db.beginTransaction();
            ContentValues values = new ContentValues();
            // Populate the database with the state/capital city
            // pairs found in the States array.
            for (String[] s : States) {
                values.put("state", s[0]);
                values.put("capital", s[1]);

                db.insert(TABLE_NAME, "state", values);
            }
            db.setTransactionSuccessful();
        }
        finally {
            db.endTransaction();
        }
    }
}

Using SimpleCursorAdapter

As mentioned above, changing the Adapter’s base class to SimpleCursorAdapter is fairly simple. Here’s a summary of the changes needed:

• The SimpleCursorAdapter constructor requires a resource ID for the layout used for items in the list. As before, we are using android.R.layout.simple_dropdown_item_1line for this layout.
• The constructor for SimpleCursorAdapter also requires parameters named from and to. These are arrays indicating which column(s) in the database hold the relevant data, and which field(s) in the View will display that data.
When the SimpleCursorAdapter is used to support an AutoCompleteTextView, only one column/field pair is used. The column, of course, is the column that supplies the names (here, “state”.) But how do we identify the field in the to array? Help comes to us through android.R.id.text1, a resource that is used to identify the first (and only) TextField within the standard item layout (android.R.layout.simple_dropdown_item_1line.)
This results in the following definitions for the from and to parameters. (I defined these in the outer SelectState class, because an inner class isn’t allowed to have static data.)

    final static int[] to = new int[] { android.R.id.text1 };
    final static String[] from = new String[] { "state" };

• The constructor is changed to call its superclass as follows:

super(SelectState.this, 
        android.R.layout.simple_dropdown_item_1line, 
        null, from, to);

• The newView and bindView methods can be removed.

In yesterday’s post on writing decorator classes that decorate methods, I left out two edge cases that can’t be completely ignored: static methods and class methods.

To illustrate, I’ll start where I left off yesterday, adding a decorated class method and a decorated static method to the example:

import types

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))

    @DebugTrace
    @classmethod
    def classHello(cls, to):
        print("*** The {0} class says hello to {1}".format(cls.__name__, to))

    @DebugTrace
    @staticmethod
    def staticHello(to):
        print("*** Something says hello to " + to)


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()
    Greeter.staticHello("you")
    Greeter.classHello("everyone")

greet()

Running this gives an error:

Tracing: hello
Traceback (most recent call last):
  File "DecoratorExample.py", line 17, in <module>
    class Greeter(object):
  File "DecoratorExample.py", line 29, in Greeter
    @classmethod
  File "DecoratorExample.py", line 5, in __init__
    print("Tracing: {0}".format(f.__name__))
AttributeError: 'classmethod' object has no attribute '__name__'

Just for this example, I’ll try removing the “Tracing” print call; but still no joy:

Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Traceback (most recent call last):
  File "DecoratorExample.py", line 48, in <module>
    greet()
  File "DecoratorExample.py", line 14, in __call__
    return self.f(*args, **kwargs)
  File "DecoratorExample.py", line 45, in greet
    Greeter.staticHello("you")
  File "DecoratorExample.py", line 10, in __get__
    return types.MethodType(self, obj)
TypeError: self must not be None

The essential problem is that class methods and static methods are not callable.¹ There’s an easy enough workaround: always use @staticmethod or @classmethod as the outermost (i.e., last) decorator in a sequence, as in:

    @classmethod
    @DebugTrace
    def classHello(cls, to):
        print("*** The Greeter class says hello to " + to)

    @staticmethod
    @DebugTrace
    def staticHello(to):
        print("*** Something says hello to " + to)

That produces the desired result:

Tracing: hello
Tracing: classHello
Tracing: staticHello
Tracing: greet
Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Calling: staticHello
*** Something says hello to you
Calling: classHello
*** The Greeter class says hello to everyone

But suppose we really, really need to decorate an already-decorated classmethod or staticmethod. The key lies again in the descriptor protocol.

First, we need to modify the decorator’s __init__ method. (Note that the only reason that we need to modify __init__ is to find the name of the classmethod or staticmethod that’s being decorated. If we didn’t produce the “Tracing:” output, we could leave __init__ alone.)

The new __init__ method detects whether the passed “function” has a __call__ method. If it doesn’t, then it’s reasonable to assume that it’s a classmethod or a staticmethod. Calling the object’s __get__ method returns a function object, from which we can get the function name:

    def __init__(self, f):
        self.f = f
        if hasattr(f, "__call__"):
            name = self.f.__name__
        else:
            # f is a class or static method.
            tmp = f.__get__(None, f.__class__)
            name = tmp.__name__
        print("Tracing: {0}".format(name))

In the decorator’s __get__ method, we’ll know that we’re dealing with a staticmethod or classmethod if the passed obj has the value None. If that’s the case, then we make a one-time adjustment to self.f, ensuring that it points to the underlying function.

Wait—why didn’t we do this in DebugTrace.__init__? It may seem redundant, but the call to f.__get__ that we made in DebugTrace.__init__ doesn’t count: that call didn’t specify the class that f actually belongs to. (Any class works for the purpose of getting the function’s name.) Now that we’re in DebugTrace.__get__, we know via the ownerClass parameter the class that self.f is associated with. This class may make its way into a classmethod call (e.g., the call to Greeter.classHello), so it matters that we get it right.

Note that we return self in this case. We don’t want to create a new method object for classmethods or staticmethods; just calling self.__call__ will call the method appropriately.

    def __get__(self, obj, ownerClass=None):
        if obj is None:
            f = self.f
            if not hasattr(f, "__call__"):
                self.f = f.__get__(None, ownerClass)
            return self
        else:
            # Return a wrapper that binds self as a method of obj (!)
            return types.MethodType(self, obj)

            return self

            return self.__class__(self.f)

Tracing: hello
Tracing: classHello
Tracing: staticHello
Tracing: greet
Calling: greet
Calling: hello
*** Greeter 1 says hello!
Calling: hello
*** Greeter 2 says hello!
Tracing: staticHello
Calling: staticHello
*** Something says hello to you
Tracing: classHello
Calling: classHello
*** The Greeter class says hello to everyone

The decorator’s __call__ method is unchanged from yesterday’s example. As before, it simply prints out the desired trace message, then invokes self.f.

Here’s the entire decorator, as revised:

import types

class DebugTrace(object):
    def __init__(self, f):
        self.f = f
        if hasattr(f, "__call__"):
            name = self.f.__name__
        else:
            # f is a class or static method
            tmp = f.__get__(None, f.__class__)
            name = tmp.__name__
        print("Tracing: {0}".format(name))

    def __get__(self, obj, ownerClass=None):
        if obj is None:
            f = self.f
            if not hasattr(f, "__call__"):
                self.f = f.__get__(None, ownerClass)
            return self
        else:
            # Return a wrapper that binds self as a method of obj (!)
            return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))

    @DebugTrace
    @classmethod
    def classHello(cls, to):
        print("*** The {0} class says hello to {1}".format(cls.__name__, to))

    @DebugTrace
    @staticmethod
    def staticHello(to):
        print("*** Something says hello to " + to)


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()
    Greeter.staticHello("you")
    Greeter.classHello("everyone")

greet()

I’ve tested this with Python 2.6, 2.7, and 3.1.

As a fairly new Python developer, my first attempt at decorators hit a snag: my simple class-based decorator failed when decorating a method. I got around the immediate problem by rewriting the decorator as a function. Yet the episode left me wondering if there were some way to fix the class-based decorator to work when applied to methods. I’ve found what seems like an elegant solution, and picked up a better understanding of decorators and descriptors in the process.

Here’s an example that illustrates the original problem. DebugTrace is the decorator class:

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Running this with Python 2.6 or 3.1 results in an error:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
Traceback (most recent call last):
  File "./DecoratorExample.py", line 31, in <module>
    greet()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
  File "./DecoratorExample.py", line 27, in greet
    g.hello()
  File "./DecoratorExample.py", line 8, in __call__
    return self.f(*args, **kwargs)
TypeError: hello() takes exactly 1 argument (0 given)

The output explains the problem. DebugTrace was instantiated only twice: once for Greeter.hello, and once for greet. That’s a reminder that decoration occurs during compile time, not run time. Accordingly, DebugTrace‘s reference to Greeter.hello represents an unbound function—it doesn’t reference any Greeter instances. So no ‘self’ argument was passed into the call to Greeter.hello; hence the TypeError.

All object-oriented languages that are worth knowing (and many that aren’t) allow container objects to redirect access to their contained objects. But of the languages that I’ve used, Python is unique in allowing object attributes to redirect calls that attempt to access them. Objects that implement this capability are called descriptors¹. As we’ll soon see, function objects, which Python uses to implement methods, are descriptors.

To see where descriptors come into play, let’s look at the calling sequences for different versions of Greeter.hello. Here’s the rough sequence before Greeter.hello was decorated:

• The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
• The value of the Greeter.hello attribute (a function object) has a __get__ method, making it a descriptor, so that __get__ method is invoked. It’s passed a reference to the Greeter instance (obj) through which Greeter.hello was called.
• The function object’s __get__ method creates and returns another callable object, which we’ll refer to as a (bound) method object. The method object references both the Greeter.hello function and obj.
• The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

When Greeter.hello is decorated with the DebugTracer class as shown above, a call to Greeter.hello runs more or less like this:

• The interpreter searches for an attribute named hello on the Greeter instance, finding it in the Greeter class object.
• The value of the Greeter.hello attribute is an instance of DebugTrace. This isn’t a function object, and it doesn’t have a __get__ method, but it does have a __call__ method. That __call__ method is invoked with the empty argument list.
• DebugTrace.__call__ then calls Greeter.hello with the empty argument list.
• Since Greeter.hello was looking for a single argument (self), rather than an empty argument list, a TypeError is raised.

To fix DebugTrace, I turned it into a descriptor class, adding a __get__ method that fills the same role as a function object’s __get__ method. However, this method binds the Greeter instance to the callable DebugTrace object.

import types
# ...
    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

Compare the new calling sequence for Greeter.hello to the sequence prior to decoration:

• The interpreter finds an attribute named hello in the Greeter class object.
• The value of the Greeter.hello attribute is an instance of DebugTrace, which is now a descriptor. DebugTrace.__get__ is called, with obj (the Greeter instance) passed as one of the arguments.
• DebugTrace.__get__ creates and returns a method object. The method object references both the DebugTrace instance and obj.
• The interpreter invokes the method object, passing it the arguments from the call to Greeter.hello (an empty list.) The method object then calls DebugTrace.__call__. That in turn calls Greeter.hello, passing obj as the first argument, followed by the (empty) argument list.

It’s worth noting that DebugTrace.__get__ is only invoked when accessing a DebugTrace object through an object’s class dictionary. Hence its presence has no effect on functions that aren’t methods, such as greet.

You can see the full, working example here (click on the “show source” link to view) :

import types

class DebugTrace(object):
    def __init__(self, f):
        print("Tracing: {0}".format(f.__name__))
        self.f = f

    def __get__(self, obj, ownerClass=None):
        # Return a wrapper that binds self as a method of obj (!)
        return types.MethodType(self, obj)

    def __call__(self, *args, **kwargs):
        print("Calling: {0}".format(self.f.__name__))
        return self.f(*args, **kwargs)


class Greeter(object):
    instances = 0

    def __init__(self):
        Greeter.instances += 1
        self._inst = Greeter.instances

    @DebugTrace
    def hello(self):
        print("*** Greeter {0} says hello!".format(self._inst))


@DebugTrace
def greet():
    g = Greeter()
    g2 = Greeter()
    g.hello()
    g2.hello()


greet()

Executing the new version gives the desired output:

Tracing: hello
Tracing: greet
Calling greet
Calling hello
*** Greeter 1 says hello!
Calling hello
*** Greeter 2 says hello!

<html>
<head>
	<!-- Stylesheets for SyntaxHighlighter -->
<link type="text/css" rel="stylesheet" href="styles/shCore.css"/>
<link type="text/css" rel="stylesheet" href="styles/shThemeDefault.css"/>
...
</head>
<body>

<em>...Some kind of interesting page content usually goes here, 
but we're not interested in that right now...</em>

	<!-- Load the SyntaxHighlighter Core and Brush scripts. A 
               separate Brush script is required for each language. -->
<script type="text/javascript" src="scripts/shCore.js"></script>
<script type="text/javascript" src="scripts/shBrushJava.js"></script>
<script type="text/javascript" src="scripts/shBrushJScript.js"></script>
	<!-- Don't forget the XML brush if you're using html-script -->
<script type="text/javascript" src="scripts/shBrushXml.js"></script>

	<!-- Now that all the script files are loaded, begin highlighting. -->
<script type="text/javascript">
SyntaxHighlighter.config.stripBrs = true;
// ... additional configuration, as needed ...

SyntaxHighlighter.all();
</script>

</body>

Some users have wanted a simpler way to integrate SyntaxHighlighter into their sites. Last year, David Chambers wrote a script that uses the Prototype library to load the required CSS and brush files. He ran into a troubling quirk in SyntaxHighlighter: It needs any required brush files to be loaded before it starts highlighting. I responded by updating the OOWB fork of SyntaxHighlighter, so that brushes could be loaded asynchronously. But that’s as far as I’d gone towards automated loading, until now.

I recently looked into adding automated loading to SyntaxHighlighter itself, coming up with not one, but two automated solutions. I should really say one and a half, as one is still in the alpha stage and will likely stay there.

The “alpha-stage” approach comprises a new JavaScript file that loads SyntaxHighlighter and the necessary brush files through AJAX, by way of a pair of simple Java servlets. I’d hoped that this would lead to faster load times, by reducing the total number of downloads. And so it does—sometimes. For testing, I used Google App Engine as the servlet host (using the free quotas.) Performance was inconsistent, to say the least; this is probably due to the Google App Engine’s method of starting up and shutting down servlets. It seemed that only by making repeated hits on the servlets in rapid succession could I be sure of getting to a container that already had my servlets running, thus getting good load times.¹ The servlets are still up there; if you’d like the Java source (to host them elsewhere), and/or the JavaScript for using them, just let me know.

I also implemented a pure JavaScript approach, which I hadn’t expected to yield better load times. Yet I’ve seen a number of cases where this is faster than the original, static way of loading SyntaxHighlighter. I’m releasing a new download so that you can try it out. I highly recommend that you test it thoroughly before putting it into real use.

To use this new method, the web page invokes SyntaxHighlighter.boot instead of SyntaxHighlighter.all. When boot is used as the entry point, SyntaxHighlighter determines which brushes are required by the web page, and issues HTTP requests for each brush file (as well as the appropriate CSS.) These requests use the age-old method of adding a tag for each brush, and a tag for each CSS file.

Using SyntaxHighlighter.boot, a web page only needs to add a couple of scripts, usually at the end of the . No CSS link elements are required.

<!--  We just need to load shCore.js near the end of the <body> element,
	then call SyntaxHighlighter.boot() with our configuration options. -->

<script type="text/javascript"
src="http://path/to/sh/directory/syntaxhighlighter/scripts/shCore.js"></script>

<script type="text/javascript">
	SyntaxHighlighter.boot(
		"http://path/to/syntaxhighlighter/root/syntaxhighlighter/",
		{theme : "Default"}, // Configuration settings
		{stripBrs : true}      // Default settings
	);
</script>

</body>

That’s it. Every page on your site can use the same set-up, regardless of which brush files it needs. (For another example, look at the HTML source for this page.)

When an HTML page includes external scripts—that is, elements with src attributes, the browser loads the script files synchronously, as it encounters them in the HTML. When the scripts are loaded dynamically, the browser can load them asynchronously; several scripts may then be loading simultaneously. This can be particularly helpful for pages that use multiple brushes. Here’s a Firebug sequence diagram of a web page with five brushes, loading in SyntaxHighlighter’s traditional way:

(Click on image for larger view)

Here’s the same page, modified to use the boot method. Notice that, unlike in the first picture, the shBrush...js files are loaded in parallel:

(Click on image for larger view)

Of course, this isn’t an entirely new discovery, but it came as a pleasant surprise nonetheless.

Availability

In addition to SyntaxHighlighter itself, I also modified Viper007Bond’s SyntaxHighlighter Evolved plugin for WordPress to use the new boot method. This simplified the code quite a bit, though it may have introduced bugs.

SyntaxHighlighter and SyntaxHighlighter Evolved are each available on the downloads page. Test before using. Note: If you’ve been using SyntaxHighlighter Evolved, you will probably need to re-set your settings if you install this plugin.

Implementation notes

In the traditional SyntaxHighlighter, all brushes must be loaded before SyntaxHighlighter.all is invoked. In this fork, “>brushes can load at any time after shCore.js has been loaded. This flexibility was key to the design of the boot method: If a brush is loaded after SyntaxHighlighter.all or SyntaxHighlighter.boot is invoked, SyntaxHighlighter looks through the page to see if there’s any input requesting the newly loaded brush.

The boot method needs to map each brush name to the script file that implements that brush. I implemented this by modifying the Perl script that builds and stages SyntaxHighlighter. The script now parses the *Brush.js files in the source tree, pulling out the names and aliases supported by each file, and writes them into an array in shCore.js. The downside of this is that testing via the boot method requires re-running the script, which builds and stages the files into a new directory, from where I can run tests. Even though the script only takes about a second, it’s crossed the boundary between having no build step in the development cycle, and having any build step in the development cycle.

When You Come To A Fork In The Road, Take It.

I had two motives for working on SyntaxHighlighter: One, to improve its display of code on my own blog and elsewhere. Two, as a training ground for learning JavaScript.

I was between jobs when I took on this project. As of this Monday, that will no longer be the case. That’s great news for me, but much as I’d like to keep the SyntaxHighlighter work moving forward, I don’t really know if or when I’ll have the time for it. On the other hand, I no longer feel like I need a private JavaScript training ground. So, I’ve pushed my repository to bitbucket.org as a fork of Alex’s repository. If he so chooses (and so far he hasn’t, which is fine), Alex is welcome to merge the fork into his branch. Perhaps more importantly, anyone else can now fork the code, and/or make contributions to this fork.

I do have some more goals in mind for SyntaxHighlighter. I’ll be writing about them in the not-too-distant future.

¹Perhaps a faster host would lead to the faster load times I’d hoped for; but that would probably require payment, which makes this approach considerably less attractive.

• In 2.1.364, the ruler functionality was removed. It’s still included here.
• 2.1.364 changed the way that wrapped lines are presented. I’ve kept the old presentation, but modified the icon that’s used to signify wrapping.
• Internally, 2.1.364 uses a separate HTML table to represent each line of source. I’ve maintained the older pattern of simply using a separate <div> to represent each line.

Other new bits in this release:

• I’ve added a “Hide” button to the toolbar. Code blocks can now be expanded/collapsed at the user’s whim. (As before, an author can still choose to present a block in the hidden/collapsed state.)
• When both a horizontal scrollbar and a gutter (the line number column) are displayed, the scrollbar no longer extends under the gutter. (I think this is an improvement; as always, feedback is welcome.) Update: I’ve backed out this change; it was causing problems with IE7.

The ZIP file is available through the Download Page.

New version of SyntaxHighlighter coming

Some folks have asked me offline if I’m going to be merging Alex Gorbatchev’s 2.1.364 version of SyntaxHighlighter with the OOWB fork that I’ve been supporting here. The answer to that is yes; if all goes well, I should be posting it in the next few days. As for creating a public repository, perhaps on bitbucket: Maybe. If you’d like to see this happen, please let me know. Actually, I’d like to hear from anyone who’s rolled out the OOWB fork: What site(s) are you using it on, and how is it working out? Please leave a comment, or send email.

Did I say “fork”? Uh, yeah. While 2.1.364 is solid, it doesn’t cover most of the improvements that I’ve made (check the Download Page to see a summary.) Ultimately, I’d like to see these changes rolled back into Alex’s version, but that doesn’t appear to be happening anytime soon. There could be good reasons for that; certainly my version looks larger, just for starters. (I think that the total footprint may be smaller, but that’s a tough call.) All I can really say is that we’ll see how it goes. I plan to keep making improvements in the software, at least for my own sake; but I regard Alex as the primary author and will continue to do so.