Creating a custom JavaScript widget in the notebook – a spreadsheet editor for pandas

We have previously introduced the new interactive features of the IPython notebook 2.0. In this recipe, we dive deeper into the subject by showing how to go beyond the existing widgets provided by IPython 2.0. Specifically, we will create a custom JavaScript-based widget that communicates with the Python kernel.

Specifically, we will create a basic interactive Excel-like data grid editor in the IPython notebook, compatible with pandas' DataFrame. Starting from a DataFrame object, we will be able to edit it within a GUI in the notebook. The editor is based on the Handsontable JavaScript library (http://handsontable.com). Other JavaScript data grid editors could be used as well.

Getting ready

You will need both IPython 2.0+ and the Handsontable JavaScript library for this recipe. The following are the instructions to load this Javascript library in the IPython notebook:

1. First, go to https://github.com/handsontable/jquery-handsontable/tree/master/dist.
2. Then, download jquery.handsontable.full.css and jquery.handsontable.full.js, and put these two files in ~\.ipython\profile_default\static\custom\.
3. In this folder, add the following line in custom.js:

   require(['/static/custom/jquery.handsontable.full.js']);

4. In this folder, add the following line in custom.css:

   @import "/static/custom/jquery.handsontable.full.css"

5. Now, refresh the notebook!

How to do it...

1. Let's import a few functions and classes as follows:

   In [1]: from IPython.html import widgets
           from IPython.display import display
           from IPython.utils.traitlets import Unicode

2. We create a new widget. The value trait will contain the JSON representation of the entire table. This trait will be synchronized between Python and JavaScript, thanks to the IPython 2.0's widget machinery.

   In [2]: class HandsonTableWidget(widgets.DOMWidget):
               _view_name = Unicode('HandsonTableView',
                                    sync=True)
               value = Unicode(sync=True)

3. Now, we write the JavaScript code for the widget. The three important functions that are responsible for the synchronization are as follows:
   • render is for the widget initialization
   • update is for Python to JavaScript update
   • handle_table_change is for JavaScript to Python update

     In [3]: %%javascript
     var table_id = 0;
     require(["widgets/js/widget"], function(WidgetManager){    
         // Define the HandsonTableView
         var HandsonTableView = IPython.DOMWidgetView.extend({
             
             render: function(){
                 // Initialization: creation of the HTML elements
                 // for our widget.
                 
                 // Add a <div> in the widget area.
                 this.$table = $('<div />')
                     .attr('id', 'table_' + (table_id++))
                     .appendTo(this.$el);
     
                 // Create the Handsontable table.
                 this.$table.handsontable({
                 });
                 
             },
             
             update: function() {
                 // Python --> Javascript update.
                 
                 // Get the model's JSON string, and parse it.
                 var data = $.parseJSON(this.model.get('value'));
     
                 // Give it to the Handsontable widget.
                 this.$table.handsontable({data: data});
                 
                 return HandsonTableView.__super__.
                                              update.apply(this);
             },
             
             // Tell Backbone to listen to the change event 
             // of input controls.
             events: {"change": "handle_table_change"},
             
             handle_table_change: function(event) {
                 // Javascript --> Python update.
                 
                 // Get the table instance.
                 var ht = this.$table.handsontable('getInstance');
     
                 // Get the data, and serialize it in JSON.
                 var json = JSON.stringify(ht.getData());
     
                 // Update the model with the JSON string.
                 this.model.set('value', json);
                 
                 this.touch();
             },
         });
         
         // Register the HandsonTableView with the widget manager.
         WidgetManager.register_widget_view(
             'HandsonTableView', HandsonTableView);
     });

4. Now, we have a synchronized table widget that we can already use. However, we would like to integrate it with pandas. To do this, we create a light wrapper around a DataFrame instance. We create two callback functions for synchronizing the pandas object with the IPython widget. Changes in the GUI will automatically trigger a change in DataFrame, but the converse is not true. We'll need to re-display the widget if we change the DataFrame instance in Python:

   In [4]: from io import StringIO
           import numpy as np
           import pandas as pd
   In [5]: class HandsonDataFrame(object):
               def __init__(self, df):
                   self._df = df
                   self._widget = HandsonTableWidget()
                   self._widget.on_trait_change(
                              self._on_data_changed, 'value')
                   self._widget.on_displayed(self._on_displayed)
                   
               def _on_displayed(self, e):
                   # DataFrame ==> Widget (upon initialization)
                   json = self._df.to_json(orient='values')
                   self._widget.value = json
                   
               def _on_data_changed(self, e, val):
                   # Widget ==> DataFrame (called every time the
                   # user changes a value in the widget)
                   buf = StringIO(val)
                   self._df = pd.read_json(buf, orient='values')
                   
               def to_dataframe(self):
                   return self._df
                   
               def show(self):
                   display(self._widget)

5. Now, let's test all that! We first create a random DataFrame instance:

   In [6]: data = np.random.randint(size=(3, 5),
                                    low=100, high=900)
           df = pd.DataFrame(data)
           df
   Out[6]:      
   352  201  859  322  352
   326  519  848  802  642
   171  480  213  619  192

6. We wrap it in HandsonDataFrame and show it as follows:

   In [7]: ht = HandsonDataFrame(df)
           ht.show()

   
7. We can now change the values interactively, and they will be changed in Python accordingly:

   In [8]: ht.to_dataframe()
   Out[8]:
   352  201  859   322  352
   326  519  848  1024  642
   171  480  213   619  192

How it works...

Let's explain briefly the architecture underlying the interactive Python-JavaScript communication in IPython 2.0+.

The implementation follows the Model-View-Controller (MVC) design pattern, which is popular in GUI applications. There is a model in the backend (Python kernel) that holds some data. In the frontend (browser), there are one or several views of that model. Those views are dynamically synchronized with the model. When an attribute of the model changes on Python's side, it also changes on JavaScript's side, and vice versa. We can implement Python and JavaScript functions to respond to model changes. These changes are generally triggered by a user action.

In Python, dynamic attributes are implemented as traits. These special class attributes automatically trigger callback functions when they are updated. In JavaScript, the Backbone.js MVC library is used. The communication between Python and the browser is done via Comms, a special communication protocol in IPython.

To create a new widget, we need to create a class deriving from DOMWidget. Then, we define trait attributes that can be synchronized between Python and JavaScript if sync=True is passed to the trait constructors. We can register callback functions that react to trait changes (from either Python or JavaScript), using widget.on_trait_change(callback, trait_name). The callback() function can have one of the following signatures:

• callback()
• callback(trait_name)
• callback(trait_name, new_value)
• callback(trait_name, old_value, new_value)

In JavaScript, the render() function creates the HTML elements in the cell's widget area upon initialization. The update() method allows us to react to changes in the model in the backend side (Python). In addition, we can use Backbone.js to react to changes in the frontend (browser). By extending the widget with the {"change": "callback"} events, we tell Backbone.js to call the callback() JavaScript function as soon as the HTML input controls change. This is how we react to user-triggered actions here.

There's more...

The following are the ways this proof-of-concept could be improved:

• Synchronizing only changes instead of synchronizing the whole array every time (the method used here would be slow on large tables)
• Avoiding recreating a new DataFrame instance upon every change, but updating the same DataFrame instance in-place
• Supporting named columns
• Hiding the wrapper, that is, make it so that the default rich representation of DataFrame in the notebook is HandsonDataFrame
• Implementing everything in an easy-to-use extension

Here are a few references about the widget architecture in the IPython notebook 2.0+:

• Official example about custom widgets, available at http://nbviewer.ipython.org/github/ipython/ipython/tree/master/examples/Interactive%20Widgets
• MVC pattern in Wikipedia, at https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller
• Backbone.js, available at http://backbonejs.org/
• Course on Backbone.js, available at www.codeschool.com/courses/anatomy-of-backbonejs
• IPEP 21: Widget Messages (comms), available at https://github.com/ipython/ipython/wiki/IPEP-21%3A-Widget-Messages
• IPEP 23: IPython widgets, available at https://github.com/ipython/ipython/wiki/IPEP-23%3A-Backbone.js-Widgets

See also

• The Processing webcam images in real time from the notebook recipe