This blog post is kind of a followup of the blog posting "Building interactive Javascript Charts" by my friend Simon Pamies. Simon implemented several sites using the Highcharts plugin and I am current working on a followup project. The objective of the portal is the optimization of the process of buying gas for large industrial companies. Gas is handled like every other piece of energy at dedicated exchanges. There are various gas products. So the decision when to buy gas and how much gas in advance etc. are a complex decision making process. The portal will give the company condensed information about their current gas consumption, the current prices for various products etc. It is both a tracking tool of the own gas consumption, it has an option for creating a projection of the consumption in the future, it allows to plan and buy gas and calculates various benchmarks. You can image that the portal has to deal with lots of data that need to be visualized.

For the project we decided to continue working with Highcharts for  rendering nice and powerful interactive charts.

For the representation of various long data tables we choose the jqGrid Javascript plugin.

Here are some screenshots:

Some remarks on the backend implementation.

We are using MongoDB for all the data. The ZODB basically is used only for holding the data of an almost naked Plone site (with some content like news etc.). MongoDB allows us to store mass data very easily and to aggregate them very easily. The "schema-less" approach of MongoDB allowed us to explore schema and database changes easily. To mention some numbers: aggregating and processing the data for a complex charts with roughtly 50.000 points takes less than half a second (data retrieval, processing, conversion to JSON). So: it's fast!

The Highcharts Javascript library for generating interactive and complex charts seems to be the best multipurpose solution for doing charts inside a browser. I investigated also other open-source chart solutions but brought to the point: tinkering stuff that can be used for kindergarten-style applications. Look at the demos on highcharts.com and decide yourself. Yes, it is a commercial solution but worth each cent. Highcharts has decent documentation and support is usually working without major issues (instead of creating new problems).

jqGrid is a powerful data-grid module implemented in Javascript  which also satisfied all of my needs. The documentation is partly unsorted or scattered around. It support large data tables, row can be edited inline or through an edition form. It provides a limited data-type model and validation support. Most of the missing aspects can be implemented through a custom method that is registered for some hook (jqGrid has lots of hooks).

Some remarks on browsers...complex charts (50.000 to 100.000 data points per charts) let browsers sweating. Currently our application is not usable with decent Firefox versions (tested with Firefox 10+). Rendering a chart with current Chrome or Opera versions takes up to three seconds for a complex chart. Also IE8+ is doing a pretty good job. Firefox sometimes takes more than 45 seconds (including complaints about long-running Javascript code). We would not the problem down to either a bug in Highcharts or Firefox.