Thursday, 13 May 2010


When a support ticket comes into a Help Desk at an institution, the help desk staff need to be able to determine what the user was trying to do and what they want to achieve. Often this information can be gathered from a conversation with the user, but frequently the Help Desk staff needs to interrogate the back end to discover the operations the user performed at the server and the state of the data that they were interacting with.
This project will address the area of exploration of events that have happened in the system allowing a Help Desk user to navigate to content and pages that the user was working on. It will focus on the collection, storage, analysis and exploration of the event based information from a cluster of Nakamura servers.

The three essential stages of the project are:

-Event creation and Processing
-Storage and Analysis
-UI Development
Event Processing:

This involves the recieving of JMS messages created and emitted by OsgiJmsBridge by an AMQ consumer and processing the message to store the vital information in the database (Cassandra). This will require configuring the AMQ brokers and setting up of consumers clients of brokers present in the network. It will also be including some Camel queries to route the messages over the network of brokers to the appropriate consumer. Once the consumer recieves the message, it will process the message and will include the code for transferring the processed data to Cassandra using Thrift potocol.

Storage and Analysis:


I will be using Cassandra for data storage and Apache Hadoop for data analysis. Cassandra is a column database with NoSQL implementation. Usage of Cassandra will be very beneficial in terms of future scalability. Apart from this, the main feature of Cassandra is its efficiency and reliability. It has much less read and write time than relational databases.


The incoming data into Cassandra from JMS will be stored in columns and column-families according to the format that will be decided. The Cassandra dataset is integrated with Hadoop. Hadoop MapReduce can talk to Cassandra and process data. A Hadoop cluster is formed which will use Pig Latin to query the cluster and retrieve information for analysis from the pig server running on user machine.


This is an initial plan of how the UI is going to work. Additional work rill be done after consulting some of the help-desk users and creating some prototypes according to their needs.
The UI creation would include the following:
A Java Thrift API for accessing the Cassandra database. This would be a java class with specifications for Thrift code. The retrived data would be persed and converted to JSON object. This object would be loaded in our JSP page which would be the page for the help-desk user. The JSP page would contain the Javascript for the Simile tools (example: Timeline [1]). The JSON feeds created will be loaded in the JSP page using inbuilt functions. Hence the data retrieved from the database will be displayed in the view page using Simile tool.

1 comment:

  1. Hi Ashish, thanks for introducing event explorer. One piece of context I'd like to add is how super-important it is for an enterprise application to build in efficient support tools. Sakai is open source, but it still costs institutions money to maintain and support it. Everyone who is deciding whether to go open source or commercial will make this crucial calculation. So the more efficient we make support, the more big adopters Sakai will get. This is serious work you are doing!