GSoC pencils down

Congratulations to our successful GSoC students Aadish, Mai and Kasun on completing GSoC, and for their excellent contributions to Sakai.  It's been a pleasure working with you. 

Unfortunately it's 'pencils down' for the official project, and all that remains is to collect the T-shirt and to plug Sakai at the GSoC Mentors' conference at Google HQ in Mountain View.  Carl Hall will be ably representing again this year - don't forget to take Sakaiger stickers Carl.

Carl is also going to be taking over from me to admin Sakai GSoC next year, provided Google continues to support this excellent programme, so watch this space for an exciting new chapter.

Summer and HBase!

I spent most of my summer dealing and learning Hadoop’s column database called HBase. Why this database? To put in Ian Boston’s words : “The reason for HBase is its the DB of choice for many systems that want to do large scale data analysis on Hadoop.” Ian declared that the choice of database was HBase on June 27 and since then, I must have clicked more number of links on Google for ‘HBase’ than perhaps the number of HBase deployments itself! Then started the quest of mine, to reconsider database driver in light of this new database, and code for the tests which would judge its performance. After a plethora of idea exchange the code was written and tests were implemented. This post aims at explaining and shedding some light on deployment of HBase as how I had done it during the summer.

World, HBase. HBase, world.

People coming from RDBMS background, just shake your head vigorously, and forget all the schema that you have ever implemented in your life! To absorb concepts of HBase, just vacuum clean your head and make room for new concepts to flow in.

Why HBase?

To answer that question I’ll have to walk you through the problems of traditional databases. Traditional databases face basically two types of problems: First is scaling and the second one, well, can be called ‘sparse-ity’. Traditional RDBMS may be reliable, widely used, developer friendly blah blah blah, but when you ask it “How do I scale you”, it would reply “Put more money in me and buy more hardware”. The second problem can be addressed thus: Imagine you are trying to stuff an intricate object graph, with many interdependent objects and relations into RDBMS schema. Definitely, you are bound to end up with a schema wherein an object may have several attributes, which well, may be seldomly used. Your RDBMS surely is going to charge you for all those extra ‘NULL’ references out there! So how does HBase deal with this, lets see!

HBase and scaling :

HBase is specialized column DB(more on it soon), mastered in scaling. It partitions horizontally and ‘distributes’ data over huge number of commodity servers. HBase is built on Hadoop , which implements functionality similar to Google's GFS and Map/Reduce systems. It provides means to efficiently organize and serve huge amount of data. If you are more interested read Google’s BigTable Paper and Map/Reduce concepts.

HBase and ‘sparse-ity’:

HBase is a column oriented database. This means that it stores contents in form of columns rather than rows. This frees up the need of attributes which may not be necessary for an object. In row DBs, where you would have nineteen NULLs and one attribute, HBase(or rather any column DB) would only save that one attribute. So, less room for storage, and high speed performance!

HBase datamodel:

An excellent read for this would be : http://wiki.apache.org/hadoop/Hbase/DataModel

How I went about setting up HBase for SparseMapContent:

Configuring Apache HBase database on Windows:
The page describes how to configure Apache HBase in a standalone mode on Windows using Cygwin.

For Windows environment, 3 technologies are required which are JAVA, SSH and Cygwin.

Installing JAVA:
Download the standard Edition JAVA plateform from here and follow the simple GUI wizard to install the same.

Installing Cygwin
Cygwin provides *nix like environment in Windows. Steps for installation are as follows:
1. Make sure you have Administrator privileges on the target system.

2. Create Root and Local Package directories. A good suggestion is to use C:\cygwin\root and C:\cygwin\setup folders.

3. Download the setup.exe utility from here and save it to the Local Package directory.

4. Run the setup.exe utility,

1. Choose the Install from Internet option,
2. Choose your Root and Local Package folders
3. and select an appropriate mirror.
4. Don't select any additional packages yet, as we only want to install Cygwin for now.
5. Wait for download and install
6. Finish the installation
5. Add CYGWIN_HOME system-wide environment variable that points to your Root directory.
6. Add %CYGWIN_HOME%\bin to the end of your PATH environment variable.
7. Reboot the sytem after making changes to the environment variables otherwise the OS will not be able to find the Cygwin utilities.
8. Test your installation by running your freshly created shortcuts or the Cygwin.bat command in the Root folder. You should end up in a terminal window that is running aBash shell. Test the shell by issuing following commands:

1. cd / should take you to the Root directory in Cygwin;
2. the LS commands that should list all files and folders in the current directory.
3. Use the exit command to end the terminal.

9. When needed, to uninstall Cygwin you can simply delete the Root and Local Package directory, and the shortcuts that were created during installation.

Installing SSH:
HBase (and Hadoop) rely on SSH for interprocess/-node communication and launching remote commands.

1. Rerun the setup.exe utility.
2. Leave all parameters as is, skipping through the wizard using the Next button until the Select Packages panel is shown.
3. Maximize the window and click the View button to toggle to the list view, which is ordered alphabetically on Package, making it easier to find the packages we'll need.
4. Select the following packages by clicking the status word (normally Skip) so it's marked for installation. Use the Next button to download and install the packages.

1. OpenSSH
2. tcp_wrappers
3. diffutils
4. zlib
5. Wait for the install to complete and finish the installation.

Installing HBase
Downlaod HBase from here, unzip it and place it under the directory C:\cygwin\usr\local\ so that it gets installed in Cygwin(C:\cygwin\usr\local\hbase-)

Configuring JAVA
1. Create a symbolic link in /usr/local to the Java home directory by using the following command and substituting the name of your chosen Java environment:
LN -s /cygdrive/c/Program\ Files/Java/ /usr/local/

2. Test your java installation by changing directories to your Java folder CD /usr/local/ and issueing the command ./bin/java -version. This should output your version of the chosen JRE.

Configuring SSH
1. On Windows Vista and above make sure you run the Cygwin shell with elevated privileges, by right-clicking on the shortcut an using Run as Administrator.

2. First of all, make sure that the rights on some crucial files are correct. Use the commands underneath and you can verify all rights by using the LS -L command on the different files. Also, notice the auto-completion feature in the shell using is extremely handy in these situations.

1. chmod +r /etc/passwd to make the passwords file readable for all
2. chmod u+w /etc/passwd to make the passwords file writable for the owner
3. chmod +r /etc/group to make the groups file readable for all
4. chmod u+w /etc/group to make the groups file writable for the owner
5. chmod 755 /var to make the var folder writable to owner and readable and executable to all

3. Edit the /etc/hosts.allow file using your favorite editor (why not VI in the shell!) and make sure the following two lines are in there before the PARANOID line:

1. ALL : localhost 127.0.0.1/32 : allow
2. ALL : [::1]/128 : allow

4. Next we have to configure SSH by using the script ssh-host-config. The following may be asked in random order but don’t worry about that.

1. If this script asks to overwrite an existing /etc/ssh_config, answer yes.
2. If this script asks to overwrite an existing /etc/sshd_config, answer yes.
3. If this script asks to use privilege separation, answer yes.
4. If this script asks to install sshd as a service, answer yes. Make sure you started your shell as Adminstrator!
5. If this script asks for the CYGWIN value, just as the default is ntsec.
6. If this script asks to create the sshd account, answer yes.
7. If this script asks to use a different user name as service account, answer no as the default will suffice.
8. If this script asks to create the cyg_server account, answer yes. Enter a password for the account.

5. Start the SSH service using net start sshd or cygrunsrv --start sshd. Notice that cygrunsrv is the utility that make the process run as a Windows service. Confirm that you see a message stating that the CYGWIN sshd service was started succesfully.

6. Harmonize Windows and Cygwin user account by using the commands:

1. mkpasswd -cl > /etc/passwd
2. mkgroup --local > /etc/group

7. Test the installation of SSH:

1. Open a new Cygwin terminal
2. Use the command whoami to verify your userID
3. Issue an ssh localhost to connect to the system itself
1. Answer yes when presented with the server's fingerprint
2. Issue your password when prompted
3. test a few commands in the remote session
4. The exit command should take you back to your first shell in Cygwin
5. Exit should terminate the Cygwin shell.

8. If you get stuck with some password problem, you can change it using the command passwd.


Configuring HBase

(2nd and 3rd steps are optional.)
1. HBase uses the ./conf/hbase-env.sh to configure its dependencies on the runtime environment. Copy and uncomment following lines just underneath their original, change them to fit your environemnt. They should read something like:

1. export JAVA_HOME=/usr/local/
2. export HBASE_IDENT_STRING=$HOSTNAME as this most likely does not inlcude spaces.

2. HBase uses the ./conf/hbase-default.xml file for configuration. Some properties do not resolve to existing directories because the JVM runs on Windows. This is the major issue to keep in mind when working with Cygwin: within the shell all paths are *nix-alike, hence relative to the root /. However, every parameter that is to be consumed within the windows processes themself, need to be Windows settings, hence C:\-alike. Change following propeties in the configuration file, adjusting paths where necessary to conform with your own installation:

1. hbase.rootdir must read e.g. file:///C:/cygwin/root/tmp/hbase/data
2. hbase.tmp.dir must read C:/cygwin/root/tmp/hbase/tmp
3. hbase.zookeeper.quorum must read 127.0.0.1 because for some reason localhost doesn't seem to resolve properly on Cygwin.

3. Make sure the configured hbase.rootdir and hbase.tmp.dirdirectories exist and have the proper rights set up e.g. by issuing a chmod 777 on them.

Testing the installation and configuration of HBase on Windows using Cygwin.
1. Start a Cygwin terminal.

2. Change directory to HBase installation using CD /usr/local/hbase-, preferably using auto-completion.

3. Start HBase using the command ./bin/start-hbase.sh

1. When prompted to accept the SSH fingerprint, answer yes.
2. When prompted, provide your password. Maybe multiple times.
3. When the command completes, the HBase server should have started.
4. However, to be absolutely certain, check the logs in the ./logs directory for any exceptions.

4. Next we start the HBase shell using the command ./bin/hbase shell

5. You can run some simple test commands

6. Leave the shell by exit

7. To stop the HBase server issue the ./bin/stop-hbase.sh command. And wait for it to complete. Killing the process might corrupt your data on disk.

8. In case of problems,

1. Verify the HBase logs in the ./logs directory.
2. Try to fix the problem.
3. Get help on the forums or IRC (#hbase@freenode.net). People are very active and keen to help out!
4. Stop, restart and retest the server.

Getting the code using git
Open the GIT bash or command prompt and follow the following commands:

$ cd
$ mkdir sparsemapcontent
$ cd sparcemapcontent
$ git clone https://github.com/ieb/sparsemapcontent.git
$ cd sparsemapcontent/
$ maven clean install
$ exit


For developing the code in eclipse

1. Import sparsemapcontent folder as existing maven project.
2. Include the following jar files into the project in case they are not there from /usr/local/hbase- folder.
· Hbase-.jar
· Hbase--test.jar
3. Start the HBase server as stated before.
4. Create the tables au, an, cn and smcindex.

So that was how I dirtied my hands in HBase. It’s a great DB to understand column DB concepts. I hope this was helpful.

Please feel free to mail me at kotwal.aadish@gmail.com. Hopefully your mail would put me in a tizzy!

Also interested readers may ponder over the matter in these site:

1. Configuration of HBase :

http://hbase.apache.org/book.html#configuration

2. HBase data model :

http://wiki.apache.org/hadoop/Hbase/DataModel

3. HBase book :

http://hbase.apache.org/book.html

4. HBase on Windows OS :

http://hbase.apache.org/docs/r0.20.6/cygwin.html

5. Place to start learning about Hadoop :

http://developer.yahoo.com/hadoop/tutorial/

6. HBase debugging and troubleshooting :

i. http://hbase.apache.org/book/trouble.html

ii. http://old.nabble.com/HBase-User-f34655.html

Sakai OAE Native mobile app: almost the end.

It is almost the end of the summer of code, (time flies!), so this post will be just a quick summary about how the project is going on, because I want to use all the remaining time to finish some code, add new features and write documentation.

So this is just a sneak view, to keep the community update. Again, I must begin this post thanking my mentor Carl Hall, for his hard work, he has asked every question that has arisen and also he gave me some freedom to make take some decisions. So, thanks Carl! I am really proud to work with you :)

Ok, let’s focus on what matters. If you read my previous post (Why should I develop native mobile application (sometimes)?), one of the bad things about writing native applications is the maintenance, when I wrote this post I have already complained about the problems of implementing functions in one platform that are completely different in the other. I have been stuck in a lot of device specific problems, I tried my best to take benefit of the iOS and Android user interface, so the layout is slightly different in this two apps.

I am going to quickly enumerate things that are already implemented:

• Skeleton application for Android : It has grown up a lot since last time I talk about it. It has 3 tabs for the main features in Sakai OAE that they match with the 3 main tabs on Sakai web version. Also it has the navigation menu inside “You” tab. All the strings than you can see are internationalized.
• Skeleton application for iOS: idem in IOS. It took me a lot of time make it work. As you could see in the demo tabs are in the lower part of the screen, to use the UI specific tabs, and make the application easier to use for iOS and Android users.
• Authenticate users in Android and iPhone: last time I wrote I had users could authenticate themselves sending their credentials in Android. Right now they can do the same also in the iOS application.
• Store and manage Sakai URL in Android: Carl and I were thinking than write down every time the Sakai URL was really tedious. So we add a new first view where the user writes the URL, the app checks if it’s a valid URL and store it inside the device. In Android it is stored inside the SharedPreferences of the app, so we can get the URL every time we needed it.
• Store and manage Sakai URL in Android: same for iOS. Here the URL is stored in a singleton class inside the NSUserDefaults.
• Calling a web service and show data in Android: Although the authentication was already a web service, we have achieve to call the Me Service with the user credentials, get back the Basic Information inside a JSON Object, parsing it, populate java classes and showing the information inside the Android App. Apache has really useful libraries to do this.
• Calling a web service and show data in iOS: again the result for iOS is the same, but the implementation has nothing to do, here I used JSONFramework to parse the data and Objective-C classes to manage the connection. This web service is inside a thread so it runs in background.
  
  

And since a picture is worth a thousand words, I have uploaded a small demo video to youtube.

I hope you find this project interesting. Thank you for your time and I’d be waiting for your comments.

Sakai CLE Mobile Application using phoneGap and jQuery mobile.

After a long time I think I should write something here as this will be the last or one before the last opportunity to write in the Sakai Google Summer of code projects blog.

Last month or few weeks was really a hard time because the things I had to handle were almost new to me and almost all the cases were issues and my mentor was getting so many mails with the title * issues (this is a regex) examples would be “Localhost issues”, “Cross domain issues” :-).

I think discussing solutions for them would be really worthy because when I googling also what I noticed in some places was “21 users has this question (Stackoverflow)” so I was the (n+1)th person to have same question.

First issue raised just after the last post because as I discussed in that I was using JSONP feeds from the server to render application on mobile but due to security reasons JSONP feeds were stopped and I was in trouble. When I searched I note that the solution is JSONP and having a proxy sever both are not going to work for mobile application as JSONP support also no longer there. But all these restrictions are for http:// and https:// protocols I realized later with the help from my mentor. Because as we are developing the application using phoneGap and it uses file:/// protocol there is no such restriction to get JSON feeds from a remote server so finally the problem solved. For completeness here is a snippet.

Next the issue with localhost, when we are going to test application with emulator (in my case Android) when it sees localhost with in the emulator (device) it is looking for a localhost inside the emulator and eventually failed to find. So I had to use my friends laptop as my dedicated Sakai server. But for mac users I saw a solution here [1] but I can't try it out. Though phoneGap wiki has something like this [2], I could not get it working, please correct me if I am wrong by adding a comment.

Next the very immediate issue I had was ajax requests that were in different pages to not working. The issue is like this, I used all the ajax requests in doument.ready event in different htmls but as jquery mobile uses ajax is used to load the contents of each page into the DOM as we navigate, and the DOM ready handler only executes for the first page. [3] So instead of DOM ready, in jquery mobile, we have to bind the pagecreate event in order to execute the code when the page is loaded and created by ajax. Note that the above snippet is using pagecreate. And one more thing I noticed most of the places is to use data-role=page and create new pages without adding a new HTML this will increase the loading time of each page as well. Use separate HTMLs if you really need only and if you are not happy with the pagecreate you can just use rel=”external” flag with your link like this,

but this will stop you by using page transition effects like slide and all and currently there is no way to get the effect with this.

And finally, I would like to add some screens that we can see in the application.

"This will be the view of a profile for an user in sakai CLE. User details are categorized in to sections as we can see and they collapsible too. Moreover users can update their status on the go via @sakaimobile :-)"

"This is how an user will see what are the new alerts from different tools in a selected site. We will, most probaby, be supporting Announcements, Assignments, Forums and Roster tools." Note that un-supported tools are grayed out. And users can see how many new alerts are there from each tool.

At the beginning as I said this will be the last or one before the last post in this blog but one day I might be writing here again but as a proud *mentor* for an student of Sakai foundation, who knows? :-)

Add you valuable comment and correct me if there is anything I have mentioned wrong. Thank you all for giving this opportunity to write in this blog and thanks to my mentor for guiding me and help me in all the issues I had.
Next week will be for any documentation stuff, if there are any, and correct the application where necessary to work with different devices.

[1]-http://stackoverflow.com/questions/5776412/access-localhost-on-mac-from-xcode-phonegap-communicating-with-ajax-to-a-local-r

[2]-http://wiki.phonegap.com/w/page/31996336/Testing%20an%20application%20via%20%22localhost%22

[3]-http://jquerymobile.com/demos/1.0b2/#/demos/1.0b2/docs/pages/page-scripting.html

Why should I develop native mobile application (sometimes)?

When we faced the question about if we should develop a web mobile application or a native mobile application, there is no universal answer (although sometimes, reading some articles, it seems there is one). When we begin a new development, first of all we must begin with a requirements analysis and functionalities that we are going to implement, and based on these we can make decisions. Just like when an architect designs a building, does not arise if there are good or bad materials or techniques, the architect just think if these materials or techniques are suitable for this particular work. We should think about every developmet independently and carry out a specific analysis for the application we will develop (otherwise the bogeyman will come and he will take away all our work, developers

who worked during the software crisis will tell us that this is true).

The purpose of this post is to analyze the case of developing a mobile application for Sakai OAE, of course, this is my analysis and I will be happy to rectify any mistake or any approach that can be wrong. Anyway, if you are interested in something more generic, google is our friend, simply search for "native apps vs web mobile" and you could spend all evening reading tons of articlees about this subject.

During this GSoC, two mobile applications are being developed for Sakai, and each one follows a different approach. I sincerely believe that both decisions are correct. I'm sure that Kasun Lakpriya and Steve Swinsburg who are developing the web-based mobile application for Sakai CLE, will give you plenty of good reasons why a web application makes sense to Sakai CLE. I'll focus on Sakai OAE.

Sakai is a young web application OAE, based on well-recognized standards, such as ajax or json, its design completely revitalizes the user experience by improving the workflow for both teachers and students. It is a highly scalable application that is developed following best practices ... It is definitely a really cool a web application. In fact, thanks to the hard work being done by the ux team, we can browse the web application using a mobile device and the application still will look really nice. So, in my point of view, since we have a good web application that with small changes may well be a mobile web application, it makes more sense to try to take advantage of the possibilities that native applications will give us.

And what these possibilities are? Well these are some of them that we could use in Sakai OAE, but certainly these aren't the only ones:

• Connectivity (Off-line mode): Although it's becoming less common, not all places have a good Internet connection, in Spain from where I write at least this is so. And web applications need, obviously, Intenet connection to work (even though we can achive some things with html5 and Google Gears). So being able to have the application running all the time is an advantage, we won't lose an alarm that warning us a deadline of a job, or stop reading some content because we don't have an Internet connection.


• Improve the user experience: Native applications, in general, improve the user experience, not only by the performance they can achive, but to use elements of the user interface of the device itself, although, of course this can also be emulated. It is also very interesting the possibility of having the application running in the background and receive certain notifications, as twitter apps do.


• Device-based caching:Certain information could be store encrypted on the device itself, which is very useful for improving the user experience, not forcing him/her to introduce the same data each time.


• Using the device features: This property could seem a bit strange in case of an educational application, but if you think about it is not. For example, when a teacher creates an activity, he/she could not only tag the date and time whe the activity will be done, also it is possible to introduce aGPS address indicating the place where the activity is take place, which is really helpful if we think about field practices, especially large campus or for students with no spatial orientation (this is my case). And surely we could imagine a lot of creative utilities using phone services such as the agenda or the camera. Why should we limit it?


• Discoverability: It's really easy to find an application in the market / app store.


• Performance: This is, in my opinion, the jewel in the crown of native applications. We can develop threads, use devices notifications, and make the application run in the background... And in the case of our application, where normally users turn on the app early in the morning when they arrive to university and they won't turn off until they go home, this is a very interesting feature to exploit.

Although there are also mobile applications features negative, we can see the positive side of these (or at least, do not consider them so negative)

• Automatic Updates: Web applications are updated automatically, while in the case of native applications we should upload them to market/app store and wait util the user updates them. This is initially a point for Web applications, but from my point of view, this is not so negative for Sakai OAE. The mobile application of a large university, must be stable and not constantly updated unless absolutely necessary (I'm thinking of a security bug), so we shouldn't worry too much.


• The censorship of the market/app store: I have write market/app store because I don't want to not arouse suspicions, but really where the applications pass quality tests and can be rejected is at the appstore. Again, this seems quite negative, but an application is rejected by quality issues primarily (or duplicate a service of IOS, which is not the case), which obliges us to work hard on the quality of the application, and Apple is not particularly ambiguous on how we should develop applications. If we'll see it this way, I honestly do not think it's an inconvenience at all.

And since not everything could be so beautiful, there are features that are inherently negative (if you can see a positive side please tell me!):

• Reaching the audience: Although Android IOS and monopolize the market for mobile devices, these are not the only platforms, which makes our mobile applications fail to reach the audience. But as I mentioned in the introduction, Sakai OAE is a good web application that can be accessed from any mobile browser.
  


• Maintenance: This really is a handicap. A native application implies that we must develop at least two projects for IOS and Android, and the rise of Windows Phone makes me think than maybe for this one too. This multiplies the cost of development and maintenance. I'm developing it for IOS and Android, and I suffer the problems of implementing functions in one platform that are completely different in the other, I can not deny this. Some months ago I was naive because I though that the development and maintenance costs would be lower since both apps are restful applications and the same web services are called. Anyway, Sakai community is big and great and I think the advantages outweigh the disadvantages.

I don't mean that web applications can't work as native applications. And every day we will find new frameworks to simply the work and get better performance.

To summarize, I don't think there is no good or bad answer about what kind of app develop. This is my analysis, and of course there are many nuances to all the features I've listed, I won't be surprise me if I'm completely wrong in more than one point, so I encourage you to leave me any comment with your thoughts and talk on here or IRC #sakai where you can find me as AdaHopper.

Mid-Term Report

Its been amost a month and a half since GSOC started and this is the mid-term report for the 1st phase of the project which is writing all the basic WebDav methods.

I am glad to say that I have been able to do that without encountering many problems.As I described in my proposal I used milton to rewrite the webdav methods.

Milton is an opensource library for adding webdav support for an existing web application.

Integration

The most common way of using milton is to add the MiltonServlet to web.xml.Milton is agnostic about the nature of the data. we can simply implement the interfaces needed for it to locate the resources.

The central concept to Milton is Resource so we extend the resource interface

for each HTTP method with methods to support that method .We can choose which http methods to allow on your data by choosing which Milton Resource interfaces

to implement.Thus a class which implements a resource will contain methods for GET,POST,MKCOL etc based on the interfaces we implemented.

Its completely appropriate to not implement any method if we do not want to support the method.

Progress

At first I thought milton was confusing but after some great support from the author(Brad Mcevoy) I found that milton is very easy to use.

I created 4 classes 3 implementing the milton interfaces and 1 helper class.The first thing that I did was to implement the File(SakaiFileResource) and Folder(SakaiFolderResource) Resources which represent

a Resource and a collection respectively.Then I implemented the ResourceFactory(SakaiResourceFactoryImpl) for finding the File and Folder Resources from the URL.

We take the path in the url entered by the user and check if it represents a resource or a collection based on this it returns instances of SakaiFileResource or SakaiFolderResource.

And finally I wrote the SakaiDavHelper Class which contains accessory methods.

The last step is to add the MiltonServlet to web.xml.

As of now I have written all the webdav methods using milton but without authentication (its a bit tricky with milton and the current dav code is confusing).The final

step is to add the milton jars to our pom.xml and we are good to go.

In my proposal I mentioned the rewrite of the nested classes DirContextSAKAI,ResourceInfoSAKAI and SakaidavServletInfo This proved unnecessary as milton

is resource centric and all the required information about a resource or a collection is contained within the file and folder objects(The function gerResourceInfo() does the job) thus eliminating the need for the rewrite.

Finally Thanks to my mentors Anthony Whyte and Seth Theriault for their support.

First month report in GSoC

Well, it's been a full month (and a little more), since I started working on coolest grant I know: GSoC. So it's time to make a properly report about project status, mainly to keep up to date with developments in the Sakai developer community.

I can only begin by thanking the work done by my mentor, Carl Hall, who patiently read my long post and still more patiently answer all my questions. Thanks Carl :)

I wish more things were finished at the end of this first month. But as recently I completed an important milestone, so I can be positive reviewing reached objectives above remaining objectives. Moreover, it's normal in the first stage takes a little bit longer development because I had to learn the technologies we work on the project (Objective-C, I'm looking at you, weirdo language), refresh the knowledge stored in layers of dust in my head (I'm sorry I left just Android), inquire about Sakai OAE (beautiful breath of fresh air I am sure will revitalize Sakai) and finish my exams (the GSoC starts early for European students).

But no excuses, let's focus on the things that if I achieved which really matters.

Sakai OAE: thanks to jar file that Carl gives me, put into operation a local version of Sakai has been trivial. So I could concentrate on reading the documentation and play a lot with Sakai trying to understand the basic operation of the application. The existing documentation is not extensive, but is compensated by the high availability which the project developers have to answer questions. If you have any problems, I would recommend that you to pass through the irc channel #sakai, there are always good people willing to help, or the mailing list sakai-dev,although the specificity for Sakai OAE is sakai -ui-dev. I have also found very useful documentation on quality tests because I have allowed to understand the user's work flow in Sakai.

Mock up: in the presentation of the project I've already showed some mock ups of the application and during this time we have been working,Kasun Kakpriya and I, trying our two mobile applications were most similar, so the user experience will be as similar as possible between the two mobile versions of Sakai CLE and Sakai OAE. We agreed in the work flow of the application, so that the two applications would have the same work flow and differ in some of the screens. This gives us flexibility to offer the best mobile version of each of the versions of Sakai and maintain the user experience. Moreover, regarding the decision that what mock up I will implement , I am still awaiting a answer from the design team Sakai OAE, but I preferred to continue developing the inner part of the application and wait a little bit longer to UI team could answer me. So, if you are part of the design team, would be really nice to get some feedback (thanks!).

Skeleton application for Android : it is ready the skeleton of the Sakai application on Android. It is a tiny skeleton for now. It has basically the view with the log in and a second view where he arrived after identifying the user. Created packages are also organized by the java classes as well as the internationalization files.

Skeleton application for iOS: idem in IOS. Although , as I said in the introduction: window management in IOS is very strange. Maybe it's because I already have experience in Android and it takes me more work to switch, or because management is really weird (I'd bet on the latter ;) ).

Authenticate users in Android: this is the milestone I have achieved. Users can now authenticate themselves sending its credentials. _Carl Hall and I, studied the possibility to authenticate users using the OAuth protocol, even it is a standard that is spreading rapidly, the implementation of this protocol is beyond the aim of GSoC. Also Nakamura's team is working on this. Nevertheless, the application is ready to change an authentication method for another one, so after the change will be not complicated.

I had this post ready for two weeks, so sorry for the delay. As you will understand the application development has advanced. If you will like to follow the development daily I will suggest to follow the repository: https://bitbucket.org/mrs.hopper/nellodeeoae

I hope you are interested about the project, I would like to hear some feedback. And if you have anything to ask or suggest me, plese do not hesitate.

Sakai CLE mobile application with phoneGap

In my previous post I mentioned how to setup the working environment and useful tools we can use to develop mobile applications using web technologies. Today I am going to present what I learnt during the past few weeks and the status of the CLE mobile application at the moment.

As discussed with my mentor, Steve Swinsberg, the basic requirements of the CLE mobile app. are as follows for the first version also listed here and the UI flows can be found here. (Screens v1.0.1) 

User login

User enters user details to enter the system.

 List of sites

s/he will be directed to a list of sites that they are in (assuming credentials are correct)

 List of tools

User selects a site will navigate them to a tools list of the selected site. Here they can see how many new announcements, assignments and etc are there at a glance.

 Mobile version of each tool

Once they select a particular tool they can view the above details in detail. This will hopefully be a read only tool for the first version of app.

Current status  

List of sites and tools for a selected site done.

Both this screens are rendered using JSONP feeds of my local host because JSON feeds did not work due to domain issues. Please find the code here and please add your suggestions.

Try out this in your emulators with phoneGap

If you want to run this in your emulator what you have to do is to create a phoneGap app. following their tutorial and add sites.html to assets/www directory.

To add JSONP support for your local sakai instance you can do the following change to core-providers in entitybroker and deploy *only* it in tomcat.

Checkout entitybroker trunk if you don’t have,

•  svn co https://source.sakaiproject.org/svn/entitybroker/trunk/ entitybroker-trunk
•  then in, core-providers/src/java/org/sakaiproject/entitybroker/providers/

edit the classes where it says:

add Formats.JSONP to the end      

• mvn clean install sakai:deploy (from entitybroker pom level)
•  re/start Tomcat.

Yeah that’s it! Now you have JSONP support.

Here is how to add a customize callback function,

That's it you should be able to see above screens. :-)

This is a useful place you can try JS, CSS, HTML http://jsfiddle.net/. Here we don't need to put all code just the snippets work nicely for example http://jsfiddle.net/HQK5w/6/

Thanks to my mentor, Steve Swinsberg, for the support and directions.

As always waiting for your comments. 

Sakai OAE Column Storage Driver Project

Hello, I am Aadish Kotwal, a now final-year-entering computer engineering student from Mumbai University. This is the first time I have entered into GSOC program and was (and am) very excited to associate myself with Sakai. Sakai Foundation is a great open-source organization. I had been through the past year’s projects and the blogs and thought working here would be a great learning experience. I have been working with many of the technologies Sakai uses for development and working here gives me a very big platform to take my interests and knowledge to a bigger level. I am associated with Sakai OAE, which in their own words is “A completely new system that incorporates all of the values of the Sakai CLE, and reimagines a new vision for academic collaboration.”

My project is titled “Sakai OAE Column Storage Driver”.

Put in Sakai’s words, Sakai OAE user content uses a storage mechanism based on a sparse map concept(for more on Sparse Map Content concept refer link [1]) representing column database type storage with a memory window onto the storage. This abstraction has allowed to create traditional RDBMS representations of the storage system with a MySQL driver capable of doing sharded storage over with 1 write/many read DB clusters. The original column driver was based on Apache Cassandra which need some updating to keep it in sync with the latest developments, and Sakai would also like to create a driver for another column DB (eg HBase, Raik, Mongo, CouchDB, etc or some network protocol approach, eg protocol buffer, thrift).

This idea involves modifying the existing Cassandra driver(why Cassandra? See link [2]) Sakai uses to include all the features present in the JDBC driver and then to create a new driver for a database which will be decided during the term of the project.

I have been in regular discussions with Ian Boston(my mentor). Based on his guidance, I came up with an analysis and a flow of the project which is described below.

What we currently have:
Cassandra driver for the sparsemapcontent with incomplete implementation

What I aim to achieve by the end of GSoC term:
A complete Cassandra driver and a new driver for a NoSQL database implemented from scratch.

The project flow which has been framed after discussion with project mentor is as follows:

• Finding all the methods and unit tests that require implementation and completing them.
• Working on the analysis of new driver.
• Start with coding API for the new driver with a structure similar to that of existing NoSQL driver. Also take feedback after completing this phase if any additional features specific to this database are expected.
• After finishing with a working API, I would start with the soak tests and will test the results by implementing it on local machines.
• Implement unit tests for the new driver to test functions such as content-addition, deletion, etc. on a local instance of database.
• Write integration tests for the new driver to check if it works well with the existing implementation of sparsemapcontent.
• Document the new driver and the implementation details at the confluence.

I have started working on technologies and codebase and have understood the basic structure. I plan to finish the entire learning phase prior to the coding stage and will also try my best to complete work on the existing database driver as early as possible so that I get more time for analysis and thoughts on implementation of new driver.

Progress till date:

1. My first task was to get serialization issues of Cassandra to be sorted, and this being my first task, Ian guided me wonderfully through it and made sure I stick to Sakai standards while focussing on efficiency. The task essentially was to write methods to convert an Object to ByteStream and vice versa. A quick link for the same for interested is: https://github.com/ieb/sparsemapcontent/blob/master/src/main/java/org/sakaiproject/nakamura/lite/types/Types.java

2. My second task was to implement indexing for efficient retrieval. This too under Ian’s guidance got implemented. This task included to find columns that were supposed to be indexed. The implementation links for the same are:

a) https://github.com/ieb/sparsemapcontent/blob/master/src/main/java/org/sakaiproject/nakamura/lite/storage/cassandra/CassandraClient.java

b) https://github.com/ieb/sparsemapcontent/blob/master/src/main/java/org/sakaiproject/nakamura/lite/storage/cassandra/CassandraClientPool.java

c) https://github.com/ieb/sparsemapcontent/blob/master/src/main/java/org/sakaiproject/nakamura/lite/ConfigurationImpl.java

3. My third task includes implementation of find on the Cassandra driver. The task is still under way.

My primary interest is to create a product of great significance which would be useful and accepted by the entire community.

A note on my mentor (Ian Boston):

GSOC on onset seemed very intimidating, but it was Ian’s support which really held me through. A thorough, detailed-oriented person, Ian has always been supportive and encouraging in writing efficient codes. The fact that such a busy man explaining the concept in so much detail really enthrals me. Any mistake and Ian not only rushes to correct me, but also shows alternative and better approach of implementation. Working under his guidance is going to be really fruitful this summer, and not to mention fun.

I will soon post of any further progress, and of course would love comments. My e-mail address is kotwal.aadish@gmail.com .

Looking forward for an awesome summer... :)

Links:

[1]: https://confluence.sakaiproject.org/display/KERNDOC/Sparse+Map+Content+-+Developer+Information

[2]: http://oreilly.com/catalog/0636920018537