Talk with Charles: Charles’ Responses

Note: This is the last in a series of three interviews with senior members of the Zoho Creator software engineering team. Previous interviews with Suganya and Sriram, conducted this past August, are also available.

It took some time before Charles could fully respond to our interview questions but I think you’ll agree it was worth the wait.

For those paying attention there have been a couple upgrades and some hasty bug fixing over the past months. It was therefore timely (but perhaps also a little annoying) that several of the questions we asked Charles essentially revolve around the growing complexity of ZC’s code base. Below he responds to all the probes in depth, discussing his team’s current approach to developing an increasingly intricate site, how their interaction with users is evolving, and what’s in store down the road.

Read on to learn much more from Charles’, including some info on an upcoming defensive strategy for dealing with XSS security issues…

LoZC: In my estimation Zoho Creator has come a long way since it was released near the middle of 2006. Can you tell us a little about how you came to be its Chief Engineer and what your favorite features of the platform are today?

On a related note, about how many people are working on ZC now and in general terms how is your team organized to handle the design, new development, bug fixes, and support? As I see you are a Chief Engineer who also seems to provide frequent support via the forum and blog comments, do a lot of people on your team end up wearing several hats?

Charles: Zoho Creator was primarily started with a focus on catering our internal needs. Initially it was not a full fledged service. It was merely a form builder. When we decided to put the service live, we wanted to get as many early feedback as possible from real users. And I can tell, we have had a long journey from implementing so many features that you see today, apart from unnoticed ones like single sign on, Zoho Creator grid, and performance improvements.

As far as the roles that we play, yes you are correct. We do wear several hats. Supporting customers comes as the first priority before development and it is very important for the team to understand users’ requirement and develop accordingly. All our developers actively participate in the forums and also provide email support.

We learn a lot from our users. We first understand and appreciate what the users require. Apart from users’ requirement, we have to take care of usability and be innovative in its class. We then validate the requirements if it is in line with the vision of Zoho Creator. Such requirements will eventually be included in the plan immediately or postponed for a future update. A lot of feature requests also come from the internal users of AdventNet.

LoZC: I am so used to using Zoho Creator 2.0 now that I can barely remember what 1.0 was like. What compelled Zoho to perform this major front-end upgrade and what do you perceive to be the main differences and progressions when you look at 1.0 vs. 2.0? Is your team already envisioning a 3.0?

Charles: Zoho Creator 1.0 or the early version of Zoho Creator was good but not at the best in terms of productivity and usability. It was simply packed with so many features without paying much attention to usability. While no one complained, we still thought it could be done in a better way. We wanted to enhance the usage of Zoho Creator and decrease the time taken to build an application. Main theme of Zoho Creator 2.0 was “Build your application in minutes”. We wanted to make sure that each and every feature was well reachable and organized in a better way. Apart from these, we concentrated on Usability, Scalability and Speed. Users were really happy and we got a lot of positive feedback.

Next, we are indeed thinking/working on Zoho Creator 3.0. Stay tuned.

LoZC: A few months after ZC was released you personally collaborated with user Craig Cmehil in creating a roadmap for ZC features where Cmehil said he’d like to “see some dates, some thoughts…Where’s the Beef?“.

Now that we’re in the middle of 2007 and ZC development has arguably advanced beyond an ability to pay attention to user-driven road map requests, how has your team evolved its approach to user collaboration and what do you think works best in terms of keeping hundreds of eager application developers happy with what ZC offers? Even though ZC is not an open source project per se, it is a development platform for users so I’m wondering if your team has ever considered following the practice of more traditional development projects by publishing an official roadmap or a list of outstanding bugs?

Charles: Craig is one of our passionate users from the beginning. He volunteered to list the features requirements then. As of today, we have completed most of the features in the list and I can fairly say that we have done even more than that. Internally we do maintain a list of bugs and features. We have our own project plan. During the course of the plan, sometimes we drop a few, include a few and there will also be slips on target dates.

Today we have that little freedom of what we can do with the list of features. It may be necessary since we are in the early days. Give us few more months. First we want to cover some of the “must to do features” that are pending in Zoho Creator. We will then publish such a document in public. At this stage of development, we really cannot predict which feature will come first and which will come next. May be in short terms, it is possible to tell that. You can find them specifically in the forums. As I mentioned earlier, we take users’ requirement very seriously and if there is a real demand for a particular feature, then the chance of getting it inside is very high.

LoZC: Let’s talk about the increasing sophistication of Zoho Creator’s front-end for a moment, and the increasing complexity of the code base that I assume must accompany it. Recent projects like the wmii window manager eschew code complexity as the “mother of bloated, hard to use, and totally inconsistent software” and articles like Cynthia Rettig’s recent publication in MIT Sloan Management Review argue that software complexity creates an environment where developers “cannot test or even anticipate every possible usage path”.

So do you think ZC’s existence as a web-based application insulates it from these sorts of indictments or do you think there’s a limit to how much feature addition and user-requested customization ZC will be able to accommodate? Do you ever worry that your team’s development and test processes might eventually be outstripped by a front-end that seems to steadily add more features and capabilities month after month?

Charles: Today the amount of code that we have written is definitely huge. We do have our own internal practices for development. Being in the product business for more than 10 years has made us strong in that arena. I must also say that what might work for one team may not work for another team. However we have a very good process of maintaining a well structured code base. So that helps us in the development.

If you were to ask me, whether we think of each and every possible use cases before implementing a feature, the answer is no. Initially we may leave out some of the use cases. That will be done intentionally to make sure that we release the feature early and get those precious early feedback from users. They help in shaping the service and taking it to the next level.

If we were to think and implement of all possible use cases, we would never release the feature and most of the features that you see in Creator might not have been there. But there are some exceptions also. For example, when we did Zoho Creator 2.0, almost 99% of the Edit mode feature was done only by internal discussions. There were so many brain storming sessions. How much of the use cases in a feature will go inside will vary vastly from feature to feature.

LoZC: “Will there ever be a point where your team must say “No” to new front-end features for ZC?”

Charles: We cannot predict it. Definitely we never want our service to be bloated. But at the same time, we would like to add as many features as possible. When we add a feature, we make sure that it is intuitive enough to be reached by a new user and, it is not intrusive to an existing user. But if we were to choose between the two, we would give more weightage to existing users. Sometimes there could be some features that are heavy for Zoho Creator. For such features, we may decide to expose it subtly and suppress the majority of the configuration for a novice user. If we feel it is really too heavy, it could even be started as a separate service. And at a later stage, we would go for a tighter integration, as and when required.

LoZC: The forum and individual blogs both seem to be great places for users to post their questions and problems regarding Zoho Creator, someone on your team often replies the same day or even hours after a request or bug is posted. Can you describe your team’s approach to addressing bugs and prioritizing feature requests from users and explain how you are able and why you choose to deploy fixes and feature additions so quickly?

Charles: We love to assist our users. Be it in the forums or through email support, AdventNet’s policy is to give the best support even for free users. We really learn a lot from our users it helps us to shape the service better. With such interaction, it will be easy for us to understand and implement features. Whenever users come up with bugs, we understand the pain users undergo and the amount of time spent by them. We are indeed grateful to all such users who turn up and report bugs. Ideally we would like to keep the service bug free which we are striving hard to achieve.

LoZC: I personally don’t use a mobile device for web access at this time but understand one of your areas of focus is Zoho Creator’s support for mobile access. What kinds of challenges were involved in making Zoho Creator available on mobile devices, what particular advantages does Zoho see in particular for accessing a ZC application away from the desktop, and do you have any indication of what percentage of Zoho Creator access current occurs via a mobile browser as opposed to desktop? Has the recent presence of the iPhone in the marketplace lead to users accessing ZC applications via that device?

Charles: There is a huge potential for accessing Zoho Creator forms and views on mobile devices. Imagine you can track your personal health records or track your travel details from your mobile. You need to simply login to Zoho Creator from your mobile and fill your form and view the records as you travel. Your data is automatically stored in Creator DB and you can view the reports later in a PC or in the mobile itself. Basically it is the convenience. We want to project Mobile as another medium to access your forms and views on the move. It is almost due for the release.

LoZC: There have been a few mentions in the forum and even in a previous LoZC interview of a soon-to-be released feature where users will be able to better customize the layout of the forms they design. Can you tell us a bit more about what kinds and levels of control are planned and discuss some of the inherent trade offs between application stability/usability and application user-customizability?

As users are given more power and control over how their application is laid out, aren’t they by definition given more opportunity to introduce bugs, cripple certain browser configurations, or even effectively break their own applications by making poor or ill-informed UI decisions? On a related note, as users are given more power to configure their ZC creations in any number of ways, do you think there might be a need for the publication of “Zoho Creator Best Practices” in the near future?

Charles: As long as defaults are set properly, which we give a detailed attention to, users need not worry much about configurations. For example, we are planning to bring templates for applications. A novice user will be provided with options to choose in his capacity and will get what he wants. It will never be intimidating. And there will also be options for a power user to dirty his hands deep down in CSS and layout. Ultimately the power user will also get what he wants. We are confident of satisfying both the users.

LoZC: A couple months ago pdp at GnuCitizen pointed out some XSS vulnerabilities in Zoho Creator, which your team quickly addressed through some experimental site changes that eventually resulted in the ability for ZC users to set the XSS security level of any application to either High or Low. Do you think the Low setting still presents a security risk for users of ZC applications and, keeping in mind your expertise in the export formats, do you think the combination of relaxed XSS settings and JSON-encoded exports holds the potential for future security exploits?

Charles: In future, web applications will have to be prepared for even more XSS exploits. We have fixed most of the exploits in all Zoho services. As far as Zoho Creator is concerned, it is best to leave XSS setting to the default which is high. Usually it is for a personal user or an advanced user who will like to have the setting as low and make use of the html formatting. For example we saw how Douglas Lockwood had done. But to make sure that it does not affect a visiting/shared user, we are planning to ask the visiting users if we can load applications that are with XSS setting as low.

LoZC: In the public applications I haven’t seen many examples of people using the charts functionality recently introduced into Zoho Creator. Was this a feature in high demand from a certain user or certain group of users, or did you just decide it was a logical addition to the platform? Perhaps it is used more in private applications? Are there any plans to enrich this feature to give it some unique capabilities, say beyond what people expect from Excel, or plans for more chart types/customizations?

Charles: I agree it is still a kind of hidden feature. But this is just the beginning and we will soon start integrating with Zoho DB, the newest service from Zoho which specializes in extensive reporting capability. The current chart feature will complement Zoho DB integration.

LoZC: This last question is a two-parter. Raffic hinted to me in an email that I could ask you what’s next for Zoho Creator, so tell us, tell us!! Will ZC 3.0 perhaps feature an immersive environment like Second Life? Also in a recent post on the Zoho Blog, Raffic alluded to the fact that you have a hard time staying at your desk for very long. You’re a software engineer, right? What’s up with that?

Charles: We have the API and the mobile launch coming soon. Internationalization, Layout customization, SSL support, backup of application with data will follow them.

Regarding my timings, yes, I don’t follow a definite pattern. May be things could change once I get married

Uncommon Support

I ran across this article from a little over a month ago and wanted to pass it along. It’s a great example of just one reason why Zoho Creator is head and shoulders above competing products, spectacular support:

The next morning the entire development team from Zoho Creator called me from India. It seems that a bug in their code caused my little program to bring their application server to a grinding halt, preventing me and anyone from logging in. They were incredibly helpful and very responsive, fixing the problem on the spot. Zoho Creator, they explained, was a work in progress, still young and prone to these sorts of problems. Although it looks great and does exactly what I want it to, it isn’t the robust timesaver I hoped it would be today. By my next project, I am sure it will be robust and reliable.

Remembering that ZC is currently an entirely free product, the level of assistance people receive from the development team is just amazing.

Talk with Charles

For technical interview #3 we have the privilege of learning from Charles, another Lead Architect Chief Engineer for Zoho Creator and a frequent, helpful assistant on the ZC forums.

I’m sure Charles will try his best to slog through a list of questions from us regardless of the specifics but as for any particular questions you might have, below is a short list of some technical areas where Charles is the resident expert:

Your questions for Charles can be sent to me as comments to this post or in an email; as with previous interviews of the ZC team I’ll send them off at the end of the week and when Charles gets his replies back to us we’ll run the unedited response in a new post.

Updated: I inadvertently left out some of Charles’ specialties the first time I posted this. Those have now been added to the above list.

Talk With R.Sriram: Sriram’s Responses

Zoho Creator’s Lead Architect, R.Sriram, responded to LoZC’s battery of incisive and hard-hitting interview questions with all the detail one would expect from a seasoned software engineer. Before he got into specific answers however he pointed me at Hyther’s post from December 2005 that gives some great background on how Zoho Creator came to be:

It all started with the long tail buzz created by Chris Anderson. We were not aware of such theory last year when Chris published his original version in wired. Last march during a casual discussion with my CEO, he introduced me to longtail. That’s when it struck us - what we do here (at AdventNet) to solve some of the internal needs are really building situated applications…Do you know what our HR folks do? Hunt for some developer to make such form-based app for them. Why not HR themself could create such simple web apps?

So with that as a starting point, read on to hear what Sriram had to say about ZC’s technical challenges and arcitectural evolution…

And many thanks to Sriram for making the time to give an interview!

LoZC: How did you get involved with the Zoho Creator project and in simple terms how does the work you do connect to or integrate with LoZC’s last interview topic: the Deluge subsystem?

Sriram: Internal needs have been the foremost reason for the evolution of ZohoCreator. Most of the time our HR, Admin and Marketing guys had to depend on the developers and UI designers to develop simple day to day applications to collect data. It involved developing an Html with PHP/JSP code in it to add/retrieve data from database. Most of the applications are throw away applications.Thats when we came up with the idea of AppCreator (now ZohoCreator).

When we started it was just simple Form Builder. ZohoCreator is completely based on relational model and Deluge was a natural fit. Gradually when we integrated Deluge (which supports action scripts/workflows and formulas), ZohoCreator transformed from a simple Form Builder to a complete Online Database Application creator.

LoZC: What other sites out there do you consider to be Zoho Creator’s competing or kindred products? I understand that you are a developer but if you were to imagine a small business owner who needed to build an application cheaply and quickly, faced with all the “Web 2.0″ options out their that they could choose from, which technical capabilities of ZC do you think might compel him or her to choose ZC over an alternative?

Sriram: Well there are quite a lot of them. Its very difficult to single any single product or service. While each service has a different approach, we believe that the approach we have taken in ZohoCreator is different, ie., enabling even non programmers to create database applications on their own.

And coming to second part of your question, let me list some points which I think as Zoho Creator’s benefit over other competing services:

1. Ease of use: Zoho Creator has very user friendly interface and easy to understand jargons which makes it easy to create applications
2. Deluge Script: Deluge helps users to build powerful applications in ZohoCreator. With ZohoCreator you are not just building a Form or set of related Forms, but a complete Application with support for customized actions. Maybe that’s the reason why our users equate ZohoCreator to VisualBasic/ MsAccess over the web ;-).
3. Embedding: Zoho Creator allows embedding of Forms and Views in personal websites/blogs. This allows users to make their website dynamic.
4. Collaboration: Users can Collaborate with each other using ZohoCreator, irrespective of their geographical separation. The sharing concept in ZohoCreator is well received by the ZohoCreator community and they use it effectively to collaborate within their community.
5. ZohoCreator Evolution: Considering our users need, we make updates every few weeks. We keep adding new features and enhance existing features regularly. ZohoCreator is definitely evolving rapidly.
6. Integration with other Zoho services: Since ZohoCreator is one of the services in Zoho suite, you would be seeing more integration with other services. In future we will be providing a complete solution rather than a standalone service with seamless integration with other services.

LoZC: Several people at Zoho have pointed out numerous times that ZC now runs on a “grid”. Wikipedia describes a grid in part as a system of “parallel computing which relies on complete computers (with onboard CPU, storage, power supply, network interface, etc.) connected to a network…by a conventional network interface…” Without giving too much away or reliving too much past pain, what it was like migrating ZC from a non-grid to grid architecture last year? Can you let us know some details about ZC’s grid architecture such as the rough number of nodes, which operating system and hardware components they feature, and how the grid in general is monitored? Approximately how many concurrent users could Zoho support before migration to the grid and how many after?

Sriram: When we started, we hosted ZohoCreator as a Standalone service. As the number of users increased, the performance started to degrade. Thats when the Grid system went into production. So we planned to migrate ZohoCreator to the Grid Environment. Migrating ZohoCreator to the Grid Environment was a huge effort. Some part of the data model for the ZohoCreator had to be modified. Especially with regards to sharing. With a standalone service, there is only one database and all the users data resides in one database. Whereas in a database cluster the user information is distributed across the clusters. Hence the sharing information is distributed across different databases in different clusters.

At that time there were around 10 thousand users and 40 thousand applications. There were couple of applications which had millions of records. Migrating them were pretty time consuming. Typically the migration involved copying the application to the grid environment along with the data. That is migrating from old data model to new data model along with their relationships and sharing information intact.

Regarding the ZohoCreator’s grid architecture, we have a number of database clusters, file clusters and application servers. In each database cluster the master data is replicated by a slave, so that when the master fails the slave automatically takes over. The Grid monitors the health of each and every cluster and the application servers, and ensures their availability. The Grid takes care of balancing the load between the servers. There are standby components which replaces the failed components in case of failure. In addition we have the ManageEngine Application Manager which monitors the database clusters and application servers for failures. There has been a substantial increase in the number of concurrent users for ZohoCreator when running in Grid, than it was running as a standalone application. While we have not monitored the number of concurrent users, the number of users who login to ZohoCreator has increased substantially over the past few months.

LoZC: In February of this year Sridhar commented on the Zoho Blog that, “Creator is in a class of its own, as far as performance tuning is concerned, because it is a development environment with a lot of dynamic/evolving set of applications; while it has gotten faster than before, much work remains to be done.” (And I think his comment actually comes after migration to your grid architecture!) With that perspective in mind, how difficult is it to manage, maintain, and upgrade Zoho Creator compared to the other offerings in the Zoho family of products? What kinds of adjustments (caching, refactoring, etc.) have you made to increase its performance over time?

Sriram: The complexity is not related to the Grid Architecture. The Grid only specifies certain guidelines for the applications. The complexity is more to do with the nature of the application. There is more dynamism to ZohoCreator as everything is created on the fly. Not only the Forms/Views are dynamic, but the entire application syntax, the deluge script logic and the final database query that is sent to the database are also dynamic.

Deluge which itself is a language of its own, enables users to add logic incrementally to the application. So ZohoCreator along with Deluge can be visualized as a development environment. With the increase in user base the requirements too keep growing and we have to enhance ZohoCreator and the Deluge sub system progressively. Unlike other development environments, we do not have a release in terms of years, but in terms of days/weeks. We need to constantly add/modify our Data Model to support new features and as well ensure its quality and performance. With the growing number of users its becoming more challenging for the team to handle upgrades with minimal/zero down time.

The performance aspect of ZohoCreator is bit unique among all other Zoho services. ZohoCreator hits the database more than any other application. There are more write operations compared to other services and hence the database is heavily loaded. These database queries are very dynamic in nature. We have reconfigured some of the database configuration to speed up the database. We also made some modifications to our Caching system to make it more efficient.

The performance operation is an ongoing process. We are continuously monitoring the database parameters for tuning the database performance. The Caching system with regards to ZohoCreator is pretty tricky as the data changes very frequently in ZohoCreator. We are coming up with new algorithms and we hope to make much more improvements in future.

LoZC: In Suganya’s interview with LoZC she commented that your team is composed of “mostly Java programmers” and noted your use of JavaCC, which is released under the BSD license. Then there is also ZC’s use of JFreeChart (LGPL license) for its charting capabilities. Can you tell us a little about how devoted to open source, Java-based solutions the ZC team is vs. other options and perhaps what other languages, tools, or platforms are involved in the ZC architecture beyond Java? For instance did or would your team consider Dabble DB’s choice for charts, FusionCharts, which requires Flash?

Sriram: Thats a nice question. As Suganya mentioned Java has had a quite an impact on most of us in Zoho. Having said that we are open to other languages too. The main intention of using the Open source solutions and tools is to pass on the benefit to our customers. As such we try hard not to charge our users. So we keep our solutions mostly based on Open Source tools without compromising much on quality. We have a good Flash expertise in http://toondoo.com. You never know we might as well develop a good charting tool using Flash . We are not averse to good commercial tools. We might bundle them as part of our paid version if the cost involved is more.

LoZC: A development team using its own product internally (Paul Maritz’s notion of “eating your own dogfood”) can say a lot about a web application. I notice for instance that the support forum for ZC is not itself a ZC application but an instance of phpBB — so do you personally or does your development team actually use any ZC applications during the work day, perhaps to track small projects or bugs, or do you tend to prefer other database applications? Do you think a ZC application could be built to replace your phpBB forum?

Sriram: “Eating your own dog food”. Well thats the philosophy we follow religiously in Zoho. But again we have the freedom to choose between the tools. Some managers use ZohoProjects, while some use ZohoSheet to track the different projects and some even use ZohoWiki.

There are quite a lot of people who have created some very good applications for their personal use. Most of the marketing persons in AdventNet and Zoho use ZohoCreator for managing their databases. There are quite a lot of situated Applications we had created in the past two years. There are music, sports, books and cultural activities within AdventNet Recreation Club where the people involved predominantly use ZohoCreator. Our HR Department uses a Leave application for tracking Employees leave details. Not only that, they have developed lots of situated applications for New Employee Orientations, Trainings, Blood Camps , Cab bookings, collecting employee details for distributing gifts and planning tours.

We have different versions of Issue Manager which we have developed and used at different points of time for managing the product features and issues. And they are truly situated as the requirements kept changing and hence the application.

While its technically feasible to build a Forums/blogs, our current focus is to allow users to build database applications that are for business and personal use. A Forum/blog requires a very flexibile and cool UI with lots of customization. We believe there is more need out there for creating situated applications rapidly which are otherwise difficult with all the learnings involved. Our focus right now is to enable our users to build more such situated applications by adding new features required to build such applications.

As an alternative we will be providing flexible API’s for the users to customize the applications created in ZohoCreator.

LoZC: This question comes from Kelly, commenting on the LoZC blog. “…I am pretty new to Zoho Creator but was curious what plans you had for supporting SSL past authentication. Also, do you have any plans for supporting encrypted data in the DB, say for instance if I wanted to add a credit card number to a registration page.”

Sriram: Kelly: SSL support and Data Encryption are some of the pending items. We have no definite time frame for doing them, but we will be doing SSL soon.

LoZC: What steps does your team take to make sure users don’t lose their data in case of a catastrophic database failure or prolonged network outage somewhere in the world? Since ZC seems to be targeting small and medium sized businesses, what technical assurances can you offer the skeptical potential customer to ensure they won’t lose access to or simply completely lose their custom applications and critical data?

Sriram: This question stands valid not just for ZohoCreator, but Zoho as a whole. As a first step we are preparing to host and replicate the data in multiple data centers across the world, so that our customers are not affected by any catastrophic database failures or prolonged network outage. This way even if one of the data center is affected, the others would work fine.

For ZohoCreator, in the short term, we are working on providing users with the ability to backup their application along with their data, which can be imported into ZohoCreator at a later point of time.

LoZC: This question comes from Teresa, commenting at the Zoho Creator blog. “…it’s obvious Zoho creator relies heavily on sophisticated client-side technologies, like Ajax and possibly Flex, that have only been introduced in the past two years. Do you think it would have been possible to produce this type of application even three years ago? How much do the capabilities of the UI tools influence the design and architecture of Zoho creator? What kind of backend platform do you use to support the sophisticated UI; Blowfish, Shale?”

Sriram: Teresa: Zoho Creator definitely benefited from a lot of recent Ajax client-side developments. It was a bit of challenge both on the server as well as the client side. While the complexity on the server side is about breaking your application details and storing it in the database on the fly. The client side involves constructing those information back in an user friendly manner. Ajax has definitely helped us in making the client look cool as well reducing the server round trips.

We take a balanced approach so that we do not compromise much on our data model, design and architecture of ZohoCreator owing to the limitations of the UI tools. On the backend we use a Java based grid platform we have custom built to scale Zoho Creator.

LoZC: I’m ridiculously curious about the Creator REST API that several at Zoho have mentioned over the past year or so. If you’re involved with its development at all, when is it coming and what will it look like?

Sriram: We are working on the REST API. We had completed it for CSV/XML. We are working on JSON based REST API’s. We will provide you with an early access by this weekend. We have developed both REST API’s and XML-RPC/CSV-RPC. So stay tuned for the API release

Talk with R.Sriram

Next up on the LoZC interview list is R.Sriram, one of Zoho Creator’s Lead Architects. I was provided with this list of Sriram’s areas of focus:

Send your questions for Sriram to me in the form of comments to this post or in an email and at the end of this week I’ll get them to his inbox. Then next week or whenever he has time to reply we’ll run his unedited responses here at LoZC.

Yet another opportunity from the ZC team to let us dig into some of the “what” and “how” of the architecture, get our curiosity satisfied, and maybe learn a thing or two…

Talk With Suganya J: Suganya’s Responses

As some may have read in previous posts, LoZC got the chance to interview Suganya, Deluge Script’s Lead Architect, regarding some of the technical aspects surrounding Deluge and Zoho Creator. The questions and her responses appear below.

Thanks again for spending some time with us Suganya, and for taking the time to respond to these questions!

LoZC: How did you come to be the lead architect of Deluge and is that your primary job focus or do you also juggle other responsibilities at AdventNet?

Suganya: Deluge started off as a pet project of our CEO, Dr.Sridhar Vembu more than 3 years back based on some new ideas he had on language design. I have been involved with Deluge starting with the initial design and development.

LoZC: What other projects, applications, or web sites out there influence your current work and perhaps helped to inspire what we see in Deluge today? And which mentors, teachers, or books do you think most influence your design style?

Suganya: Deluge was initially built as a proprietary language to meet our internal needs. Deluge’s design is not influenced by any one language, however since the underlying execution engine is Java, and we are mostly Java programmers, that had some subconscious influence, but Deluge is a very different animal from Java. We did have a look at Ruby on Rails, PHP, Python and Lisp for inspiration and also read few research papers on data driven web applications.

LoZC: This question comes from Bill, commenting at the LoZC blog. Could you give us a high-level introduction to the overall ZC architecture and system setup, with special attention to how it handles the growth of traffic and balancing the load?

Suganya: Zoho Creator is a database-driven application and all the information with respect to the applications created in Zoho Creator is stored in a bunch of tables. You can create an application in the GUI mode or script mode. i.e. The GUI mode and script mode are just two different ways of viewing the application. All the changes that you make in the script mode will get reflected in the GUI and vice versa. And all the changes done in either of the modes will reflect in the live mode. Zoho Creator also includes the Deluge workflow execution engine that executes the deluge action scripts.

Zoho Creator runs on grid of servers and that ensures a high system availability. If there is an increase in the traffic, we need to just add more servers to the grid. We are continually enhancing this system, as we learn from experience, so that we can serve more traffic.

LoZC: Deluge seems to play an integral role in the functioning of Zoho Creator. Is this in appearance only or does Deluge really exist in the core of ZC’s execution engine, driving the applications’ runtime and making most every other feature possible?

Suganya: Indeed, Deluge is a part of the core ZC engine. The features in ZC where Deluge is indispensable include creation of views, specifying criteria, creating calculated / formula fields and of course the deluge action scripts.

LoZC: On the GigaOM blog in September of last year, you agreed with Anil Sharma of Vertex Logic, commenting that data manipulation “does become simpler when you think programatically” and going on to mention Deluge as an aid in creating complex applications. Since one of ZC’s oft-cited features is that a user does not need to do any scripting to build an application, yet it seems that some Deluge is almost always required for the creation of non-trivial applications, is it a goal for ZC to eventually allow a user to create any kind of application whether or not they choose to write Deluge script or is it assumed that Deluge will always be required for users to create the more complex applications?

Suganya: The design philosophy here is to marry GUI based development with Deluge scripts in a very seamless way, so that the transition from GUI to script feels natural. We do not believe that GUI based development is always and everywhere superior to scripts. We believe the two approaches need to be integrated well, exploiting the GUI for doing simple things, and weaving scripting in for complex needs.

You can create forms/fields, define relationship across forms, create different kinds of views, specify criteria without writing code. For e.g. you don’t have to write code to create a simple expense tracker application. These are actually what most online application creation tools allow you to do. But Zoho Creator goes a step ahead and allows you to include some logic also and what better way to present the logic than a few lines of code?

One difficulty you might face while writing scripts is remembering the syntax and the functions that are available. We are trying to solve that problem by providing a GUI based Script builder. We are also looking at ways to improve the script builder.

LoZC: Wikipedia describes software scripts as typically remaining “in their original form…interpreted command-by-command each time they are run”, but I suspect this is not exactly how Deluge works. And FOLDOC mentions that “many languages fall between being interpreted or compiled”, often “compiled into machine-language, interpreted, tokenized, or byte-compiled at the start of each run, or any mixture of these”. Could you explain a bit about where Deluge fits along this continuum and how Deluge communicates or integrates with the rest of the Zoho Creator system?

Suganya: Deluge program is parsed and stored in the database in a set of tables. The Deluge execution engine executes the code by fetching the code from the database. A lot of validation is done while parsing the code since the deluge parser understands the data model of the application. For e.g. Assume that a ‘Employee’ form has the field ‘Joining Data’ as one of the fields and this field is used in the “validate” script. Now if the user tries to delete the field, the parser will not allow it since it knows that the field is being referred to in the ‘validate’ script.

LoZC: It seems that Deluge provides some facilities for declarative programming, where the user can tell ZC what their application should contain in terms of forms and views, as well as some object-oriented support through the built-in classes and methods. Can you comment a bit on the design approach for Deluge’s syntax and what it tries to emphasize in terms of giving the user what they need to build ZC applications?

Suganya: The deluge form definition is similar to what HTML is for creating web pages. As of now, it defines only the structure of the form. How it gets displayed (I mean the layout and style) is not customizable. But we will be supporting that in future and the syntax that we have provided to define the structure of a form is simpler than a tag based language. The deluge action scripts are comparable to a server-side scripting language where you write the application logic.

The goal of deluge is to make programming accessible to more people, eliminating a lot of the complex context often needed to start programming. We want the syntax to be easy to understand even for the user who is looking at it for the first time, without consulting a manual. We try to build in that crucial context to help the new programmer in the script builder. We have abstracted things like database connections, SQL statements and the like, so Deluge is a lot simpler in the specific domain it targets: database driven web applications.

We also plan to give many more built-in functions so that users will care only about the logic needed for their applications. One good example I can give is the Hide and show statements that you can write in the “on load” of a form. We are actually generating javascript to do that in the background. The user just writes “Hide ” to do the work. In this case users need not care about browser compatibility. We make sure that it runs on all the browsers.

LoZC: This question comes from Paul G commenting on the LoZC blog: “I see deluge close to Java syntax. How different it is to Java?”

Suganya: Deluge does take some syntactic inspiration from Java, but the underlying model is quite different. Deluge deals in relational database tables, views, user input forms and the like, quite distinct from the basic object model of Java. The syntactic resemblance is because all of us who are developing Deluge are basically Java programmers and it comes from a familiar and very popular family of languages. But Deluge is not a syntax-determined language - the actual code is stored as data in a bunch of tables, and the syntax you see is a kind of “user interface” or “skin” for the language. It can be changed at will, without affecting the programs written at all. Indeed, it would be possible at a future date to see Deluge programs in different syntactic forms, with each user choosing the syntax they are most familiar with.

As an example, the code to display an alert box in deluge is

alert "Hello";

But you could generate and view the same code in your favourite VB like syntax.

MsgBox("Hello")

What would be stored in either case is identical.

LoZC: What drove the decision to provide a proprietary scripting language for ZC as opposed to, say, offering a subset of JavaScript, Groovy, Jython, or JRuby syntax to your users?

Suganya: The alternative would have been to let the user write PHP or Ruby or Javascript and then we host and execute that on the server side. But in that case we would merely be offering convenient hosting, with a web based editor/IDE for PHP or Ruby. They still have to worry about connecting to a database etc. But our vision for Zoho Creator is to enable far better productivity, and enable it for someone who is not necessarily a professional programmer. Spreadsheets have traditionally achieved that, through formulas and scripts, and that is the kind of audience we target for Zoho Creator. The reason Deluge offers more productivity is the natural and easy integration it achieves with all the disparate elements that makes up a web application: the relational database, the web server, the scripting environment and so on, and make it all accessible through a browser.

LoZC: Is Deluge built on some of the more common tools for building a scripting language in Java, such as ANTLR and the ScriptEngine interface, or would you describe Deluge as built from the ground up, independent of the typical Java building blocks?

Suganya: Deluge uses JavaCC parser generator and the Deluge execution engine is written in Java.

LoZC: Recently there have been some questions posed in the forums and on personal blogs out there requesting such things as user-defined functions in Deluge and an API for external interaction with ZC applications. Can you let us know what might be in store for Deluge’s future? Can we expect more and more functionality that gradually has Deluge resembling a “full-fledged” scripting language like Groovy or will the functionality stay narrowly focused with most future enhancements to ZC taking place outside Deluge? For instance, are there any plans to allow Deluge to request data from other ZC applications or even external URLs?

Suganya: Yes, we have major plans for Deluge and all that you have asked for are already in the pipeline. I believe we have just scratched the surface of what is possible here. In a nutshell, we have already started working on providing APIs to add, delete, update and view records in Zoho Creator. We will also be working on creating user-defined functions in Deluge, executing code based on time schedules (we now do it only based on user actions) and lot more. Very soon, we will also provide integration with other applications, including the web APIs of third party (i.e not from Zoho) applications. Stay tuned!

Talk With Suganya J

I’m happy to announce that in the coming days LoZC will be conducting and publishing an interview with Suganya, Deluge Script’s Lead Architect.

Please forward your own questions for Suganya to me in the form of comments to this post or in an email and then next Tuesday I’ll send them along and run her responses here at LoZC, unedited, as soon as she gets back to me.

It’s a chance for ZC fans and users to ask anything not suited for the ZC forum: future plans for Deluge and the ZC site, how the team develops and supports the site, whatever. Get all your burning questions answered!