Land of Zoho Creator

As some may have read in previous posts, LoZC got the chance to interview Suganya, Deluge Script’s Lead Architect, regarding some of the technical aspects surrounding Deluge and Zoho Creator. The questions and her responses appear below.

Thanks again for spending some time with us Suganya, and for taking the time to respond to these questions!

LoZC: How did you come to be the lead architect of Deluge and is that your primary job focus or do you also juggle other responsibilities at AdventNet?

Suganya: Deluge started off as a pet project of our CEO, Dr.Sridhar Vembu more than 3 years back based on some new ideas he had on language design. I have been involved with Deluge starting with the initial design and development.

LoZC: What other projects, applications, or web sites out there influence your current work and perhaps helped to inspire what we see in Deluge today? And which mentors, teachers, or books do you think most influence your design style?

Suganya: Deluge was initially built as a proprietary language to meet our internal needs. Deluge’s design is not influenced by any one language, however since the underlying execution engine is Java, and we are mostly Java programmers, that had some subconscious influence, but Deluge is a very different animal from Java. We did have a look at Ruby on Rails, PHP, Python and Lisp for inspiration and also read few research papers on data driven web applications.

LoZC: This question comes from Bill, commenting at the LoZC blog. Could you give us a high-level introduction to the overall ZC architecture and system setup, with special attention to how it handles the growth of traffic and balancing the load?

Suganya: Zoho Creator is a database-driven application and all the information with respect to the applications created in Zoho Creator is stored in a bunch of tables. You can create an application in the GUI mode or script mode. i.e. The GUI mode and script mode are just two different ways of viewing the application. All the changes that you make in the script mode will get reflected in the GUI and vice versa. And all the changes done in either of the modes will reflect in the live mode. Zoho Creator also includes the Deluge workflow execution engine that executes the deluge action scripts.

Zoho Creator runs on grid of servers and that ensures a high system availability. If there is an increase in the traffic, we need to just add more servers to the grid. We are continually enhancing this system, as we learn from experience, so that we can serve more traffic.

LoZC: Deluge seems to play an integral role in the functioning of Zoho Creator. Is this in appearance only or does Deluge really exist in the core of ZC’s execution engine, driving the applications’ runtime and making most every other feature possible?

Suganya: Indeed, Deluge is a part of the core ZC engine. The features in ZC where Deluge is indispensable include creation of views, specifying criteria, creating calculated / formula fields and of course the deluge action scripts.

LoZC: On the GigaOM blog in September of last year, you agreed with Anil Sharma of Vertex Logic, commenting that data manipulation “does become simpler when you think programatically” and going on to mention Deluge as an aid in creating complex applications. Since one of ZC’s oft-cited features is that a user does not need to do any scripting to build an application, yet it seems that some Deluge is almost always required for the creation of non-trivial applications, is it a goal for ZC to eventually allow a user to create any kind of application whether or not they choose to write Deluge script or is it assumed that Deluge will always be required for users to create the more complex applications?

Suganya: The design philosophy here is to marry GUI based development with Deluge scripts in a very seamless way, so that the transition from GUI to script feels natural. We do not believe that GUI based development is always and everywhere superior to scripts. We believe the two approaches need to be integrated well, exploiting the GUI for doing simple things, and weaving scripting in for complex needs.

You can create forms/fields, define relationship across forms, create different kinds of views, specify criteria without writing code. For e.g. you don’t have to write code to create a simple expense tracker application. These are actually what most online application creation tools allow you to do. But Zoho Creator goes a step ahead and allows you to include some logic also and what better way to present the logic than a few lines of code?

One difficulty you might face while writing scripts is remembering the syntax and the functions that are available. We are trying to solve that problem by providing a GUI based Script builder. We are also looking at ways to improve the script builder.

LoZC: Wikipedia describes software scripts as typically remaining “in their original form…interpreted command-by-command each time they are run”, but I suspect this is not exactly how Deluge works. And FOLDOC mentions that “many languages fall between being interpreted or compiled”, often “compiled into machine-language, interpreted, tokenized, or byte-compiled at the start of each run, or any mixture of these”. Could you explain a bit about where Deluge fits along this continuum and how Deluge communicates or integrates with the rest of the Zoho Creator system?

Suganya: Deluge program is parsed and stored in the database in a set of tables. The Deluge execution engine executes the code by fetching the code from the database. A lot of validation is done while parsing the code since the deluge parser understands the data model of the application. For e.g. Assume that a ‘Employee’ form has the field ‘Joining Data’ as one of the fields and this field is used in the “validate” script. Now if the user tries to delete the field, the parser will not allow it since it knows that the field is being referred to in the ‘validate’ script.

LoZC: It seems that Deluge provides some facilities for declarative programming, where the user can tell ZC what their application should contain in terms of forms and views, as well as some object-oriented support through the built-in classes and methods. Can you comment a bit on the design approach for Deluge’s syntax and what it tries to emphasize in terms of giving the user what they need to build ZC applications?

Suganya: The deluge form definition is similar to what HTML is for creating web pages. As of now, it defines only the structure of the form. How it gets displayed (I mean the layout and style) is not customizable. But we will be supporting that in future and the syntax that we have provided to define the structure of a form is simpler than a tag based language. The deluge action scripts are comparable to a server-side scripting language where you write the application logic.

The goal of deluge is to make programming accessible to more people, eliminating a lot of the complex context often needed to start programming. We want the syntax to be easy to understand even for the user who is looking at it for the first time, without consulting a manual. We try to build in that crucial context to help the new programmer in the script builder. We have abstracted things like database connections, SQL statements and the like, so Deluge is a lot simpler in the specific domain it targets: database driven web applications.

We also plan to give many more built-in functions so that users will care only about the logic needed for their applications. One good example I can give is the Hide and show statements that you can write in the “on load” of a form. We are actually generating javascript to do that in the background. The user just writes “Hide ” to do the work. In this case users need not care about browser compatibility. We make sure that it runs on all the browsers.

LoZC: This question comes from Paul G commenting on the LoZC blog: “I see deluge close to Java syntax. How different it is to Java?”

Suganya: Deluge does take some syntactic inspiration from Java, but the underlying model is quite different. Deluge deals in relational database tables, views, user input forms and the like, quite distinct from the basic object model of Java. The syntactic resemblance is because all of us who are developing Deluge are basically Java programmers and it comes from a familiar and very popular family of languages. But Deluge is not a syntax-determined language - the actual code is stored as data in a bunch of tables, and the syntax you see is a kind of “user interface” or “skin” for the language. It can be changed at will, without affecting the programs written at all. Indeed, it would be possible at a future date to see Deluge programs in different syntactic forms, with each user choosing the syntax they are most familiar with.

As an example, the code to display an alert box in deluge is

alert "Hello";

But you could generate and view the same code in your favourite VB like syntax.

MsgBox("Hello")

What would be stored in either case is identical.

LoZC: What drove the decision to provide a proprietary scripting language for ZC as opposed to, say, offering a subset of JavaScript, Groovy, Jython, or JRuby syntax to your users?

Suganya: The alternative would have been to let the user write PHP or Ruby or Javascript and then we host and execute that on the server side. But in that case we would merely be offering convenient hosting, with a web based editor/IDE for PHP or Ruby. They still have to worry about connecting to a database etc. But our vision for Zoho Creator is to enable far better productivity, and enable it for someone who is not necessarily a professional programmer. Spreadsheets have traditionally achieved that, through formulas and scripts, and that is the kind of audience we target for Zoho Creator. The reason Deluge offers more productivity is the natural and easy integration it achieves with all the disparate elements that makes up a web application: the relational database, the web server, the scripting environment and so on, and make it all accessible through a browser.

LoZC: Is Deluge built on some of the more common tools for building a scripting language in Java, such as ANTLR and the ScriptEngine interface, or would you describe Deluge as built from the ground up, independent of the typical Java building blocks?

Suganya: Deluge uses JavaCC parser generator and the Deluge execution engine is written in Java.

LoZC: Recently there have been some questions posed in the forums and on personal blogs out there requesting such things as user-defined functions in Deluge and an API for external interaction with ZC applications. Can you let us know what might be in store for Deluge’s future? Can we expect more and more functionality that gradually has Deluge resembling a “full-fledged” scripting language like Groovy or will the functionality stay narrowly focused with most future enhancements to ZC taking place outside Deluge? For instance, are there any plans to allow Deluge to request data from other ZC applications or even external URLs?

Suganya: Yes, we have major plans for Deluge and all that you have asked for are already in the pipeline. I believe we have just scratched the surface of what is possible here. In a nutshell, we have already started working on providing APIs to add, delete, update and view records in Zoho Creator. We will also be working on creating user-defined functions in Deluge, executing code based on time schedules (we now do it only based on user actions) and lot more. Very soon, we will also provide integration with other applications, including the web APIs of third party (i.e not from Zoho) applications. Stay tuned!