Talk with Charles

For technical interview #3 we have the privilege of learning from Charles, another Lead Architect Chief Engineer for Zoho Creator and a frequent, helpful assistant on the ZC forums.

I’m sure Charles will try his best to slog through a list of questions from us regardless of the specifics but as for any particular questions you might have, below is a short list of some technical areas where Charles is the resident expert:

Your questions for Charles can be sent to me as comments to this post or in an email; as with previous interviews of the ZC team I’ll send them off at the end of the week and when Charles gets his replies back to us we’ll run the unedited response in a new post.

Updated: I inadvertently left out some of Charles’ specialties the first time I posted this. Those have now been added to the above list.

New Contributor: Lockworld

Right on the heels of Alan Bradford, I’m pleased to add Lockworld to the list of people contributing the occasional post to LoZC. Lockworld runs The Web For You blog that, among other useful technical tips for the web entrepreneur, recently featured a detailed four-part series on setting up a PayPal-integrated product catalog using Zoho Creator. I suspect even the experienced ZC user will learn a thing or two reading through that series…

So a hearty LoZC welcome to Lockworld! I’m personally looking forward to picking up some more tips and tricks as he shares his experience.

New Contributor: Alan Bradford

Just a quick note to welcome Alan Bradford as a contributor to LoZC. He’s currently very busy with life as a full-time student but has considerable experience using Zoho Creator in the workplace and you might just see a post or two from him once in awhile. Welcome, Alan!

Link to Edit an Individual Record

I’ve been doing some work for a client who chose Zoho Creator to prototype an auction application they are building. One of the requirements for the application is that after a user fills out its registration form, the application needs to send that user an email containing a link back to their editable registration info so they can later correct any mistakes or update the information.

It’s a pretty common requirement for any web application but unfortunately one that isn’t readily available in the current version of Zoho Creator. Though I expect a convenient facility will soon be introduced, it’s not here yet and in fact the only links to ZC data that are easy to create point to either entire views or individual read-only records.

Suspecting there must be a way to link to an editable record and of the opinion that the rest of what ZC had to offer my client through its “official” interface (letting them do much of the wireframing themselves, customizing the forms and views, embedding those forms and views in their test web site, etc.) would make figuring out how to do so worth my effort, I spent several (non-billed!) hours doing some research and tinkering. My results take up the rest of this post.

Digging into ZC “internals”

The first thing I did was employ the powers of Feed43 to analyze the HTML of an editable Zoho Creator view (I just grabbed a public view that was readily available for this example, the form fields it contains don’t actually have anything to do with user registration).

If you click Switch to advanced mode [?] on that page and click on the various [Reload] and [Preview] buttons that are available, you should be able to see how I pieced together links to editable records like this one by directly accessing a page on the ZC server called showEditForm.do.

All I did was use Feed43 to extract values normally passed to a ZC JavaScript function and include them in a URL. The arguments are typically only relevant to the internals of Zoho Creator operation, like:

• formid: 50432000000414441
• viewid: 50432000000414492
• tableName: t_50432000000414441
• record: 50432000000414604
• viewlinkId: 58

Though you might recognize that last one, viewlinkId, since its value corresponds to the Permalink ZC gives you. Anyway, I pulled all those fields out just so I can build an editable link to any record in my view.

Usage with Your Own Views

Here are the basic steps you can use to create an “link template” so someone could edit any record in any of your views.

• Visit that Feed43 page I created
• Change the value in the Address* field to the Permalink of the ZC view you are working with
• Click [Reload]
• Click [Extract]
• Click [Preview]
• Copy one of the links from the Feed Preview: [?] field.
• In the link you copied, replace landofzc with your own ZC username

If you visit your newly created link, it should bring up a page that lets you edit one of the individual records from your view.

Now, in conjunction with the magic ID field that Zoho Creator provides you, in your own ZC application you can hard code the majority of that link into an on success {} sendmail() call but replace the value of the pkValue= key with the current value input.ID and thus dynamically create emails that contain something like this:

Thank you for registering! You can visit your profile at any time to update the information.

Note that such a page will also respond to any On Edit events as well (on load{}, on validate {}, on success {}) you might want to include.

“Iraqi Wildlife Observation”

Amazing what shows up in Zoho Creator’s public applications list sometimes.

I’m not sure how long it will stay out there but a ZC user, jtrouern, is hosting an application that seems to track wildlife sightings in Iraq.

Accessing it now I only see one entry in the database but it’s a good glimpse of ZC’s localization capabilities since the app appears in both Arabic and English versions:

Male bird hit helicopter traveling north to Balad Airbase, breaking a window. Dead bird found inside aircraft upon landing.

Talk With R.Sriram: Sriram’s Responses

Zoho Creator’s Lead Architect, R.Sriram, responded to LoZC’s battery of incisive and hard-hitting interview questions with all the detail one would expect from a seasoned software engineer. Before he got into specific answers however he pointed me at Hyther’s post from December 2005 that gives some great background on how Zoho Creator came to be:

It all started with the long tail buzz created by Chris Anderson. We were not aware of such theory last year when Chris published his original version in wired. Last march during a casual discussion with my CEO, he introduced me to longtail. That’s when it struck us - what we do here (at AdventNet) to solve some of the internal needs are really building situated applications…Do you know what our HR folks do? Hunt for some developer to make such form-based app for them. Why not HR themself could create such simple web apps?

So with that as a starting point, read on to hear what Sriram had to say about ZC’s technical challenges and arcitectural evolution…

And many thanks to Sriram for making the time to give an interview!

LoZC: How did you get involved with the Zoho Creator project and in simple terms how does the work you do connect to or integrate with LoZC’s last interview topic: the Deluge subsystem?

Sriram: Internal needs have been the foremost reason for the evolution of ZohoCreator. Most of the time our HR, Admin and Marketing guys had to depend on the developers and UI designers to develop simple day to day applications to collect data. It involved developing an Html with PHP/JSP code in it to add/retrieve data from database. Most of the applications are throw away applications.Thats when we came up with the idea of AppCreator (now ZohoCreator).

When we started it was just simple Form Builder. ZohoCreator is completely based on relational model and Deluge was a natural fit. Gradually when we integrated Deluge (which supports action scripts/workflows and formulas), ZohoCreator transformed from a simple Form Builder to a complete Online Database Application creator.

LoZC: What other sites out there do you consider to be Zoho Creator’s competing or kindred products? I understand that you are a developer but if you were to imagine a small business owner who needed to build an application cheaply and quickly, faced with all the “Web 2.0″ options out their that they could choose from, which technical capabilities of ZC do you think might compel him or her to choose ZC over an alternative?

Sriram: Well there are quite a lot of them. Its very difficult to single any single product or service. While each service has a different approach, we believe that the approach we have taken in ZohoCreator is different, ie., enabling even non programmers to create database applications on their own.

And coming to second part of your question, let me list some points which I think as Zoho Creator’s benefit over other competing services:

1. Ease of use: Zoho Creator has very user friendly interface and easy to understand jargons which makes it easy to create applications
2. Deluge Script: Deluge helps users to build powerful applications in ZohoCreator. With ZohoCreator you are not just building a Form or set of related Forms, but a complete Application with support for customized actions. Maybe that’s the reason why our users equate ZohoCreator to VisualBasic/ MsAccess over the web ;-).
3. Embedding: Zoho Creator allows embedding of Forms and Views in personal websites/blogs. This allows users to make their website dynamic.
4. Collaboration: Users can Collaborate with each other using ZohoCreator, irrespective of their geographical separation. The sharing concept in ZohoCreator is well received by the ZohoCreator community and they use it effectively to collaborate within their community.
5. ZohoCreator Evolution: Considering our users need, we make updates every few weeks. We keep adding new features and enhance existing features regularly. ZohoCreator is definitely evolving rapidly.
6. Integration with other Zoho services: Since ZohoCreator is one of the services in Zoho suite, you would be seeing more integration with other services. In future we will be providing a complete solution rather than a standalone service with seamless integration with other services.

LoZC: Several people at Zoho have pointed out numerous times that ZC now runs on a “grid”. Wikipedia describes a grid in part as a system of “parallel computing which relies on complete computers (with onboard CPU, storage, power supply, network interface, etc.) connected to a network…by a conventional network interface…” Without giving too much away or reliving too much past pain, what it was like migrating ZC from a non-grid to grid architecture last year? Can you let us know some details about ZC’s grid architecture such as the rough number of nodes, which operating system and hardware components they feature, and how the grid in general is monitored? Approximately how many concurrent users could Zoho support before migration to the grid and how many after?

Sriram: When we started, we hosted ZohoCreator as a Standalone service. As the number of users increased, the performance started to degrade. Thats when the Grid system went into production. So we planned to migrate ZohoCreator to the Grid Environment. Migrating ZohoCreator to the Grid Environment was a huge effort. Some part of the data model for the ZohoCreator had to be modified. Especially with regards to sharing. With a standalone service, there is only one database and all the users data resides in one database. Whereas in a database cluster the user information is distributed across the clusters. Hence the sharing information is distributed across different databases in different clusters.

At that time there were around 10 thousand users and 40 thousand applications. There were couple of applications which had millions of records. Migrating them were pretty time consuming. Typically the migration involved copying the application to the grid environment along with the data. That is migrating from old data model to new data model along with their relationships and sharing information intact.

Regarding the ZohoCreator’s grid architecture, we have a number of database clusters, file clusters and application servers. In each database cluster the master data is replicated by a slave, so that when the master fails the slave automatically takes over. The Grid monitors the health of each and every cluster and the application servers, and ensures their availability. The Grid takes care of balancing the load between the servers. There are standby components which replaces the failed components in case of failure. In addition we have the ManageEngine Application Manager which monitors the database clusters and application servers for failures. There has been a substantial increase in the number of concurrent users for ZohoCreator when running in Grid, than it was running as a standalone application. While we have not monitored the number of concurrent users, the number of users who login to ZohoCreator has increased substantially over the past few months.

LoZC: In February of this year Sridhar commented on the Zoho Blog that, “Creator is in a class of its own, as far as performance tuning is concerned, because it is a development environment with a lot of dynamic/evolving set of applications; while it has gotten faster than before, much work remains to be done.” (And I think his comment actually comes after migration to your grid architecture!) With that perspective in mind, how difficult is it to manage, maintain, and upgrade Zoho Creator compared to the other offerings in the Zoho family of products? What kinds of adjustments (caching, refactoring, etc.) have you made to increase its performance over time?

Sriram: The complexity is not related to the Grid Architecture. The Grid only specifies certain guidelines for the applications. The complexity is more to do with the nature of the application. There is more dynamism to ZohoCreator as everything is created on the fly. Not only the Forms/Views are dynamic, but the entire application syntax, the deluge script logic and the final database query that is sent to the database are also dynamic.

Deluge which itself is a language of its own, enables users to add logic incrementally to the application. So ZohoCreator along with Deluge can be visualized as a development environment. With the increase in user base the requirements too keep growing and we have to enhance ZohoCreator and the Deluge sub system progressively. Unlike other development environments, we do not have a release in terms of years, but in terms of days/weeks. We need to constantly add/modify our Data Model to support new features and as well ensure its quality and performance. With the growing number of users its becoming more challenging for the team to handle upgrades with minimal/zero down time.

The performance aspect of ZohoCreator is bit unique among all other Zoho services. ZohoCreator hits the database more than any other application. There are more write operations compared to other services and hence the database is heavily loaded. These database queries are very dynamic in nature. We have reconfigured some of the database configuration to speed up the database. We also made some modifications to our Caching system to make it more efficient.

The performance operation is an ongoing process. We are continuously monitoring the database parameters for tuning the database performance. The Caching system with regards to ZohoCreator is pretty tricky as the data changes very frequently in ZohoCreator. We are coming up with new algorithms and we hope to make much more improvements in future.

LoZC: In Suganya’s interview with LoZC she commented that your team is composed of “mostly Java programmers” and noted your use of JavaCC, which is released under the BSD license. Then there is also ZC’s use of JFreeChart (LGPL license) for its charting capabilities. Can you tell us a little about how devoted to open source, Java-based solutions the ZC team is vs. other options and perhaps what other languages, tools, or platforms are involved in the ZC architecture beyond Java? For instance did or would your team consider Dabble DB’s choice for charts, FusionCharts, which requires Flash?

Sriram: Thats a nice question. As Suganya mentioned Java has had a quite an impact on most of us in Zoho. Having said that we are open to other languages too. The main intention of using the Open source solutions and tools is to pass on the benefit to our customers. As such we try hard not to charge our users. So we keep our solutions mostly based on Open Source tools without compromising much on quality. We have a good Flash expertise in http://toondoo.com. You never know we might as well develop a good charting tool using Flash . We are not averse to good commercial tools. We might bundle them as part of our paid version if the cost involved is more.

LoZC: A development team using its own product internally (Paul Maritz’s notion of “eating your own dogfood”) can say a lot about a web application. I notice for instance that the support forum for ZC is not itself a ZC application but an instance of phpBB — so do you personally or does your development team actually use any ZC applications during the work day, perhaps to track small projects or bugs, or do you tend to prefer other database applications? Do you think a ZC application could be built to replace your phpBB forum?

Sriram: “Eating your own dog food”. Well thats the philosophy we follow religiously in Zoho. But again we have the freedom to choose between the tools. Some managers use ZohoProjects, while some use ZohoSheet to track the different projects and some even use ZohoWiki.

There are quite a lot of people who have created some very good applications for their personal use. Most of the marketing persons in AdventNet and Zoho use ZohoCreator for managing their databases. There are quite a lot of situated Applications we had created in the past two years. There are music, sports, books and cultural activities within AdventNet Recreation Club where the people involved predominantly use ZohoCreator. Our HR Department uses a Leave application for tracking Employees leave details. Not only that, they have developed lots of situated applications for New Employee Orientations, Trainings, Blood Camps , Cab bookings, collecting employee details for distributing gifts and planning tours.

We have different versions of Issue Manager which we have developed and used at different points of time for managing the product features and issues. And they are truly situated as the requirements kept changing and hence the application.

While its technically feasible to build a Forums/blogs, our current focus is to allow users to build database applications that are for business and personal use. A Forum/blog requires a very flexibile and cool UI with lots of customization. We believe there is more need out there for creating situated applications rapidly which are otherwise difficult with all the learnings involved. Our focus right now is to enable our users to build more such situated applications by adding new features required to build such applications.

As an alternative we will be providing flexible API’s for the users to customize the applications created in ZohoCreator.

LoZC: This question comes from Kelly, commenting on the LoZC blog. “…I am pretty new to Zoho Creator but was curious what plans you had for supporting SSL past authentication. Also, do you have any plans for supporting encrypted data in the DB, say for instance if I wanted to add a credit card number to a registration page.”

Sriram: Kelly: SSL support and Data Encryption are some of the pending items. We have no definite time frame for doing them, but we will be doing SSL soon.

LoZC: What steps does your team take to make sure users don’t lose their data in case of a catastrophic database failure or prolonged network outage somewhere in the world? Since ZC seems to be targeting small and medium sized businesses, what technical assurances can you offer the skeptical potential customer to ensure they won’t lose access to or simply completely lose their custom applications and critical data?

Sriram: This question stands valid not just for ZohoCreator, but Zoho as a whole. As a first step we are preparing to host and replicate the data in multiple data centers across the world, so that our customers are not affected by any catastrophic database failures or prolonged network outage. This way even if one of the data center is affected, the others would work fine.

For ZohoCreator, in the short term, we are working on providing users with the ability to backup their application along with their data, which can be imported into ZohoCreator at a later point of time.

LoZC: This question comes from Teresa, commenting at the Zoho Creator blog. “…it’s obvious Zoho creator relies heavily on sophisticated client-side technologies, like Ajax and possibly Flex, that have only been introduced in the past two years. Do you think it would have been possible to produce this type of application even three years ago? How much do the capabilities of the UI tools influence the design and architecture of Zoho creator? What kind of backend platform do you use to support the sophisticated UI; Blowfish, Shale?”

Sriram: Teresa: Zoho Creator definitely benefited from a lot of recent Ajax client-side developments. It was a bit of challenge both on the server as well as the client side. While the complexity on the server side is about breaking your application details and storing it in the database on the fly. The client side involves constructing those information back in an user friendly manner. Ajax has definitely helped us in making the client look cool as well reducing the server round trips.

We take a balanced approach so that we do not compromise much on our data model, design and architecture of ZohoCreator owing to the limitations of the UI tools. On the backend we use a Java based grid platform we have custom built to scale Zoho Creator.

LoZC: I’m ridiculously curious about the Creator REST API that several at Zoho have mentioned over the past year or so. If you’re involved with its development at all, when is it coming and what will it look like?

Sriram: We are working on the REST API. We had completed it for CSV/XML. We are working on JSON based REST API’s. We will provide you with an early access by this weekend. We have developed both REST API’s and XML-RPC/CSV-RPC. So stay tuned for the API release

Encrypting Zoho Creator Data

Goal:

Encrypt and decrypt data in a Zoho Creator application.

Ideas:

We have two example applications this time:

• The first one demonstrates how you might encrypt, store, and decrypt U.S. Social Security and credit card numbers in a ZC application. It’s a barebones web page just to give you an idea of practical usage. You’ll only be able to successfully decrypt the records encrypted with your passphrase, the others will display as garbled text. I made a quick screencast of me putting it through its paces:

  

 

• The second example is about as simple as you can get in Zoho Creator, one form with one field, but on the whole this is more complex than the first example because the embedding web page is designed to help you encrypt and decrypt data for any Zoho Creator form field you care to designate. You need to be pretty familiar with the ZC GUI and configure several parameters to get it working; it’s mainly here to demonstrate the guts of what is going on and to help you build more domain-specific applications of the idea. The screencast shows me filling out the form and doing some encryption:

  

Blabber:

In both example applications, all encryption-related activity is taking place inside web pages that load a modified version of John Walker’s JavaScrypt tools. Take a look around that site if you’re interested in the implementation details.

No Web Server Necessary

You don’t actually need to access the example web pages on the LoZC web server, you can download them and run them from your own machine’s desktop. Walker says:

…to have any degree of security, it is essential that all processing be done on your computer, without involving any transmission or interaction with other sites on the Internet. A Web browser with JavaScript makes this possible, since the programs embedded in these pages run entirely on your own computer and do not transmit anything over the Internet.

So if you’re especially paranoid, you can download the web pages to your own machine, examine the source, load them from there, and be next to certain that no sensitive data is being transmitted during the encryption or decryption process.

The fact is that whether or not you take the extra-paranoid step of making local copies, all the encryption/decryption activity is happening in your own browser. The only data that Zoho Creator or the rest of the internet for that matter will ever see is the encrypted version of your data.

You can also feel confident in the quality of the encryption that is taking place. JavaScrypt uses AES 256-bit keys. AES provides “a cipher approved by NSA for encryption of TOP SECRET information”.

All that being said, JavaScript as a technology has its own inherent security issues so caveat usor.

Finally, local copies of these pages work great but if you’re using Internet Explorer it’s going to give you a security warning before the JavaScript will execute. Once you disable the warning or approve the page for local use, IE should work fine.

Clunkiness

There is no Zoho Creator API (yet) so getting our encrypted data into our ZC applications is clunky. Each time we want to save a record with some encrypted data, we’re locally creating encrypted data, embedding a ZC form in an HTML iframe that is preset to load that encrypted data, and finally asking the user to manually submit the ZC form. Yuck.

I suppose if you really wanted to hide the Zoho Creator form from your user, you might be able to do something with your own copy of the ZC HTML form, a hidden iframe, and polling one of the export format links to make sure the data made it to the ZC database, but that’s really too much of a hack. Even for me.

Once an API does become available there will likely be better ways to facilitate field-level encryption in ZC and in fact it’s not hard to imagine the Zoho Creator team someday providing “native” client-side encryption functionality in a future release of the ZC GUI. Until then, if you need support for encrypted fields you can use an approach like the one described in this post.

Data Bloat

Note that the encryption web page communicates with Zoho Creator forms by sending preset values through the query string. Given Internet Explorer’s 2048 character limit for GET requests and the inherent size increase that comes along with our hex-based encryption, as a general rule you probably shouldn’t try to encrypt any data that is longer than about 780 characters.

Private Applications & Private Links

If the ZC application you’re trying to access with the encryption pages is set to Private, make sure you’re actually logged in to ZC before you try to use them. Otherwise they won’t work because the links required to configure the encryption won’t be accessible to you.

Also note that Private Links, e.g.:

http://creator.zoho.com/username/form/1/dejgODOeBwhvEsE...

won’t work either. ZC does not seem to support sending in preset form field values in the query string with this kind of URL.

Related Links:

JavaScrypt and Gmail, Encrypt This! Firefox Extension

Here’s a ZIP file of all the HTML and JavaScript used in the example pages.

ZC Weirdness (potential bugs in the system):

Form presets sent through the query string don’t seem to work for Drop Down fields (i.e., when field cc_type was defined as a Drop Down, its value was not being set in the Account Information example application, even though the query string was trying to set it. So even though the external web page uses an HTML dropdown, I had to change the ZC application so that Credit Card Type was of type Single Line.

Also there is the Private Links issue mentioned above but possibly that one is by design.

Seattle Coffee Shops

Goal:

Update a Google Maps map using Zoho Creator.

Ideas:

The example application asks the user for the name of a coffee shop in Seattle and its street address, then saves the information. The web page embeds the form and updates a Google Map map every minute, indicating any new coffee shops with a coffee cup icon:

Mousing over a coffee cup icon should show the name of the shop and its address. The user can add a new coffee shop and expect to see it update the map within the next minute. They can zoom in by double clicking on the map, zoom out by double right-clicking on the map, or drag the map by holding down click and dragging the mouse.

Blabber:

Since any public Zoho Creator application can export its view data in JSON format, using that data in conjunction with the Google Maps API is fairly straightforward.

Here are the 10 steps I took to get the example up and running:

1. Create the application in Zoho Creator
2. Add some sample coffee shops
3. Set up a free landofzc.mywebcommunity.org site at AgilityHoster (I’ll probably use this site for more LoZC examples later, as well…)
4. Get a Google Maps API Key
5. Locate the JSON export URL for the ZC application view
6. Add the text callback=showAddresses to the URL so the data is wrapped in a JavaScript function
7. Refer to this article on XML.com to refresh my memory and retrieve jsr_class.zip
8. Make a small change to the jsr_class code per Charles’ directions
9. Compose the web page
10. FTP the web page to landofzc.mywebcommunity.org

A simple diagram depicting the JavaScript-based interactions within the page shows both my Zoho Creator application and Google Maps responding to requests and returning the requested data:

The HTML of the web page doesn’t validate; like most LoZC examples so far this one was thrown together quickly. Another product of haste is that if you’re zooming around and panning when the map refreshes, it will recenter and revert to its original zoom factor. These annoyances could be addressed with more time spent on the code.

In general terms and provided you can hack out some JavaScript, using the fields in your Zoho Creator views to fill-in an external page doesn’t take much time. And once ZC gets its own API, in theory your external pages should be able to save data back to ZC too, rather than the current situation where you’re limited to the use of an embedded form.

Related Links:

Creating Dynamic Client-side Maps Mashups with Google Spreadsheets

ZC Weirdness (potential bugs in the system):

None! The ZC application in this case is very simple and relies on what ZC does best: serving a custom form, saving the submitted results, and providing flexible export options for the saved data. Update: Oops, I spoke too soon. It seems that the embedded form isn’t re-rendering properly after you submit an initial entry. I’ll update here again when I get more info… Update: Fixed. The issue was with an on load {} event I was using to record visits to the form. I have reported this bug to the ZC team.

Checking In, 82 Weeks Later

Beta review on Zoho Creator:

The very intention of Zoho Creator is that a novice user

• should be able to build applications very quickly that will suit his/her own needs.
• should be able to build forms and various views of the data effortlessly
• should be able to share the pages built with other people

Depending on your definition of effortlessly but keeping in mind the substantial improvements we’ve seen with the recent release of version 2.0, ZC is arguably still very much on track with realizing the original mission…

Talk with R.Sriram

Next up on the LoZC interview list is R.Sriram, one of Zoho Creator’s Lead Architects. I was provided with this list of Sriram’s areas of focus:

Send your questions for Sriram to me in the form of comments to this post or in an email and at the end of this week I’ll get them to his inbox. Then next week or whenever he has time to reply we’ll run his unedited responses here at LoZC.

Yet another opportunity from the ZC team to let us dig into some of the “what” and “how” of the architecture, get our curiosity satisfied, and maybe learn a thing or two…