Technical details on techniques used in the development competition

The post on winning the development competition has generated a lot of interest (relative to my other posts) so I’d thought I’d go into more detail about the technology we used.

First, there are some guiding principles that I follow when it comes to xquery web apps on MarkLogic. They are not set in stone, and individual projects may deviate. But these are my starting points:

1. Embrace the functional programming paradigm and the XML data model. Don’t fight it. Don’t try to turn XQuery into Java, or XML into an object model. Don’t try to reproduce hibernate, spring, jsp, php, rails, or any other object-oriented framework in MarkLogic. Let the server do its thing. You bought the server, so use it for what it is. Otherwise, write your own server.
2. Use a conceptually simple data model. Don’t start off with creating a data model structured with how it will be queried. Think of the entities that you will need and keep the number of them low. Have each type of entity have minimal references to other entities and keep them coarse. Keep it simple.
3. Make the application data- and configuration-driven for flexibility. Write the application so that it works according to the data it has. Facets are an example of this in search. The facets that are displayed are based on what came back in the results. You don’t have to specify up front what the individual facets will be. The server will figure that out at runtime. Build navigation and other options off of values in the DB. Write modular code that is discoverable at runtime. This is an advanced topic that can’t be sufficiently covered here, but the Masquerade approach (discussed below) is example of this.
4. Leave the HTML files intact. A rich interface requires plenty of HTML work so leave it alone. Don’t shred the HTML into XQuery. Don’t make the HTML only work on through a server. Let the front-end development take place in a static context, and then make replacements as needed dynamically at runtime. When the HTML needs to be modifued (which is constantly) the front-end dev can easily do it because the file didn’t have to change to be served on the server. The Nonob approach discussed below is an example of this.
5. Let collections and permissions restrict the data access for the user. Don’t write your own access restriction. Write code so that it just operates on the correct data based on the user’s permissions and the current collection in use.
6. Code to “bare metal” as much as possible. Use the built-in APIs if they can do what you need, even if you have to compose them together a little. Avoid custom functions just to “keep the code clean”. Avoid creating “gateway” functions, that is, functions that get the data the “right” way for you. Let the code get the data it needs, when and how it needs it. Not only does this reduce the amount of code you write, it allows you do surgically get just the data you want using predicates you want. It also reduces bug risk because the built-in APIs are going to be much more likely to be bug-free than something custom you wrote.
7. Use static javascript and CSS assets. MarkLogic is a powerful markup machine so use it to handle all the markup, and let the js and CSS be static and cacheable. This way most requests to the server are just for new markup that will have very little jas or CSS in it.

Ok, so how exactly do you do this? Below are some approaches, patterns, libraries, and techniques that my team used on the application:

1. Non-obtrusive HTML replacement (nonob).In this approach, the front-end dev creates the static HTML with whatever js and CSS assets he wants, using dummy data in the HTML, so that it’s fully mature right off of the filesystem. He then checks the HTML and assets into source control and the XQuery dev then finds which parts on the HTML page need to be dynamically replaced and writes replacement rules in an XQuery file. The XQuery dev checks in his code and leaves the HTML alone. The MarkLogic server executes the XQuery file, pulling in the HTML, makes the replacements using the the rules in the file and pulling in whatever data it needs, and returns the resulting HTML to the browser. The assets are also in MarkLogic so those requests just simply return the requested js and CSS files that the front-end dev put it. This is probably the single biggest time saver in development. There are several reasons why:
   • Front-end development and server development can happen in parallel. Given a design, both the front-end dev and XQuery dev can determine what needs to be done and they can bot start working on their areas. It’s common for the front-end dev to check in HTML, which then the XQuery dev refers to for replacements, even while the front-end dev keeps making changes to that HTML.
   • Changes can be made at any time by front-end or XQuery devs. At any point, if either front-end devs or XQuery devs need to change code or HTML, they can do it without having to consult with each other. There are limits, but this is true for the most part.
   • XQuery code only needs to focus on making the replacements, not building the page. The only XQuery code to write for HTML pages is getting the necessary data to make the replacements. Let the front-end guy deal with doing all the HTML.
   • The same XQuery code can do replacements on different HTML files. You can have one HTML page for desktop, one for tablet, one for mobile, one for IE6, whatever, and the same XQuery code can be used to do the replacements for any of the files. You can also have multiple views (Large image, no image, banner, etc) of a page and the same XQuery code is used. It’s just that the input HTML page changes.
2. Masquerade.This is a technique that abstracts the addressing of a resource from the code that will service that resource. Friendly URLs are a common example of this where there are rules or configuration that map URLs to pages. Masquerade takes it a few steps farther by querying the DB for code that has indicated that it responds to that URL and the appropriate code is used to generate the HTML based on the session or user agent. Here is how it works:
   • Request comes into the rewriter in MarkLogic
   • The rewriter queries the DB (particular collection, and using the permission set of the user) to find a “masq” that has an “url” element that matches the URL path. An example XML file for a masq may look like this:

     <masq key="aboutus">
         <url type="preferred">/about-us</url>
         <url>/aboutus</url>
         <url>/about-us.html</url>
         <view type="mobile" xml:lang="en">
              <html>about-us.html</html>
             <code>about-us.xqy</code>
         </view>
         <view type="desktop" xml:lang="en">
               <html>about-us-mobile.html</html>
              <code>about-us.xqy</code>
          </view>
     </masq>

   • The rewriter picks the best “view” of the masq that fits the client.
   • The rewriter calls the code that the view specifies and passes the starting HTML that the view specifies
   • The code uses nonob to make replacements on the HTML and returns the HTML back to the browser.
   • Making a new page and new page rule involves writing the HTML, the code, and creating a masq XML file. No changes to the rewriter are needed because masqs are queried at runtime
   • Pages that are off-limits to the certain users can be protected by setting the permissions on the masq XML file so that the query for masqs would not find the restricted ones for that user. You can also set the permissions on the xqy file.
3. Nonob i18n.In order to facilitate the page being in multiple languages, the nonob i18n library does the following:
   • In the static HTML that the front-end developer writes, custom HTML attributes are added to all the part of the page that need to be i18n-ized. These custom attributes do not interfere with any HTML coding tools and browsers ignore attributes they don’t understand, but the attributes are removed before being sent to the browser at part of the nonob replacement.
   • Here’s an example of i18n attribute on a div. The value of the i18n attribute is the key to lookup the translation in a resource file.

     <div id="mydiv" i18n="myapp.welcome-page.title">Welcome to my site!</div>

   • Here’s an example where you want to i18n-ize an input field:

     <input type="textbox" i18n="pages.register.name.placeholder" 
     i18n-target="value" value="Name"/>

   • The resource files are stored by language. And example of one might be:

     <resource-bundle xml:lang="spa">
         <resource key="pages.register.name.placeholder">Nombre</resource>
         <resource key="pages.register.submit">Submito</resource>
         ...
     </resource>

   • The resource to use is determined at runtime using the user’s language preferences sent in the HTTP headers (or previously set settings) and the resource bundle language available. So if the browser says, “ru,es,en” then the framework will pick the most appropriate language to use, which in our example is Spanish (es), because there is no Russian version.
   • After nonob has made the replacements (which is does in one pass through the HTML), and then i18n-nonob does it’s replacements in one pass, where it finds all the i18n attributes, gets the values from the resource files for the given language, makes the replacements, and returns the HTML to the browser.
   • This requires one line of code to use the library, and the only effort is to add the i18n attributes to the static HTML and create the resource files. Note that the page can dynamically set i18n attributes if it wants in the nonob step because the i18n-nonb step is just going to parse the whole HTML again after nonob is done.
   • Nonob + i18n nonob takes about 50 milliseconds usually
   • Nonob + masquerade + i18n-nonob provides a lot of flexibility and permutations with minimal coding.

Those are the more important parts that contributed the most to rapid development. We also already had a configure and deployment tool, and an authentication library so we didn’t have to spend much time with those. The book I am writing goes into better detail about how to do some of these things.

Enjoy

Avoid data coupling as well as code coupling

We talk a lot about reducing code coupling as a way to reduce the interdependency in the code so that changes to one piece of code can be made more safely. Implementations can be refactored, code can be modified, and new features can be added more easily if the code you are changing only has 5 interface points with other code, rather than 50.

Isn’t it the same with data? When you define a table in a relational database, you are defining what data parts an entity can have. And when you put data in that table, you are mixing that entity’s data with all the other entities that have data in that table. You are essentially coupling the data among all the entities in that table. Now you can’t separate the entities very easily and you can’t change the data structure or types for an entity without affecting all the other entities.

In a document-centric model, if each entity has it’s own document you can insert, delete and modify them with no effect on other entities. You can also create new types of entities, and even convert some old entities into the new type, without affecting other entities.

But even if you are using XML you still may gravitate to data coupling if the data gets sharded too much. I use a rule of thumb that an entity is something that has it’s own lifecycle. It can reference other entities (like a foreign key) but those are references to other entities, not sharded data parts of the same entity.

For example, a Person entity may have a “name” element which does not exist independently and does not have its own lifecycle. So “name” is an element of a Person. But that Person may be in a Company and that Company does exist independently from any Person and it has it’s own lifecycle, so Company should be a separate entity and each Person entity would have an id reference to the Company that it belongs to.

Now what happens if a Company entity gets deleted when a Person references it? Then you have an ID to nothing, but the Person still exists, still can be viewed, modified, and potentially reassigned, but the Person entity is still complete. And when a Person referencing some Company is deleted, the Company doesn’t know, doesn’t care.

I typically expect that the number of data models for entities in a MarkLogic web application should be able to be counted on one hand. And usually starting with what makes conceptual sense is the best approach: what “entities” do there seem to be that have their own lifecycles and how do they relate to each other?

This reduces data coupling so changes can be made easily and with low risk. Entities can be inserted and deleted without affecting other entities. It also tends to foster queries that are simple and “fully searchable” so that they utilize the indexes appropriately for maximum performance and minimal disk reads.

MarkLogic decidedly wins development competition

The challenge

A few weeks ago I got called out of my coding lair and was brought into a meeting with management where they informed me and 5 other people that there was going to be a friendly internal competition. A web application for tracking portfolios, initiatives, projects, and deliverables had been thoroughly designed for desktop\tablet and phone use. There would be two teams, each consisting of 1 QA and 2 developers. We were to stop our current work and devote our time and efforts on this, and in 1 week the two teams would reveal what they had done. Management would take care  of clearing our schedules. At the end of the week, the implementation that won would go on to being used for real by our group (about 120 people) and possibly other groups within the organization.

We were allowed to use any technology we wanted. The requirements (if we could meet them) were that the web application used our organization’s login accounts, was behind SSL, worked on a desktop and tablet (which had one design) and phone (which had a modified design). We were given high-fidelity mockups and access to the designer and the main user, and we were told to come back in a week and show how much we got done.

The requirements

I can’t post any screenshots because I don’t have permission, but I will describe the application a little. The main page of the application showed a timeline of all the projects for all the initiatives for a given portfolio. There may be 10 to 15 initiatives and each initiative may have 10 projects. Projects are 1 to 12 months and have usually a few deliverables. There are about 12 portfolios. The timeline on the main page has different colors for each deliverable indicating status, which can be modified via the main page if the user has permissions to do so.

There are three levels of access in the application: View, Edit, and Admin, and users are scoped to the particular portfolios that they should have any access to. Admin users can add other users and assign portfolios and roles. Users are added just by entering the user’s existing internal organization account username and the web app will import the user information from LDAP.

Editors can add and modify initiatives and projects, which each have their own screens. Initiatives have names, descriptions, year, and a few other fields. Projects have name, description, start and end dates, overall budget, dependencies on other portfolios, labor calculator (list of individuals and their bill rate, percentage engagement, number of days engagement, etc with dynamically calculated labor rate total), deliverables (with name, are delivery date) and a few other fields. Adding and removing labor rate line items and deliverables is in-browser, not a screen refresh, and the deliverable dates are scoped to be between the start and end dates of the project.

There is also a summary screen that is similar to the timeline page except it show all the initiatives and projects for a portfolio in sort of a spreadsheet view where you can collapse and expand initiatives to show or hide their projects. The budget totals are also shown for initiatives and projects.

That’s the gist of this application. It is non-trivial and had a very rich design and interaction. My team had an excellent QA, excellent front end dev, and me who was the only one who knew MarkLogic. The other team chose to implement theirs using a Javascript front-end architecture communicating with CouchDB (later Java with MongoDB) on the backend. The two teams involved very skilled people. If these two technology approaches were going to go head-to-head, these were the people to do it.

The results

So how did it turn out? We deployed our implementation to a server behind SSL hooked up to our auth scheme in my organization. We implemented all the requirements, and then some. We had three experiences fully implemented: desktop\tablet, phone, and dumbphone (meaning no CSS or javascript and limited functionality) which the server would pick for you based on detected device abilities. We had SOAP\XML and REST\JSON services. We had the entire application translated into 9 languages: English, Spanish, Russian, Greek, Japanese, Korean, Chinese, Arabic, and something else (I can’t remember).

We also ran the application through a series of speed, load, penetration, and injection tests. Average HTML page size was 1 or 2 KB. Average response time for HTML was 250 ms (assets were all cached after the first hit). With 1000 concurrent users, there was no perceived change in performance. There were no failures or warnings with session hijacking, cross-site scripting, malicious code injection, malicious character injection, or any other security test. This was using an old-ish former Apache server which was not even half-power for a MarkLogic recommended setup. One ED Node, no caching.

We did this all in one week.

The other team did not finish. They had a few pages that showed some data, but it was not deployed, didn’t have authentication, you couldn’t create or change data, no phone or other experiences, at least based on what they showed. No other languages than English either. No security, penetration, load, or injection tests. For whatever reason, that’s how things ended up.

What does it prove?

To me, this demonstrated hands down the speed at which you can implement non-trivial, fully featured, mature, enterprise-class, performant, and secure web applications very quickly, in only one week, with only three guys, sometimes.

So that application my team created is being used for real now, and it has had some bug fixes and other tweaks, especially with IE (did I mention it was cross-browser compatible?). Oh and did I mention the build and deployment time is less than 30 seconds?

It’s hard to argue with end results. People’s time is expensive, and the less time you need to spend on people creating frameworks, building stacks, adding in security and other enterprise stuff, the better. And with all the claims and arguments that get tossed around, all I really think is: scoreboard.

The 1 Millisecond Page

I had an epiphany today while writing some code. I was writing a utility for web apps to query the WURFL dataset to determine if the remote client is on a mobile device or not. The WURFL dataset has all the manufactured mobile devices with their user agents and capability profiles. There are about 15,000 devices and consequently about 15,000 device records. Not everyone may think it’s a good idea to query at runtime 15,000 records for every request, but I really wanted to make it work.

Everyone wants to get their query times low, and there’s a lot of satisfaction knocking time off and running tight and lean. After some tinkering, I got my query time for this mobile device detection down to 0 seconds. That’s Zero. It was a moment of Zen for me. Seeing the profiler say 0 seconds is just not the same as 0.001 seconds or anything else. In theory of course it took some time, but having the calculation come out to 0 for a very non-trivial feature got me thinking: how many other features can I get down to 0 seconds?

What if all the features of a web page took 0 seconds? Impossible? I have written several applications and the target server time for generating a page is around 250 milliseconds, or 1/4 of a second. This includes checking for authorization, querying the data, formatting the data, dynamically generating the HTML, and even translating it into a foreign language, all on the fly. So I know 250 milliseconds is possible because I’ve done it repeatedly. I also realize that going from 500 milliseconds to 250 milliseconds is easier than going from 250 milliseconds to 0 milliseconds.

But processors and RAM are getting faster, SSD drives may totally change the time profile of data retrieval, MarkLogic will itself get faster, as will web application code. So at some point, someone is going to write a web application where the server time to generate the HTML for it will take 1 millisecond. That will be faster than the TCP overhead of the computers trying to negotiate a connection to each other.

Gone are the days of pages taking 10 seconds to load or your visitor goes away. Now the expectation is that server-side HTML response is sub-second. For my stuff, I aim for 250 milliseconds, and that’s very doable, and I often seen times of 60 to 70 milliseconds (under load with 1000 concurrent users, on one box, checking auth for each request, etc. etc). But now I’m thinking that’s too long. I’m going to expect sub-100 millisecond pages for now, but I’m going to ultimately be looking to cross the 1 millisecond boundary, but a lot of that will come from hardware improvements.

Maybe this is like the sound barrier or the 4 minute mile, where it sounds crazy until it is actually done by someone, and then it’s the new standard. For dynamic, non-trivial web applications, I don’t think any technology would be more likely than MarkLogic to be able to pull it off. And I don’t think it’s really that far off in the future.

Happy coding.